Gc doesn't work for one repository on Stratum1 (but works on another server)

Stratum 1
1

On Euclid project we have several Stratum1 servers connected to our Stratum0.

There are two repositories on Stratum0 : euclid.in2p3.fr and euclid-dev.in2p3.fr
Garbage collector is activated only on euclid-dev.in2p3.fr.

So on the first Stratum1 gc works well on euclid-dev.in2p3.fr.

But on the second Stratum1 gc reply :

> cvmfs_server gc -a euclid-dev.in2p3.fr
There are no active garbage-collectable repositories

cvmfs_server check also have some problems on this repository.

> cvmfs_server check euclid-dev.in2p3.fr
Verifying integrity of euclid-dev.in2p3.fr...
failed to fetch repository manifest (1 - failed to download)
failed to load repository manifest
ERROR from cvmfs_server check!
Finished euclid-dev.in2p3.fr at Mon Sep 30 09:53:29 UTC 2024

It’s certainly indicate that euclid-dev has some problem on this Stratum1.

But we would like to have some advice :

  1. why it happens ?
  2. how could we fix it ?
2

First, FYI the gc -a option is intended to be used without additional parameters. The -a is supposed to figure out which repositories are collectable and select those. From my reading of the script, by giving it the repository name I think the -a tells it to only select the given name if it is collectable. However in this case it thinks the repo is not collectable …

If it is unable to read the manifest, that would explain why it thinks that repository is not collectable. I bet if you removed the -a option on gc you would see a similar error as on check. Does check work on the other repository? It would be very unusual if one repo was readable and the other wasn’t. Assuming you’re using the default access to local disk, the access goes via apache. I would suspect a problem with the repository’s configuration in /etc/httpd/conf.d or a permission problem on the files. You can look in apache’s access_log and error_log to see if that helps narrow it down. Or you could do

curl -vs http://localhost/cvmfs/euclid-dev.in2p3.fr/.cvmfspublished | cat -v

to see if that gives good debug information.

Dave

3

Thank you for quick reply, Dave.

Yes, globally we do cvmfs_server gc -a in our cron, so launching “cvmfs_server gc -a euclid-dev.in2p3.fr” it’s some bad copy-paste.

But fundamentally the problem is still here.
We have 2 Stratum1 connected to the same Stratum0.

On the first Stratum1 everything seems to work :
gc works for euclid-dev.in2p3.fr (just checked, I launched it also with “-a” option), check works as well and I can do curl/wget to .cvmfspublished (to localhost and to Stratum0).

On the second Stratum1.
Repo euclid.in2p3.fr seems to be fine : gc doesn’t work because it’s not activated on stm0, but “cvmfs_server check” works.
But repo euclid-dev.in2p3.fr has some problems : gc says it’s not activated and check says “failed to load repository manifest”.

I have not direct access to the second Stratum1 (only the first one), so I transmit the questions to this Stratum1’s admin.
I’ll be back here when I get the responses.

4

Thank you, Dave.

The issue is solved.

As you supposed there was some problem with rights and apache.
We had some problem with space, so some files were moved and symlinked and apparently Apache don’t like that.

Once symlink were fixed it works perfectly now.