"mismatch of file name and content hash" error following Stm0 reinstall and disk 100% full

Stratum 1
1

This is probably a follow-up to ticket :
https://cernvm-forum.cern.ch/t/failed-to-fetch-repository-manifest-2-incomplete-manifest-due-to-a-disk-corrup

Once we recovered our Stratum0, we relaunched snapshot on Stratum1.
But we weren’t careful enough and the disk of our Stratum1 disk ended up 100% full.

We increased the disk and relaunched snapshot and it seems to work now, but we found that we have some problems.

  1. GC is activated on Stratum0, but it doesn’t work on Stratum1.

Running Garbage Collection
→ marking unreferenced objects [Wed, 05 Aug 2026 08:30:41 GMT]
→ sweeping unreferenced objects [Wed, 05 Aug 2026 10:21:31 GMT]
HTTP protocol error 200: http://localhost/cvmfs//data/a7/392a<very_long_hash2> (9)
→ done garbage collecting [Wed, 05 Aug 2026 10:22:41 GMT]
garbage collection failed

  1. As our disk was 100% it seems to be similar to
    How to recover from filling the disk to 100% on a stratum-1 server?

We launched
cvmfs_server check -i <fqrn>

and we get a lot of “mismatch of file name and content hash”.

But the solution proposed on the forume doesn’t work for us.
For example the file from gc error (with hashes a7 and 392a<very_long_hash2>) doesn’t exist on Stratum0 (so we could not do curl).

We will probably reinstall our Stratum1 from scratch.

But I post this problem here in case if there are some fix that we could do before/instead of reinstall.

2

If there are no copies on the stratum 0 or on any other stratum 1, taking a fresh snapshot in that case is the correct thing to do.

If your stratum 1 is in active use there are ways to do that with practically no downtime, assuming there’s sufficient disk space to keep another copy. Reply here if you would like details on how to do that.

3

Thank you for reply.

There are some other Stratum1 which could be used by clients.

But I’m not sure we could reinstall Stratum1 server right now.

How could we take a fresh snapshot on existing Stratum1 (if we could allow some downtime) ?

4

If you can allow downtime, the easiest way is to cvmfs_server rmfs <fqrn> and then do a new cvmfs_server add-replica <stratum 0 url> <public key> and cvmfs_server snapshot <fqrn>. If the repo is big and you want to save some time you can first mv /srv/cvmfs/<fqrn> /srv/cvmfs/<fqrn>.delete and start a background process doing rm -rf /srv/cvmfs/<fqrn>.delete.

5

Ok. Thank you Dave.
I’m away this week, but I’ll try it when I come back.

6

Just to close this topic.

We finally recreate the new Stratum1, relaunch snapshot and everything is good for now.
But it was useful to learn how to do it if the have to use the same server.