What to do when metadata files are empty?

Stratum 1
1

Hi,

what is the best way to proceed when suddenly the metadata files for a replica are empty?

[root@stratum01 a.replica.org]# ls -ltra
total 132
drwxr-xr-x 259 root root 65536 Apr 14  2021 data
drwxr-xr-x 141 root root 65536 Jul 18 14:40 ..
-rw-r--r--   1 root root    29 Sep 21 12:36 .cvmfs_is_snapshotting
-rw-r--r--   1 root root     0 Sep 21 16:10 .cvmfsreflog
-rw-r--r--   1 root root     0 Sep 21 16:10 .cvmfswhitelist
-rw-r--r--   1 root root     0 Sep 21 16:10 .cvmfspublished
-rw-r--r--   1 root root     0 Sep 21 16:10 .cvmfs_status.json
-rw-r--r--   1 root root     0 Sep 21 16:10 .cvmfs_last_snapshot
drwxr-xr-x   4 root root   209 Sep 21 16:10 .
drwxr-xr-x   2 root root   192 Oct  2 09:12 stage

Thanks a lot in advance.
Cheers,
Jose

2

If it’s a small repository I would just start from scratch with a fresh replica, because who knows how many other files are also zeroed out.
I believe that John DeStefano has experienced that type of symptom before due to some type of disk errors or possibly crashes. You could also try removing all the .cvmfs* files and running snapshot again. I think it would probably then check all the files to make sure they exist. It wouldn’t however check to verify that the files all have content matching the hashes in their names. That could be done with a cvmfs_server check, or you might try doing a find for other zero-length files under data.
Oh, I now see that the “stage” directory is there, meaning that you are showing the top level of a repository managed by my cvmfs-hastratum1 package. So in that case the cvmfs_server snapshot command is actually writing down into the stage subdirectory. If all the .cvmfs_* files there are not also empty on the master machine, I would be really surprised, because the pull_and_push command copies all the .cvmfs* files from the stage subdirectory on the master machine to the top level of both machines when it has successfully completed a snapshot on both sides. Those .cvmfs* files under stage would then be the files you would need to remove if you were trying to recover without starting from scratch with an add-repository command.
Dave

3

They are 3 fifeuser<N>.opensciencegrid.org repos. I will check their sizes.

4

Those are pretty small, about 130GB each.

5

We used to have some problems with 0-size files but I don’t recall if they were metadata files.
Then we set CVMFS_SYNCFS_LEVEL=cautious (which seems what you’d always want IMHO) and it has been fine.