Problems repairing the.cvmfsreflog

[sgaravat]

Dear all

I have the following deployment:

• CVMFS stratum 0 configured also as gw, hosting several repos
• Backend implemented through S3 (ceph-rgw)
• Publishing is node through a publisher node

I see problems publishing on one specific repo:

ubuntu@cvmfs-publisher:~$ sudo cvmfs_server transaction repo15.infn.it
Gateway reply: ok

ubuntu@cvmfs-publisher:~$ sudo cvmfs_server publish repo15.infn.it
Using auto tag 'generic-2024-07-04T10:16:51Z'
Processing changes...
Waiting for upload of files before committing...
Committing file catalogs...
Wait for all uploads to finish
Exporting repository manifest
Lease end request - error reply: {"reason":"missing_reflog","status":"error"}
SessionContext: could not commit session. Aborting.
[ERROR] Failed to commit transaction.
Statistics stored at: /var/spool/cvmfs/repo15.infn.it/stats.db
Synchronization failed

Executed Command:
cvmfs_swissknife sync -u /cvmfs/repo15.infn.it -s /var/spool/cvmfs/repo15.infn.it/scratch/current -c /var/spool/cvmfs/repo15.infn.it/rdonly -t /var/spool/cvmfs/repo15.infn.it\
/tmp -b 81e511020b1c8ec759f650a2194bed9a8ce74daf -r gw,/srv/cvmfs/repo15.infn.it/data/txn,http://cvmfs.wp6.cloud.infn.it:4929/api/v1 -w https://rgw.cloud.infn.it:443/cvmfs/re\
po15.infn.it -o /var/spool/cvmfs/repo15.infn.it/tmp/manifest -e sha1 -Z default -C /etc/cvmfs/repositories.d/repo15.infn.it/trusted_certs -N repo15.infn.it -K /etc/cvmfs/keys\
/repo15.infn.it.pub -L -D generic-2024-07-04T10:16:51Z -H /etc/cvmfs/keys/repo15.infn.it.gw -P /var/spool/cvmfs/repo15.infn.it/session_token -f overlayfs -p -l 4194304 -a 838\
8608 -h 16777216 -i
ubuntu@cvmfs-publisher:~$

So it complains about missing reflog, but the reflog file actually exists in the S3 backend and it is accessible e.g. via a curl.
But it looks like there is a problem with this file:

ubuntu@cvmfs-publisher:~$ sudo cvmfs_server check repo15.infn.it
Verifying integrity of repo15.infn.it...
Inspecting log of references
The .cvmfsreflog has unexpected content hash 487d675d36da9f366f74497c321bdd9f22216ca7 (expected d0b8a8d831e27989edde05640be95fc1580ee2bc)
failed to verify reflog
ubuntu@cvmfs-publisher:~$

On the stratum 0 node I tried to repair the .cvmfsreflog and apparently it worked:

[root@cvmfs-s0-s3cloudveneto cvmfs]# cvmfs_server check -r repo15.infn.it
Verifying integrity of repo15.infn.it...
Info: found https://rgw.cloud.infn.it:443/cvmfs/repo15.infn.it/.cvmfsreflog with content hash 487d675d36da9f366f74497c321bdd9f22216ca7
Warning: restored reflog checksum as 487d675d36da9f366f74497c321bdd9f22216ca7 (was: d0b8a8d831e27989edde05640be95fc1580ee2bc)
Inspecting log of references
Inspecting tag database
[inspecting catalog] 81e511020b1c8ec759f650a2194bed9a8ce74daf at /
no problems found
[root@cvmfs-s0-s3cloudveneto cvmfs]# 
[root@cvmfs-s0-s3cloudveneto cvmfs]# 
[root@cvmfs-s0-s3cloudveneto cvmfs]# cvmfs_server check repo15.infn.it
Verifying integrity of repo15.infn.it...
Inspecting log of references
Inspecting tag database
[inspecting catalog] 81e511020b1c8ec759f650a2194bed9a8ce74daf at /
no problems found
[root@cvmfs-s0-s3cloudveneto cvmfs]#

But on the publisher I still see the problem:

ubuntu@cvmfs-publisher:~$ sudo cvmfs_server check repo15.infn.it
/cvmfs/repo15.infn.it is not mounted properly.
Note: Trying to mount /cvmfs/repo15.infn.it... success
Verifying integrity of repo15.infn.it...
Inspecting log of references
The .cvmfsreflog has unexpected content hash 487d675d36da9f366f74497c321bdd9f22216ca7 (expected d0b8a8d831e27989edde05640be95fc1580ee2bc)
failed to verify reflog

and if I try the repair also on the publisher node:

ubuntu@cvmfs-publisher:~$ sudo cvmfs_server check -r repo15.infn.it
Verifying integrity of repo15.infn.it...
Failed to configure gateway uploader. Missing session token file.

terminate called after throwing an instance of 'ECvmfsException'
  what():  PANIC: /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/ubuntu2204/build/wd_src/cvmfs/upload_gateway.cc : 65
Error in parsing the spooler definition
Aborted (core dumped)
Warning: a reflog hash is registered in the manifest, re-creating missing reflog
Failed to configure gateway uploader. Missing session token file.

terminate called after throwing an instance of 'ECvmfsException'
  what():  PANIC: /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/ubuntu2204/build/wd_src/cvmfs/upload_gateway.cc : 65
Error in parsing the spooler definition
Aborted (core dumped)
Inspecting log of references
The .cvmfsreflog has unexpected content hash 487d675d36da9f366f74497c321bdd9f22216ca7 (expected d0b8a8d831e27989edde05640be95fc1580ee2bc)
failed to verify reflog
ubuntu@cvmfs-publisher:~$

Any hints ?
Shall I recreate the repo in the publisher node ?

Thanks, Massimo

[vavolkl]

Hi Massimo,

it shouldn’t be necessary to recreate the repo. You could try to just fix the reflog checksum with something like:

curl https://rgw.cloud.infn.it:443/cvmfs/repo15.infn.it/.cvmfsreflog > ~/reflog
$ cvmfs_publish hash -a sha1 < ~/reflog > /var/spool/cvmfs/repo15.infn.it/reflog.chksum

Let me know if that works!
Cheers,
Valentin

[sgaravat]

It worked !
Thanks a lot !!
Cheers, Massimo

[dwd]

More generally you should be able to use cvmfs_server check -r <reponame> to repair reflog issues. It often repairs it immediately and then goes on to do other checks and you can safely abort those if you’re in a hurry.

Oh, I see you tried that and it didn’t work. So Valentin, shouldn’t there be an effort to make it work in Massimo’s situation?