Dear all
I have the following deployment:
- CVMFS stratum 0 configured also as gw, hosting several repos
- Backend implemented through S3 (ceph-rgw)
- Publishing is node through a publisher node
I see problems publishing on one specific repo:
ubuntu@cvmfs-publisher:~$ sudo cvmfs_server transaction repo15.infn.it
Gateway reply: ok
ubuntu@cvmfs-publisher:~$ sudo cvmfs_server publish repo15.infn.it
Using auto tag 'generic-2024-07-04T10:16:51Z'
Processing changes...
Waiting for upload of files before committing...
Committing file catalogs...
Wait for all uploads to finish
Exporting repository manifest
Lease end request - error reply: {"reason":"missing_reflog","status":"error"}
SessionContext: could not commit session. Aborting.
[ERROR] Failed to commit transaction.
Statistics stored at: /var/spool/cvmfs/repo15.infn.it/stats.db
Synchronization failed
Executed Command:
cvmfs_swissknife sync -u /cvmfs/repo15.infn.it -s /var/spool/cvmfs/repo15.infn.it/scratch/current -c /var/spool/cvmfs/repo15.infn.it/rdonly -t /var/spool/cvmfs/repo15.infn.it\
/tmp -b 81e511020b1c8ec759f650a2194bed9a8ce74daf -r gw,/srv/cvmfs/repo15.infn.it/data/txn,http://cvmfs.wp6.cloud.infn.it:4929/api/v1 -w https://rgw.cloud.infn.it:443/cvmfs/re\
po15.infn.it -o /var/spool/cvmfs/repo15.infn.it/tmp/manifest -e sha1 -Z default -C /etc/cvmfs/repositories.d/repo15.infn.it/trusted_certs -N repo15.infn.it -K /etc/cvmfs/keys\
/repo15.infn.it.pub -L -D generic-2024-07-04T10:16:51Z -H /etc/cvmfs/keys/repo15.infn.it.gw -P /var/spool/cvmfs/repo15.infn.it/session_token -f overlayfs -p -l 4194304 -a 838\
8608 -h 16777216 -i
ubuntu@cvmfs-publisher:~$
So it complains about missing reflog, but the reflog file actually exists in the S3 backend and it is accessible e.g. via a curl.
But it looks like there is a problem with this file:
ubuntu@cvmfs-publisher:~$ sudo cvmfs_server check repo15.infn.it
Verifying integrity of repo15.infn.it...
Inspecting log of references
The .cvmfsreflog has unexpected content hash 487d675d36da9f366f74497c321bdd9f22216ca7 (expected d0b8a8d831e27989edde05640be95fc1580ee2bc)
failed to verify reflog
ubuntu@cvmfs-publisher:~$
On the stratum 0 node I tried to repair the .cvmfsreflog and apparently it worked:
[root@cvmfs-s0-s3cloudveneto cvmfs]# cvmfs_server check -r repo15.infn.it
Verifying integrity of repo15.infn.it...
Info: found https://rgw.cloud.infn.it:443/cvmfs/repo15.infn.it/.cvmfsreflog with content hash 487d675d36da9f366f74497c321bdd9f22216ca7
Warning: restored reflog checksum as 487d675d36da9f366f74497c321bdd9f22216ca7 (was: d0b8a8d831e27989edde05640be95fc1580ee2bc)
Inspecting log of references
Inspecting tag database
[inspecting catalog] 81e511020b1c8ec759f650a2194bed9a8ce74daf at /
no problems found
[root@cvmfs-s0-s3cloudveneto cvmfs]#
[root@cvmfs-s0-s3cloudveneto cvmfs]#
[root@cvmfs-s0-s3cloudveneto cvmfs]# cvmfs_server check repo15.infn.it
Verifying integrity of repo15.infn.it...
Inspecting log of references
Inspecting tag database
[inspecting catalog] 81e511020b1c8ec759f650a2194bed9a8ce74daf at /
no problems found
[root@cvmfs-s0-s3cloudveneto cvmfs]#
But on the publisher I still see the problem:
ubuntu@cvmfs-publisher:~$ sudo cvmfs_server check repo15.infn.it
/cvmfs/repo15.infn.it is not mounted properly.
Note: Trying to mount /cvmfs/repo15.infn.it... success
Verifying integrity of repo15.infn.it...
Inspecting log of references
The .cvmfsreflog has unexpected content hash 487d675d36da9f366f74497c321bdd9f22216ca7 (expected d0b8a8d831e27989edde05640be95fc1580ee2bc)
failed to verify reflog
and if I try the repair also on the publisher node:
ubuntu@cvmfs-publisher:~$ sudo cvmfs_server check -r repo15.infn.it
Verifying integrity of repo15.infn.it...
Failed to configure gateway uploader. Missing session token file.
terminate called after throwing an instance of 'ECvmfsException'
what(): PANIC: /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/ubuntu2204/build/wd_src/cvmfs/upload_gateway.cc : 65
Error in parsing the spooler definition
Aborted (core dumped)
Warning: a reflog hash is registered in the manifest, re-creating missing reflog
Failed to configure gateway uploader. Missing session token file.
terminate called after throwing an instance of 'ECvmfsException'
what(): PANIC: /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/ubuntu2204/build/wd_src/cvmfs/upload_gateway.cc : 65
Error in parsing the spooler definition
Aborted (core dumped)
Inspecting log of references
The .cvmfsreflog has unexpected content hash 487d675d36da9f366f74497c321bdd9f22216ca7 (expected d0b8a8d831e27989edde05640be95fc1580ee2bc)
failed to verify reflog
ubuntu@cvmfs-publisher:~$
Any hints ?
Shall I recreate the repo in the publisher node ?
Thanks, Massimo