We have an error on a publisher node when trying to publish a transaction :
cvmfs_server publish repo
Using auto tag 'generic-2023-02-20T12:24:03Z'
Processing changes...
..............................................................................................................................................................................................................................................................
Waiting for upload of files before committing...
Committing file catalogs...
Wait for all uploads to finish
Exporting repository manifest
Lease end request - error reply: {"reason":"could not obtain statistics counters: no statistics counters for lease repo/","status":"error"}
SessionContext: could not commit session. Aborting.
[ERROR] Failed to commit transaction.
Statistics stored at: /var/spool/cvmfs/repo/stats.db
Synchronization failed
Executed Command:
cvmfs_swissknife sync -u /cvmfs/repo -s /var/spool/cvmfs/repo/scratch/current -c /var/spool/cvmfs/repo/rdonly -t /var/spool/cvmfs/repo/tmp -b 080236f170f456c5d452fa2009b786cc80fc5d51 -r gw,/srv/cvmfs/repo/data/txn,http://stm0.in2p3.fr:4929/api/v1 -w http://stm0.in2p3.fr/cvmfs/repo -o /var/spool/cvmfs/repo/tmp/manifest -e sha1 -Z default -C /etc/cvmfs/repositories.d/repo/trusted_certs -N repo -K /etc/cvmfs/keys/repo.pub -L -D generic-2023-02-20T12:24:03Z -H /etc/cvmfs/keys/repo.gw -P /var/spool/cvmfs/repo/session_token -f overlayfs -p -l 4194304 -a 8388608 -h 16777216 -i
When trying to abort :
cvmfs_server abort repo
You are about to DISCARD ALL CHANGES OF THE CURRENT TRANSACTION for repo! Are you sure (y/N)? y
Error: 'no statistics counters for lease repo/'
Publisher node and stratum 0 node are two separated server with cvmfs 2.10 : have you some ideas to help us ?
Hi,
Just a bit more of context, as our production platform is blocked due to this issue, we don’t have much time before deciding to do a full rollback.
This happen while trying to activate cvfsm gateway functionality. This errors pop only on one out of the 2 CVMFS repos we have. The other is working fine.
We see two errors in the logs that we are unable to debug:
1/ HTTP 404 on .cvmfsreflog : the file is indeed missing on the filesystem (while it is present on the repository that is working)
2/ (sql) opening database file /var/spool/cvmfs/repo/tmp/tags.gHB8Jw [02-20-2023 13:55:10 CET]
(sql) failed to read-ahead /var/spool/cvmfs/repo/tmp/tags.gHB8Jw (22) [02-20-2023 13:55:10 CET]
No idea what this means
Are all software components (cvmfs-server, cvmfs-gateway, cvmfs) on publisher & gateway updated to 2.10? Unfortunately, there has been a bug in the 2.10 release that made it necessary to update the components in lockstep.
So do I understand correctly that at the moment you can neither publish nor abort the transaction?
You can try a manual reset of the gateway and the publisher. On the publisher, remove /var/spool/cvmfs/$reponame/in_transaction and /var/spool/cvmfs/$reponame/session_token and everything under /var/spool/cvmfs/$reponame/scratch/current/.
On the gateway, stop the cvmfs gateway service and remove the lease database in /var/lib/cvmfs-gateway/gw.db (Note: if the gateway is used for multiple repositories, this will remove the lease database for all of them).
Then restart the gateway and try to publishing again.
I see, so actually the missing reflog is a fatal error in this case. So the original repository must have been created a while ago? Reflogs were added in version 2.3 and, unfortunately, there is currently no good command support to update older repositories to the gateway deployment.
A reflog is automatically created during garbage collection, but I assume garbage collection is not used with this repository.
We can create the reflog from the existing repository using the (internal) cvmfs_swissknife command on the gateway. To use it with the correct options, we’d need to know the repository configuration (/etc/cvmfs/repositories.d/$reponame/server.conf, you can send it in a private message if you prefer).
How complicated is your rollback procedure? It may be a good idea not to recreate the reflog under time pressure today.
Apologies, I missed that message. In general, it is easier to handle delayed follow-ups with a new topic.
With the just released cvmfs 2.10.1 we have added some fixes to bring back the repository in a working state. You can try (after updating) to use cvmfs_server abort -f.