We are experiencing problems with our publishers, the transactions run without errors but the changes do not appear on the stratum0.
For example :
cvmfs_server transaction repo
Gateway reply: ok
cvmfs_server publish repo
Using auto tag 'generic-2023-01-06T13:46:04Z'
Waiting for upload of files before committing...
Committing file catalogs...
WARNING: catalog at / has more than 200000 entries (525665). Large catalogs stress the CernVM-FS transport infrastructure. Please split it into nested catalogs or increase the limit.
Wait for all uploads to finish
Exporting repository manifest
Statistics stored at: /var/spool/cvmfs/repo/stats.db
Changes submitted to repository gateway
On the stratum0 :
# ls /cvmfs/repo/ | grep foo.txt
How to debug the problem?
Is the stratum 0 machine different from the publisher? Can you describe your setup better?
As a general rule, you may want to check if the Stratum 0 is locked to the last repository version with
cvmfs_server tag -l. You can also umount/remount the repository and check if that helps.
Stratum0 and gateway are the same machine. We have 2 publisher nodes on two different machines for the stratum0. After a successfull transaction we have to umount/remount to access new files on stm0.
That is expected: Considering your Stratum 0 client is not on a publisher box, it does not automatically switch to a new version of the repository when something is published by the publishers nodes. A simple remount should fix the issue, with the client locking to the latest repository version.
Ok thank you for your response. Is it the same behavior on the publisher side? (remount to see new version)
In a setup with a single publisher (which would be in practice the Stratum 0), changes would be immediately visible when the publication is over and the filesystem is remounted RO.
In the case of multiple publishers, changes will be immediately visible to the publisher node applying them; other publishers will have to remount. A remount is automatically done when opening a transaction in case the publisher realizes it is not using the latest repository revision (i.e., another publisher published something)
Thank you very much for your clarification, it is clearer now.