This seems to be a relatively new (and errant) error message seen frequently during long running transactions: CernVM-FS error: Unable to find the configuration file server.conf for the cvmfs publisher: ${repo}
This string comes from cvmfs_publisher [1], but I’m not sure why it’s triggered, as the configuration file certainly exists, and the transaction completes successfully. But it’s worrisome enough that repo maintainers ask me about it when they see it.
I wonder what could trigger that error message during a transaction. It should be displayed, if at all, only at the beginning. Is this coming from a local publisher or do you use the gateway services for publishing? Does the error message appear when a second transaction is started while another transaction is still active?
I wonder what could trigger that error message during a transaction. It should be displayed, if at all, only at the beginning.
It does seem to be displayed somehow at the beginning of long transactions – and then the publisher eventually “overcomes” the missing file state, finds it, and continues the transaction through to completion.
Is this coming from a local publisher or do you use the gateway services for publishing?
We had written a custom set of publication scripts (before the current gateway services existed) that begin a transaction, rsync changed data from a staging area, and publish the transaction. When it occurs, it does so after the transaction is initiated, e.g.:
Running "/usr/bin/cvmfs_server transaction repo.example.com":
CernVM-FS error: Unable to find the configuration file `server.conf` for the cvmfs publisher: repo.example.com
Does the error message appear when a second transaction is started while another transaction is still active?
From what I have seen during production runs and my own tests, no: a second transaction correctly aborts with the appropriate notification.
I’m still a bit puzzled, so let me try to summarize
This should mean that the cvmfs_server transaction command fails to identify the file /etc/cvmfs/repositories.d/repo.example.com/server.conf. It looks for a regular file with this name. Could it be that the file is a symlink?
In any case, as a result the transaction command fails with return code 1 and won’t create the transaction marker, neither would it open the /cvmfs/repo.example.conf repository for read/write access. Unless there is a retry for the transaction command, I don’t see how anything could be subsequently written to the repository or how the final cvmfs_server publish command can succeed.
Would it be possible to share the full publication script?
The repo configs are not symlinks, and not moving nor disappearing, which leaves me puzzled as well.
The publication triggers and scripts are highly customized, possibly proprietary, and maybe some embarrassing code, so I’ll share them with you privately, but I hope we’ll eventually have results or improvements to share.
Thanks for making the script available to me! I don’t see anything that can explain the behavior from there.
One thing to try is exporting CVMFS_SERVER_DEBUG=3 before running the cvmfs_server command. That prints out more debug information and may help finding the cause of the issue.
This was closed but the issue persists, even in the most recent version
# rpm -qa --last|grep cvmfs
cvmfs-2.11.2-1.el7.x86_64 Tue 14 Nov 2023 08:52:41 AM EST
cvmfs-server-2.11.2-1.el7.x86_64 Tue 14 Nov 2023 08:52:38 AM EST
cvmfs-libs-2.11.2-1.el7.x86_64 Tue 14 Nov 2023 08:52:33 AM EST
cvmfs-config-default-2.0-1.noarch Fri 09 Oct 2020 05:07:14 PM EDT
cvmfs-release-2-6.noarch Thu 30 Mar 2017 11:47:15 AM EDT
The error seems to be raised whenever a repo transaction is active but left open for an extended period of time – I am not sure what that period or time-out is, but some of these transactions involve syncing quite a bit of data to be published and can take a few hours. Perhaps a function times out and tries to re-configure itself during the transaction?