Cannot load manifest

Publisher / Stratum 0
1

Hi,

how do I troubleshoot/fix this error?

[root @ stratum-0 ] # cvmfs_server transaction my.cvmfs.repository
cannot load manifest

The file /srv/cvmfs/my.cvmfs.repository/.cvmfspublished seems to be OK.

Any suggestion is more than welcome.

Cheers,
Jose

2

Hi Jose, as with any error with cvmfs_server, please set CVMFS_SERVER_DEBUG=3. Next suggestion would be to check if you can curl .cvmfspublished from the CVMFS_STRATUM0 url , something like http://localhost/cvmfs/my.cvmfs.repository/.cvmfspublished ( check /etc/cvmfs/repositories.d/…/server.conf for the url) and if the whitelist needs to be resigned.
Hope this helps!
Valentin

3

Hi Valentin,

Before asking here, I resigned the whitelist and I indeed verified that I can wget the file.
I have just tried with DEBUG level, but no help:

[root@stratum-0 ~]# CVMFS_SERVER_DEBUG=3; cvmfs_server transaction my.cvmfs.repository
cannot load manifest

Cheers,
Jose

4

Oh. This is new (to me):

cvmfs_server check
...
failed to fetch repository manifest (7 - bad signature, failed to verify repository manifest)
...

I don’t think I have seen bad signature before…

5

Sometimes it is useful to run cvmfs_server with bash -x to see exactly what command is failing and try running that independently.

However in this case since you know it is complaining about a bad signature and had just done a resign, I would suspect a mismatch between the private and public keys for the repository.

Dave

6

interesting. Let me check that

7

I have to admit I am struggling a little bit to find out the exact command to check the keys.
Googling it, I get many disparate suggestions, using ssh-keygen, multiple different openssl commands, etc.
Tried all of that with a pair that it supposed to work, but none of the combinations match. So hard to tell the difference between a matching key and a broken one.

Which commands should I use?

8

The easiest way is to look at timestamps and be aware of any manipulation that might have been done to the repository’s keys. In this case it is saying the manifest has a problem so that’s the .cvmfspublished file and the .key and .crt files in /etc/cvmfs/keys.

Oh, another thing that might help is to do cvmfs_server resign -p <repo> because that re-creates the signature on the .cvmfspublished file using the current .key file.