Invalid whitelist

Stratum 1
1

We have the following error on several Stratum 1:

$ CVMFS_SERVER_DEBUG=3 cvmfs_server info euclid-dev.in2p3.fr(cvmfs) Parsing config file /etc/cvmfs/server.local    [10-17-2023 10:48:48 CEST]
(cvmfs) execve'd /bin/sh (PID: 28466)    [10-17-2023 10:48:48 CEST]
(cvmfs) Parsing config file /etc/cvmfs/repositories.d/euclid-dev.in2p3.fr/server.conf    [10-17-2023 10:48:48 CEST]
(cvmfs) execve'd /bin/sh (PID: 28468)    [10-17-2023 10:48:48 CEST]
(cvmfs) Parsing config file /etc/cvmfs/repositories.d/euclid-dev.in2p3.fr/replica.conf    [10-17-2023 10:48:48 CEST]
(cvmfs) execve'd /bin/sh (PID: 28470)    [10-17-2023 10:48:49 CEST]
(download) escaped http://localhost/cvmfs/euclid-dev.in2p3.fr/.cvmfswhitelist to http://localhost/cvmfs/euclid-dev.in2p3.fr/.cvmfswhitelist    [10-17-2023 10:48:49 CEST]
(download) Verify downloaded url http://localhost/cvmfs/euclid-dev.in2p3.fr/.cvmfswhitelist, proxy DIRECT (curl error 0)    [10-17-2023 10:48:49 CEST]
(signature) whitelist UTC expiry timestamp in localtime: 16 Nov 2023 02:00:00    [10-17-2023 10:48:49 CEST]
(signature) local time: 17 Oct 2023 10:48:49    [10-17-2023 10:48:49 CEST]
(signature) VerifyRsa, no public key fits    [10-17-2023 10:48:49 CEST]
(cvmfs) failed to verify repository whitelist    [10-17-2023 10:48:49 CEST]
(unexpected termination) cannot load whitelist [invalid whitelist signature]

Stacktrace:
/lib64/libcvmfs_server_debug.so.2.10.1(+0x1099ac) [0x7f9368a229ac]
/lib64/libcvmfs_server_debug.so.2.10.1(+0x114b30) [0x7f9368a2db30]
/lib64/libcvmfs_server_debug.so.2.10.1(_ZN7publish10Repository19DownloadRootObjectsERKSsS2_S2_+0x176) [0x7f9368a24038]
/lib64/libcvmfs_server_debug.so.2.10.1(_ZN7publish10RepositoryC1ERKNS_18SettingsRepositoryEb+0x6b7) [0x7f9368a23775]
/usr/bin/cvmfs_publish_debug() [0x47b19b]
/usr/bin/cvmfs_publish_debug() [0x489d57]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f9366c46555]
/usr/bin/cvmfs_publish_debug() [0x42ad87]

Replication seems to work fine. I thought it could be related to the bug CVM-2005 but we are using CVMFS_PUBLIC_KEY=/etc/cvmfs/keys/in2p3.fr/euclid-dev.in2p3.fr.pub so this should not apply. Stratum 1 are running on CentOS 7.9 and using cvmfs 2.10.1 and 2.11.0.

The Stratum 0 has been migrated recently to a new server but we checked that the keys are identical. Is there anything else that we could have missed? Thanks.

2

I add a little information.

Stratum0 is running under Debian 11 with cvmfs 2.11.0

We have also another Stratum1 under Debian 11 (with cvmfs 2.11.0 as well).
The command cvmfs_server info works well on this stratum1.

Here is the part corresponding to the part where it stopped on CentOS stratums :


[skip]
(signature) whitelist UTC expiry timestamp in localtime: 16 Nov 2023 02:00:00    [10-17-2023 11:06:01 CEST]
(signature) local time: 17 Oct 2023 11:06:01    [10-17-2023 11:06:01 CEST]
(download) escaped http://localhost/cvmfs/euclid-dev.in2p3.fr/.cvmfspublished to http://localhost/cvmfs/euclid-dev.in2p3.fr/.cvmfspublished    [10-17-2023 11:06:01 CEST]
(curl) {header/out} GET /cvmfs/euclid-dev.in2p3.fr/.cvmfspublished HTTP/1.1(signature) whitelist UTC expiry timestamp in localtime: 16 Nov 2023 02:00:00    [10-17-2023 11:06:01 CEST]
(signature) local time: 17 Oct 2023 11:06:01    [10-17-2023 11:06:01 CEST]
(download) escaped http://localhost/cvmfs/euclid-dev.in2p3.fr/.cvmfspublished to http://localhost/cvmfs/euclid-dev.in2p3.fr/.cvmfspublished    [10-17-2023 11:06:01 CEST]
(curl) {header/out} GET /cvmfs/euclid-dev.in2p3.fr/.cvmfspublished HTTP/1.1
[skip a lot of curl stuff]

I could paste the complete output if needed.

3

That certainly does look as if the public key defined does not match the key used to create the signature. Do verify that the key file pointed to by CVMFS_PUBLIC_KEY in /etc/cvmfs/repositories.d/euclid-dev.in2p3.fr/replica.conf does indeed exist, is world-readable, and is the key used to sign the whitelist. If that all looks correct, you could try cvmfs_config resign euclid-dev.in2p3.fr to see if that helps.