Failed to fetch repository manifest (2 - incomplete manifest) : due to a disk corruption

Publisher / Stratum 0
1

We have a very critical issue on our production CVMFS server.

Due to some unattended maintenance the server had some issues and the filesystem contained cvmfs got corrupted.

Normally (I was not here at the moment) the filesystem has been repaired. At least it seems to work well.

But the cvmfs still has the issues.

We have two repositories.
One of them seems to work but probably not well (but we could discuss it later).

Another repository is shown in “cvmfs_server list”

> cvmfs_server list
repo-dev (stratum0 / local)
repo (stratum0 / local)

but even the “tag -l” doesn’t work.

> cvmfs_server tag -l repo.in2p3.fr
failed to fetch repository manifest (2 - incomplete manifest)
failed to load manifest file
failed to init environment

Here is the output with debug information :

> CVMFS_SERVER_DEBUG=3 cvmfs_server tag -l repo
(download) (id 0) escaped http://localhost/cvmfs/repo/.cvmfspublished to http://localhost/cvmfs/repo/.cvmfspublished    [07-18-2026 09:36:47 CEST]
(curl) (id 0) {header/sent} GET /cvmfs/repo/.cvmfspublished HTTP/1.1
Host: localhost
Accept: /
Connection: Keep-Alive
User-Agent: cvmfs Fuse 2.13.3    [07-18-2026 09:36:47 CEST]
(curl) (id 0) {header/recv} HTTP/1.1 200 OK    [07-18-2026 09:36:47 CEST]
(curl) (id 0) {header/recv} Date: Sat, 18 Jul 2026 07:36:47 GMT    [07-18-2026 09:36:47 CEST]
(curl) (id 0) {header/recv} Server: Apache/2.4.67 (Debian)    [07-18-2026 09:36:47 CEST]
(curl) (id 0) {header/recv} Accept-Ranges: bytes    [07-18-2026 09:36:47 CEST]
(curl) (id 0) {header/recv} Content-Length: 601    [07-18-2026 09:36:47 CEST]
(curl) (id 0) {header/recv} Cache-Control: max-age=61    [07-18-2026 09:36:47 CEST]
(curl) (id 0) {header/recv} Expires: Sat, 18 Jul 2026 07:37:48 GMT    [07-18-2026 09:36:47 CEST]
(curl) (id 0) {header/recv} Keep-Alive: timeout=5, max=100    [07-18-2026 09:36:47 CEST]
(curl) (id 0) {header/recv} Connection: Keep-Alive    [07-18-2026 09:36:47 CEST]
(curl) (id 0) {header/recv} Content-Type: application/x-cvmfs    [07-18-2026 09:36:47 CEST]
(curl) (id 0) {header/recv}     [07-18-2026 09:36:47 CEST]
(curl) (id 0) {data/recv}     [07-18-2026 09:36:47 CEST]
(download) (manager ‘standard’ - id 0) Verify downloaded url http://localhost/cvmfs/repo/.cvmfspublished, proxy DIRECT (curl error 0)    [07-18-2026 09:36:47 CEST]
failed to fetch repository manifest (2 - incomplete manifest)
failed to load manifest file
failed to init environment

Do you have any idea what could we do to understand the problem and especially to fix it ?

Thank you for your help.

Thanks,
Paul Zakharov

P.S.: Just for information “repo” is very old repository and gc is not activated on it.

2

Dear Paul,

most likely the repository can still be repaired. Can you attach the manifest file (.cvmfspublished , the location will probably be /srv/cvmfs/repo.in2p3.fr/.cvmfspublished , or you could curl it from http://localhost/cvmfs/repo/.cvmfspublished)

As a next step I would recommend to run cvmfs_server check -c, but probably the manifest file needs to be fixed by hand first.

Best,
Valentin

3

Dear Valentin,

At first, thank you very much for your reply.

I did not expect to receive a response during the weekend. I was away last week, discovered the issue yesterday, and decided to submit it immediately.
I am very touched by such a quick response.

I’ll send you .cvmfspublished for our repository in private message.
If it’s better to share it here I also could do it (I don’t realize is there are sensible information in .cvmfspublished).

Merci,
Paul Zakharov

4

I would first do cvmfs_server check -ic repo, replacing repo with the FQRN of your repo name to see if it will tell you exactly what files are having a problem, and then you can copy those files from a stratum 1.

If that doesn’t work or becomes too cumbersome, an alternative is to take a whole snapshot from a stratum 1. First, the stratum 1 needs to have a zero-length /srv/cvmfs/repo/.cvmfs_master_replica file so it can be considered a replica source. Then on the stratum 0 do cvmfs_server add-replica -n repo.new http://stratum1.url/cvmfs/repo /etc/cvmfs/keys/repo.pub, replacing repo with your repo name and stratum1.url with the FQDN of the stratum 1. Then you should be able to take a fresh snapshot with cvmfs_server snapshot repo.new. After that completes, unmount the two mountpoints related to your repo and swap the /srv/cvmfs/repo/data and /srv/cmfs/repo.new/data directories. After that try to normal stratum 0 operations (which should automatically re-mount it). If that’s fully successful you can clean up with cvmfs_server rmfs repo.new.

5

I would like to do a brief debrief to document the steps to resolve this issue.

The problem was solved mostly by exchange of personal messages with CVMFS Team.

And at first, I’d like to say thank you very much for CVMFS Team (@vavolkl , @dwd , @jblomer-social-test ). They helped us a lot in very hard emergency situation.

In summary, the issue was resolved by recreating the Stratum0 server from a copy of the Stratum1 server.
Our Stratum1 server was also corrupted, but we had external Stratum1 servers connected to our Stratum0 server.
The following steps were taken:

  1. A new server was created.
  2. Data from /srv/cvmfs was copied from the Stratum1 server to the new Stratum0 server.
  3. Once the copy was complete, the following actions were performed:
    a. The keychain was restored by copying the keys from the old Stratum0 server to the new one:
    cp <fqrn>.pub <fqrn>.key <fqrn>.crt <fqrn>.masterkey /etc/cvmfs/keys/
    b. The data was imported as Stratum0:
    cvmfs_server import -o cvmfsUser -g -m <fqrn>
    c. The whitelist was refreshed, verified, and replication was re-enabled:
    cvmfs_server resign <fqrn>
cvmfs_server check -r

We tested publish and it worked! :slight_smile:

Additionally, I would like to share some notes regarding minor details:

  • Normally, the command cvmfs_server import -o cvmfsUser assigns files to the cvmfsUser. However, it appears that it did not change the group. So may be it could be wise to run chown -R before executing cvmfs_server import.
  • During one of the imports, we encountered the message: “Updating global JSON information… fail”. From what I understand, this is not critical. It can be resolved by running:
    cvmfs_server update-info -p -e <fqrn>

I believe that covers everything.
Once again, thank you to the entire CVMFS Team for your support.