Job for var-spool-cvmfs-...-rdonly.mount failed

Hello,
We have a standalone cvmfs which was running fine for months. But after a powercut the filesystem was corrupted.
So I’ve completely removed the filesystem and recreated it ( cvmfs_server rmfs/cvmfs_server mkfs).
But at some point now when I’m adding the data on a fresh filesystem, during a transaction it fails, and I’m not able to do anything abort or publish.
So, I’ve also tried to split the data into different transactions/publish. but at one point it also fails.

How can I at least abort the last transaction to restart the last sync ?

cvmfs-libs-2.11.2-1.el8.x86_64
cvmfs-config-default-2.1-1.noarch
cvmfs-2.11.2-1.el8.x86_64
cvmfs-server-2.11.2-1.el8.x86_64

Thank you in advance.

Can you also post the syslog (messages mentioning cvmfs from /var/log/messages or journalctl)? The mountpoint repair should leave some log messages there.

Thank you for the attention.

[root@io-ls-cvmfs01:~]# systemctl status var-spool-cvmfs-imas.iter.org-rdonly.mount
● var-spool-cvmfs-imas.iter.org-rdonly.mount - /var/spool/cvmfs/imas.iter.org/rdonly
   Loaded: loaded (/etc/fstab; generated)
   Active: failed (Result: exit-code) since Thu 2024-04-04 10:51:48 CEST; 1h 27min ago
    Where: /var/spool/cvmfs/imas.iter.org/rdonly
     What: cvmfs2#imas.iter.org
     Docs: man:fstab(5)
           man:systemd-fstab-generator(8)

Apr 04 10:51:24 io-ls-cvmfs01.iter.org systemd[1]: Mounting /var/spool/cvmfs/imas.iter.org/rdonly...
Apr 04 10:51:24 io-ls-cvmfs01.iter.org mount[7868]: CernVM-FS: running with suid support
Apr 04 10:51:24 io-ls-cvmfs01.iter.org cvmfs2[7869]: (imas.iter.org) switching proxy from (none) to DIRECT (set proxies)
Apr 04 10:51:24 io-ls-cvmfs01.iter.org cvmfs2[7869]: (imas.iter.org) switching proxy from (none) to DIRECT (cloned)
Apr 04 10:51:48 io-ls-cvmfs01.iter.org mount[7868]: CernVM-FS: loading Fuse module... Failed to initialize root file catalog (16 - file catalog failure)
Apr 04 10:51:48 io-ls-cvmfs01.iter.org cvmfs2[7869]: (imas.iter.org) Failed to initialize root file catalog (16 - file catalog failure)
Apr 04 10:51:48 io-ls-cvmfs01.iter.org systemd[1]: var-spool-cvmfs-imas.iter.org-rdonly.mount: Mount process exited, code=exited status=16
Apr 04 10:51:48 io-ls-cvmfs01.iter.org systemd[1]: var-spool-cvmfs-imas.iter.org-rdonly.mount: Failed with result 'exit-code'.
Apr 04 10:51:48 io-ls-cvmfs01.iter.org systemd[1]: Failed to mount /var/spool/cvmfs/imas.iter.org/rdonly.
-- Unit var-spool-cvmfs-imas.iter.org-rdonly.mount has begun starting up.
Apr 04 12:21:07 io-ls-cvmfs01.iter.org mount[8845]: CernVM-FS: running with suid support
Apr 04 12:21:07 io-ls-cvmfs01.iter.org cvmfs2[8846]: (imas.iter.org) switching proxy from (none) to DIRECT (set proxies)
Apr 04 12:21:07 io-ls-cvmfs01.iter.org cvmfs2[8846]: (imas.iter.org) switching proxy from (none) to DIRECT (cloned)
Apr 04 12:21:31 io-ls-cvmfs01.iter.org mount[8845]: CernVM-FS: loading Fuse module... Failed to initialize root file catalog (16 - file catalog failure)
Apr 04 12:21:31 io-ls-cvmfs01.iter.org cvmfs2[8846]: (imas.iter.org) Failed to initialize root file catalog (16 - file catalog failure)
Apr 04 12:21:31 io-ls-cvmfs01.iter.org systemd[1]: var-spool-cvmfs-imas.iter.org-rdonly.mount: Mount process exited, code=exited status=16
Apr 04 12:21:31 io-ls-cvmfs01.iter.org systemd[1]: var-spool-cvmfs-imas.iter.org-rdonly.mount: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- The unit var-spool-cvmfs-imas.iter.org-rdonly.mount has entered the 'failed' state with result 'exit-code'.
Apr 04 12:21:31 io-ls-cvmfs01.iter.org systemd[1]: Failed to mount /var/spool/cvmfs/imas.iter.org/rdonly.
-- Subject: Unit var-spool-cvmfs-imas.iter.org-rdonly.mount has failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit var-spool-cvmfs-imas.iter.org-rdonly.mount has failed.
--
-- The result is failed.

Apr  4 12:21:07 io-ls-cvmfs01 systemd[1]: Mounting /var/spool/cvmfs/imas.iter.org/rdonly...
Apr  4 12:21:07 io-ls-cvmfs01 mount[8845]: CernVM-FS: running with suid support
Apr  4 12:21:07 io-ls-cvmfs01 cvmfs2[8846]: (imas.iter.org) switching proxy from (none) to DIRECT (set proxies)
Apr  4 12:21:07 io-ls-cvmfs01 cvmfs2[8846]: (imas.iter.org) switching proxy from (none) to DIRECT (cloned)
Apr  4 12:21:31 io-ls-cvmfs01 mount[8845]: CernVM-FS: loading Fuse module... Failed to initialize root file catalog (16 - file catalog failure)
Apr  4 12:21:31 io-ls-cvmfs01 cvmfs2[8846]: (imas.iter.org) Failed to initialize root file catalog (16 - file catalog failure)
Apr  4 12:21:31 io-ls-cvmfs01 systemd[1]: var-spool-cvmfs-imas.iter.org-rdonly.mount: Mount process exited, code=exited status=16
Apr  4 12:21:31 io-ls-cvmfs01 systemd[1]: var-spool-cvmfs-imas.iter.org-rdonly.mount: Failed with result 'exit-code'.
Apr  4 12:21:31 io-ls-cvmfs01 systemd[1]: Failed to mount /var/spool/cvmfs/imas.iter.org/rdonly.

Sounds like there is still some corruption in the repository. What does

cvmfs_server check -r imas.iter.org

give?

Here it is:

# cvmfs_server check -r imas.iter.org
/var/spool/cvmfs/imas.iter.org/rdonly is not mounted properly.
/cvmfs/imas.iter.org is not mounted properly.
imas.iter.org is in a transaction but /cvmfs/imas.iter.org is not mounted read/write
Repository imas.iter.org is in a transaction and cannot be repaired.
--> Run `cvmfs_server abort imas.iter.org` to revert and repair.

Tried:

# cvmfs_server abort imas.iter.org
You are about to DISCARD ALL CHANGES OF THE CURRENT TRANSACTION for imas.iter.org!  Are you sure (y/N)? y
/var/spool/cvmfs/imas.iter.org/rdonly is not mounted properly
/cvmfs/imas.iter.org is not mounted properly
Job for var-spool-cvmfs-imas.iter.org-rdonly.mount failed.
See "systemctl status var-spool-cvmfs-imas.iter.org-rdonly.mount" and "journalctl -xe" for details.
Trying to mount /var/spool/cvmfs/imas.iter.org/rdonly... fail
Trying to mount /var/spool/cvmfs/imas.iter.org/rdonly... fail

Meanwhile I did a reboot of the server, just in case of, but same.

Hmmm, do you still have a client that reads from this repository? Can you mount it from another machine?

Another thing to try is to remove the file

rm /var/spool/cvmfs/test.cern.ch/client.local

and try to abort again.

It would also be helpful to get more details about the error

You should be able to do that by setting

CVMFS_DEBUGLOG=/tmp/imas.iter.org.debuglog

in /etc/cvmfs/repositories.d/imas.iter.org/client.conf, adding debug to the comma-separated list of mount options for your mountpoint in /etc/fstab, and doing systemctl daemon-reload before the mount attempt. Then look in the debuglog for that error and I think it should give you a more specific error.

Don’t forget to undo the changes for debugging once the problem is fixed.

Hi,
I tried different things and same.
To not use your time and mine, I think it will be faster to recreate the FS and try to push the data again during the week-end.
Do you have some ideas on how we can avoid this problem?

OK - Let me know how it goes. Possibly using an external s3 object store for data is more resilient. We can try to simulate various disk failures and see if there are possible improvements, but it won’t be a short-term project.

Hello,
I’ve tryed with the autocatalog enabled this time, instead of the file,
all went well until the last publish: