Hello,
We have a standalone cvmfs which was running fine for months. But after a powercut the filesystem was corrupted.
So I’ve completely removed the filesystem and recreated it ( cvmfs_server rmfs/cvmfs_server mkfs).
But at some point now when I’m adding the data on a fresh filesystem, during a transaction it fails, and I’m not able to do anything abort or publish.
So, I’ve also tried to split the data into different transactions/publish. but at one point it also fails.
Can you also post the syslog (messages mentioning cvmfs from /var/log/messages or journalctl)? The mountpoint repair should leave some log messages there.
[root@io-ls-cvmfs01:~]# systemctl status var-spool-cvmfs-imas.iter.org-rdonly.mount
● var-spool-cvmfs-imas.iter.org-rdonly.mount - /var/spool/cvmfs/imas.iter.org/rdonly
Loaded: loaded (/etc/fstab; generated)
Active: failed (Result: exit-code) since Thu 2024-04-04 10:51:48 CEST; 1h 27min ago
Where: /var/spool/cvmfs/imas.iter.org/rdonly
What: cvmfs2#imas.iter.org
Docs: man:fstab(5)
man:systemd-fstab-generator(8)
Apr 04 10:51:24 io-ls-cvmfs01.iter.org systemd[1]: Mounting /var/spool/cvmfs/imas.iter.org/rdonly...
Apr 04 10:51:24 io-ls-cvmfs01.iter.org mount[7868]: CernVM-FS: running with suid support
Apr 04 10:51:24 io-ls-cvmfs01.iter.org cvmfs2[7869]: (imas.iter.org) switching proxy from (none) to DIRECT (set proxies)
Apr 04 10:51:24 io-ls-cvmfs01.iter.org cvmfs2[7869]: (imas.iter.org) switching proxy from (none) to DIRECT (cloned)
Apr 04 10:51:48 io-ls-cvmfs01.iter.org mount[7868]: CernVM-FS: loading Fuse module... Failed to initialize root file catalog (16 - file catalog failure)
Apr 04 10:51:48 io-ls-cvmfs01.iter.org cvmfs2[7869]: (imas.iter.org) Failed to initialize root file catalog (16 - file catalog failure)
Apr 04 10:51:48 io-ls-cvmfs01.iter.org systemd[1]: var-spool-cvmfs-imas.iter.org-rdonly.mount: Mount process exited, code=exited status=16
Apr 04 10:51:48 io-ls-cvmfs01.iter.org systemd[1]: var-spool-cvmfs-imas.iter.org-rdonly.mount: Failed with result 'exit-code'.
Apr 04 10:51:48 io-ls-cvmfs01.iter.org systemd[1]: Failed to mount /var/spool/cvmfs/imas.iter.org/rdonly.
-- Unit var-spool-cvmfs-imas.iter.org-rdonly.mount has begun starting up.
Apr 04 12:21:07 io-ls-cvmfs01.iter.org mount[8845]: CernVM-FS: running with suid support
Apr 04 12:21:07 io-ls-cvmfs01.iter.org cvmfs2[8846]: (imas.iter.org) switching proxy from (none) to DIRECT (set proxies)
Apr 04 12:21:07 io-ls-cvmfs01.iter.org cvmfs2[8846]: (imas.iter.org) switching proxy from (none) to DIRECT (cloned)
Apr 04 12:21:31 io-ls-cvmfs01.iter.org mount[8845]: CernVM-FS: loading Fuse module... Failed to initialize root file catalog (16 - file catalog failure)
Apr 04 12:21:31 io-ls-cvmfs01.iter.org cvmfs2[8846]: (imas.iter.org) Failed to initialize root file catalog (16 - file catalog failure)
Apr 04 12:21:31 io-ls-cvmfs01.iter.org systemd[1]: var-spool-cvmfs-imas.iter.org-rdonly.mount: Mount process exited, code=exited status=16
Apr 04 12:21:31 io-ls-cvmfs01.iter.org systemd[1]: var-spool-cvmfs-imas.iter.org-rdonly.mount: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- The unit var-spool-cvmfs-imas.iter.org-rdonly.mount has entered the 'failed' state with result 'exit-code'.
Apr 04 12:21:31 io-ls-cvmfs01.iter.org systemd[1]: Failed to mount /var/spool/cvmfs/imas.iter.org/rdonly.
-- Subject: Unit var-spool-cvmfs-imas.iter.org-rdonly.mount has failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit var-spool-cvmfs-imas.iter.org-rdonly.mount has failed.
--
-- The result is failed.
# cvmfs_server check -r imas.iter.org
/var/spool/cvmfs/imas.iter.org/rdonly is not mounted properly.
/cvmfs/imas.iter.org is not mounted properly.
imas.iter.org is in a transaction but /cvmfs/imas.iter.org is not mounted read/write
Repository imas.iter.org is in a transaction and cannot be repaired.
--> Run `cvmfs_server abort imas.iter.org` to revert and repair.
Tried:
# cvmfs_server abort imas.iter.org
You are about to DISCARD ALL CHANGES OF THE CURRENT TRANSACTION for imas.iter.org! Are you sure (y/N)? y
/var/spool/cvmfs/imas.iter.org/rdonly is not mounted properly
/cvmfs/imas.iter.org is not mounted properly
Job for var-spool-cvmfs-imas.iter.org-rdonly.mount failed.
See "systemctl status var-spool-cvmfs-imas.iter.org-rdonly.mount" and "journalctl -xe" for details.
Trying to mount /var/spool/cvmfs/imas.iter.org/rdonly... fail
Trying to mount /var/spool/cvmfs/imas.iter.org/rdonly... fail
Meanwhile I did a reboot of the server, just in case of, but same.
It would also be helpful to get more details about the error
You should be able to do that by setting
CVMFS_DEBUGLOG=/tmp/imas.iter.org.debuglog
in /etc/cvmfs/repositories.d/imas.iter.org/client.conf, adding debug to the comma-separated list of mount options for your mountpoint in /etc/fstab, and doing systemctl daemon-reload before the mount attempt. Then look in the debuglog for that error and I think it should give you a more specific error.
Don’t forget to undo the changes for debugging once the problem is fixed.
Hi,
I tried different things and same.
To not use your time and mine, I think it will be faster to recreate the FS and try to push the data again during the week-end.
Do you have some ideas on how we can avoid this problem?
OK - Let me know how it goes. Possibly using an external s3 object store for data is more resilient. We can try to simulate various disk failures and see if there are possible improvements, but it won’t be a short-term project.
