# Job for var-spool-cvmfs-...-rdonly.mount failed **URL:** \ **Category:** Publisher / Stratum 0\ **Created:** [April 4, 2024, 9:36am UTC](https://cernvm-forum.cern.ch/t/job-for-var-spool-cvmfs-rdonly-mount-failed/422 "2024-04-04T09:36:11Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![fred\_iter](https://cernvm-forum.cern.ch/letter_avatar_proxy/v4/letter/f/e0b2c6/32.png) [@fred\_iter](https://cernvm-forum.cern.ch/u/fred_iter)\ **Post date:** [April 4, 2024, 9:36am UTC](https://cernvm-forum.cern.ch/t/job-for-var-spool-cvmfs-rdonly-mount-failed/422/1 "2024-04-04T09:36:12Z")
Hello, We have a standalone cvmfs which was running fine for months. But after a powercut the filesystem was corrupted. So I’ve completely removed the filesystem and recreated it ( cvmfs\_server rmfs/cvmfs\_server mkfs). But at some point now when I’m adding the data on a fresh filesystem, during a transaction it fails, and I’m not able to do anything abort or publish. So, I’ve also tried to split the data into different transactions/publish. but at one point it also fails. ![image](https://cernvm-forum.cern.ch/uploads/default/original/1X/14c3cd7d51d41c74a44e0979b919bf5064f022b7.png) How can I at least abort the last transaction to restart the last sync ? cvmfs-libs-2.11.2-1.el8.x86\_64 cvmfs-config-default-2.1-1.noarch cvmfs-2.11.2-1.el8.x86\_64 cvmfs-server-2.11.2-1.el8.x86\_64 Thank you in advance. ---
**Author:** ![vavolkl](https://cernvm-forum.cern.ch/letter_avatar_proxy/v4/letter/v/ebca7d/32.png) [@vavolkl](https://cernvm-forum.cern.ch/u/vavolkl)\ **Post date:** [April 4, 2024, 9:58am UTC](https://cernvm-forum.cern.ch/t/job-for-var-spool-cvmfs-rdonly-mount-failed/422/2 "2024-04-04T09:58:48Z")
Can you also post the syslog (messages mentioning cvmfs from /var/log/messages or journalctl)? The mountpoint repair should leave some log messages there. ---
**Author:** ![fred\_iter](https://cernvm-forum.cern.ch/letter_avatar_proxy/v4/letter/f/e0b2c6/32.png) [@fred\_iter](https://cernvm-forum.cern.ch/u/fred_iter)\ **Post date:** [April 4, 2024, 10:22am UTC](https://cernvm-forum.cern.ch/t/job-for-var-spool-cvmfs-rdonly-mount-failed/422/3 "2024-04-04T10:22:06Z")
Thank you for the attention. ```auto [root@io-ls-cvmfs01:~]# systemctl status var-spool-cvmfs-imas.iter.org-rdonly.mount ● var-spool-cvmfs-imas.iter.org-rdonly.mount - /var/spool/cvmfs/imas.iter.org/rdonly Loaded: loaded (/etc/fstab; generated) Active: failed (Result: exit-code) since Thu 2024-04-04 10:51:48 CEST; 1h 27min ago Where: /var/spool/cvmfs/imas.iter.org/rdonly What: cvmfs2#imas.iter.org Docs: man:fstab(5) man:systemd-fstab-generator(8) Apr 04 10:51:24 io-ls-cvmfs01.iter.org systemd[1]: Mounting /var/spool/cvmfs/imas.iter.org/rdonly... Apr 04 10:51:24 io-ls-cvmfs01.iter.org mount[7868]: CernVM-FS: running with suid support Apr 04 10:51:24 io-ls-cvmfs01.iter.org cvmfs2[7869]: (imas.iter.org) switching proxy from (none) to DIRECT (set proxies) Apr 04 10:51:24 io-ls-cvmfs01.iter.org cvmfs2[7869]: (imas.iter.org) switching proxy from (none) to DIRECT (cloned) Apr 04 10:51:48 io-ls-cvmfs01.iter.org mount[7868]: CernVM-FS: loading Fuse module... Failed to initialize root file catalog (16 - file catalog failure) Apr 04 10:51:48 io-ls-cvmfs01.iter.org cvmfs2[7869]: (imas.iter.org) Failed to initialize root file catalog (16 - file catalog failure) Apr 04 10:51:48 io-ls-cvmfs01.iter.org systemd[1]: var-spool-cvmfs-imas.iter.org-rdonly.mount: Mount process exited, code=exited status=16 Apr 04 10:51:48 io-ls-cvmfs01.iter.org systemd[1]: var-spool-cvmfs-imas.iter.org-rdonly.mount: Failed with result 'exit-code'. Apr 04 10:51:48 io-ls-cvmfs01.iter.org systemd[1]: Failed to mount /var/spool/cvmfs/imas.iter.org/rdonly. ``` ```auto -- Unit var-spool-cvmfs-imas.iter.org-rdonly.mount has begun starting up. Apr 04 12:21:07 io-ls-cvmfs01.iter.org mount[8845]: CernVM-FS: running with suid support Apr 04 12:21:07 io-ls-cvmfs01.iter.org cvmfs2[8846]: (imas.iter.org) switching proxy from (none) to DIRECT (set proxies) Apr 04 12:21:07 io-ls-cvmfs01.iter.org cvmfs2[8846]: (imas.iter.org) switching proxy from (none) to DIRECT (cloned) Apr 04 12:21:31 io-ls-cvmfs01.iter.org mount[8845]: CernVM-FS: loading Fuse module... Failed to initialize root file catalog (16 - file catalog failure) Apr 04 12:21:31 io-ls-cvmfs01.iter.org cvmfs2[8846]: (imas.iter.org) Failed to initialize root file catalog (16 - file catalog failure) Apr 04 12:21:31 io-ls-cvmfs01.iter.org systemd[1]: var-spool-cvmfs-imas.iter.org-rdonly.mount: Mount process exited, code=exited status=16 Apr 04 12:21:31 io-ls-cvmfs01.iter.org systemd[1]: var-spool-cvmfs-imas.iter.org-rdonly.mount: Failed with result 'exit-code'. -- Subject: Unit failed -- Defined-By: systemd -- Support: https://access.redhat.com/support -- -- The unit var-spool-cvmfs-imas.iter.org-rdonly.mount has entered the 'failed' state with result 'exit-code'. Apr 04 12:21:31 io-ls-cvmfs01.iter.org systemd[1]: Failed to mount /var/spool/cvmfs/imas.iter.org/rdonly. -- Subject: Unit var-spool-cvmfs-imas.iter.org-rdonly.mount has failed -- Defined-By: systemd -- Support: https://access.redhat.com/support -- -- Unit var-spool-cvmfs-imas.iter.org-rdonly.mount has failed. -- -- The result is failed. ``` ---
**Author:** ![fred\_iter](https://cernvm-forum.cern.ch/letter_avatar_proxy/v4/letter/f/e0b2c6/32.png) [@fred\_iter](https://cernvm-forum.cern.ch/u/fred_iter)\ **Post date:** [April 4, 2024, 10:23am UTC](https://cernvm-forum.cern.ch/t/job-for-var-spool-cvmfs-rdonly-mount-failed/422/4 "2024-04-04T10:23:30Z")
```auto Apr 4 12:21:07 io-ls-cvmfs01 systemd[1]: Mounting /var/spool/cvmfs/imas.iter.org/rdonly... Apr 4 12:21:07 io-ls-cvmfs01 mount[8845]: CernVM-FS: running with suid support Apr 4 12:21:07 io-ls-cvmfs01 cvmfs2[8846]: (imas.iter.org) switching proxy from (none) to DIRECT (set proxies) Apr 4 12:21:07 io-ls-cvmfs01 cvmfs2[8846]: (imas.iter.org) switching proxy from (none) to DIRECT (cloned) Apr 4 12:21:31 io-ls-cvmfs01 mount[8845]: CernVM-FS: loading Fuse module... Failed to initialize root file catalog (16 - file catalog failure) Apr 4 12:21:31 io-ls-cvmfs01 cvmfs2[8846]: (imas.iter.org) Failed to initialize root file catalog (16 - file catalog failure) Apr 4 12:21:31 io-ls-cvmfs01 systemd[1]: var-spool-cvmfs-imas.iter.org-rdonly.mount: Mount process exited, code=exited status=16 Apr 4 12:21:31 io-ls-cvmfs01 systemd[1]: var-spool-cvmfs-imas.iter.org-rdonly.mount: Failed with result 'exit-code'. Apr 4 12:21:31 io-ls-cvmfs01 systemd[1]: Failed to mount /var/spool/cvmfs/imas.iter.org/rdonly. ``` ---
**Author:** ![vavolkl](https://cernvm-forum.cern.ch/letter_avatar_proxy/v4/letter/v/ebca7d/32.png) [@vavolkl](https://cernvm-forum.cern.ch/u/vavolkl)\ **Post date:** [April 4, 2024, 11:13am UTC](https://cernvm-forum.cern.ch/t/job-for-var-spool-cvmfs-rdonly-mount-failed/422/5 "2024-04-04T11:13:41Z")
Sounds like there is still some corruption in the repository. What does ```auto cvmfs_server check -r imas.iter.org ``` give? ---
**Author:** ![fred\_iter](https://cernvm-forum.cern.ch/letter_avatar_proxy/v4/letter/f/e0b2c6/32.png) [@fred\_iter](https://cernvm-forum.cern.ch/u/fred_iter)\ **Post date:** [April 4, 2024, 12:26pm UTC](https://cernvm-forum.cern.ch/t/job-for-var-spool-cvmfs-rdonly-mount-failed/422/6 "2024-04-04T12:26:31Z")
Here it is: ```auto # cvmfs_server check -r imas.iter.org /var/spool/cvmfs/imas.iter.org/rdonly is not mounted properly. /cvmfs/imas.iter.org is not mounted properly. imas.iter.org is in a transaction but /cvmfs/imas.iter.org is not mounted read/write Repository imas.iter.org is in a transaction and cannot be repaired. --> Run `cvmfs_server abort imas.iter.org` to revert and repair. ``` Tried: ```auto # cvmfs_server abort imas.iter.org You are about to DISCARD ALL CHANGES OF THE CURRENT TRANSACTION for imas.iter.org! Are you sure (y/N)? y /var/spool/cvmfs/imas.iter.org/rdonly is not mounted properly /cvmfs/imas.iter.org is not mounted properly Job for var-spool-cvmfs-imas.iter.org-rdonly.mount failed. See "systemctl status var-spool-cvmfs-imas.iter.org-rdonly.mount" and "journalctl -xe" for details. Trying to mount /var/spool/cvmfs/imas.iter.org/rdonly... fail Trying to mount /var/spool/cvmfs/imas.iter.org/rdonly... fail ``` Meanwhile I did a reboot of the server, just in case of, but same. ---
**Author:** ![vavolkl](https://cernvm-forum.cern.ch/letter_avatar_proxy/v4/letter/v/ebca7d/32.png) [@vavolkl](https://cernvm-forum.cern.ch/u/vavolkl)\ **Post date:** [April 4, 2024, 1:30pm UTC](https://cernvm-forum.cern.ch/t/job-for-var-spool-cvmfs-rdonly-mount-failed/422/7 "2024-04-04T13:30:31Z")
Hmmm, do you still have a client that reads from this repository? Can you mount it from another machine? Another thing to try is to remove the file ```auto rm /var/spool/cvmfs/test.cern.ch/client.local ``` and try to abort again. ---
**Author:** ![dwd](https://cernvm-forum.cern.ch/letter_avatar_proxy/v4/letter/d/b9bd4f/32.png) [@dwd](https://cernvm-forum.cern.ch/u/dwd)\ **Post date:** [April 4, 2024, 6:28pm UTC](https://cernvm-forum.cern.ch/t/job-for-var-spool-cvmfs-rdonly-mount-failed/422/8 "2024-04-04T18:28:15Z")
It would also be helpful to get more details about the error > [@fred\_iter](#): > > `16 - file catalog failure` You should be able to do that by setting ```auto CVMFS_DEBUGLOG=/tmp/imas.iter.org.debuglog ``` in `/etc/cvmfs/repositories.d/imas.iter.org/client.conf`, adding `debug` to the comma-separated list of mount options for your mountpoint in /etc/fstab, and doing `systemctl daemon-reload` before the mount attempt. Then look in the debuglog for that error and I think it should give you a more specific error. Don’t forget to undo the changes for debugging once the problem is fixed. ---
**Author:** ![fred\_iter](https://cernvm-forum.cern.ch/letter_avatar_proxy/v4/letter/f/e0b2c6/32.png) [@fred\_iter](https://cernvm-forum.cern.ch/u/fred_iter)\ **Post date:** [April 5, 2024, 10:43am UTC](https://cernvm-forum.cern.ch/t/job-for-var-spool-cvmfs-rdonly-mount-failed/422/9 "2024-04-05T10:43:31Z")
Hi, I tried different things and same. To not use your time and mine, I think it will be faster to recreate the FS and try to push the data again during the week-end. Do you have some ideas on how we can avoid this problem? ---
**Author:** ![vavolkl](https://cernvm-forum.cern.ch/letter_avatar_proxy/v4/letter/v/ebca7d/32.png) [@vavolkl](https://cernvm-forum.cern.ch/u/vavolkl)\ **Post date:** [April 5, 2024, 12:19pm UTC](https://cernvm-forum.cern.ch/t/job-for-var-spool-cvmfs-rdonly-mount-failed/422/10 "2024-04-05T12:19:38Z")
OK - Let me know how it goes. Possibly using an external s3 object store for data is more resilient. We can try to simulate various disk failures and see if there are possible improvements, but it won’t be a short-term project. ---
**Author:** ![fred\_iter](https://cernvm-forum.cern.ch/letter_avatar_proxy/v4/letter/f/e0b2c6/32.png) [@fred\_iter](https://cernvm-forum.cern.ch/u/fred_iter)\ **Post date:** [April 12, 2024, 1:29pm UTC](https://cernvm-forum.cern.ch/t/job-for-var-spool-cvmfs-rdonly-mount-failed/422/11 "2024-04-12T13:29:25Z")
Hello, I’ve tryed with the autocatalog enabled this time, instead of the file, all went well until the last publish: ![image](https://cernvm-forum.cern.ch/uploads/default/original/1X/a34dea2b3eddf718b1768bb2bd2f962c6f6e93e6.png) ---
**Author:** ![fred\_iter](https://cernvm-forum.cern.ch/letter_avatar_proxy/v4/letter/f/e0b2c6/32.png) [@fred\_iter](https://cernvm-forum.cern.ch/u/fred_iter)\ **Post date:** [April 23, 2024, 9:35am UTC](https://cernvm-forum.cern.ch/t/job-for-var-spool-cvmfs-rdonly-mount-failed/422/12 "2024-04-23T09:35:59Z")
Tried again the publish, it started, but crashed again with the exact same error ---
**Author:** ![system](https://cernvm-forum.cern.ch/uploads/default/original/1X/3e336816dfbfe00295701fad16d3bf5b3fca3a02.jpeg) [@system](https://cernvm-forum.cern.ch/u/system)\ **Post date:** [May 8, 2024, 9:36am UTC](https://cernvm-forum.cern.ch/t/job-for-var-spool-cvmfs-rdonly-mount-failed/422/13 "2024-05-08T09:36:26Z")
This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.