Abort fails after reboot

Publisher / Stratum 0
1

After a reboot (during a hung abort), the repository cannot enter a transaction:

cvmfs-stratum0 ~ # cvmfs_server transaction icecube.opensciencegrid.org
/var/spool/cvmfs/icecube.opensciencegrid.org/rdonly is not mounted properly
/cvmfs/icecube.opensciencegrid.org is not mounted properly
Repository icecube.opensciencegrid.org is in a transaction and cannot be repaired.
--> Run `cvmfs_server abort $name` to revert and repair.
(unexpected termination) cannot establish writable mountpoint

Stacktrace:
/usr/bin/cvmfs_publish() [0x43b01a]
/usr/bin/cvmfs_publish() [0x431078]
/usr/bin/cvmfs_publish() [0x437c3e]
/usr/bin/cvmfs_publish() [0x40f2a6]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f7e03477555]
/usr/bin/cvmfs_publish() [0x40f490]

and the abort also fails with:

cvmfs-stratum0 ~ # cvmfs_server abort icecube.opensciencegrid.org
You are about to DISCARD ALL CHANGES OF THE CURRENT TRANSACTION for icecube.opensciencegrid.org!  Are you sure (y/N)? y
/var/spool/cvmfs/icecube.opensciencegrid.org/rdonly is not mounted properly.
/cvmfs/icecube.opensciencegrid.org is not mounted properly.
icecube.opensciencegrid.org is in a transaction but /cvmfs/icecube.opensciencegrid.org is not mounted read/write
Note: Trying to mount /var/spool/cvmfs/icecube.opensciencegrid.org/rdonly... Failed to start mnt-build-spool-icecube.opensciencegrid.org-rdonly.mount: Unit not found.
fail

Not sure what to do at this point. Any advice?

Best,
David

2

Talking offline with @dwd, this turned out to be a systemd mount issue. Running

cvmfs-stratum0 ~ # systemctl daemon-reload

fixed the problem.

David

3

It’s not at all obvious from the initial error messages, but cvmfs_server mount icecube.opensciencegrid.org implicated a systemctl mount unit problem. From there we found that the .mount files were not present in /run/systemd/generated. Those were supposed to be generated by systemd.generator at either boot time or reload time. The system had been brought up in an unusual via single user mode so apparently something strange happened to prevent the systemd.generator from running. The really weird thing was that there was another repository on the same server that was mounted.

Dave