Hi all,
I imagine this has happend to people previously, but I could not find good information on how to deal with this.
On our worker nodes for a Tier 2 grid site, we have various CVMFS repos mounted (mainly: cms, belle, alice).
Occasionally - and I was unable to reproduce this - we get said errors:
ls: cannot open directory /cvmfs/belle.kek.jp: Too many levels of symbolic links
Probing /cvmfs/belle.kek.jp... Failed!
srun: error: clip-c2-2: task 0: Exited with exit code 1
in the logs I see some information, that cvmfs may have crashed previously:
Jan 17 00:02:16 clip-c2-2 cvmfs2: (belle.kek.jp) looks like cvmfs has been crashed previously
Jan 17 00:02:17 clip-c2-2 cvmfs2: (belle.kek.jp) geographic order of servers retrieved from cvmfs-s1fnal.opensciencegrid.org
Jan 17 00:02:17 clip-c2-2 cvmfs2: (belle.kek.jp) CernVM-FS: linking /cvmfs/belle.kek.jp to repository belle.kek.jp
Jan 17 00:02:17 clip-c2-2 automount[2403]: do_mount_indirect: indirect trigger not valid or already mounted /cvmfs/belle.kek.jp
Jan 17 00:02:17 clip-c2-2 automount[2403]: dev_ioctl_send_fail:499: AUTOFS_DEV_IOCTL_FAIL: error Invalid argument
Jan 17 00:02:17 clip-c2-2 automount[2403]: do_mount_indirect: indirect trigger not valid or already mounted /cvmfs/belle.kek.jp
Jan 17 00:02:17 clip-c2-2 automount[2403]: dev_ioctl_send_fail:499: AUTOFS_DEV_IOCTL_FAIL: error Invalid argument
Jan 17 00:02:17 clip-c2-2 automount[2403]: do_mount_indirect: indirect trigger not valid or already mounted /cvmfs/belle.kek.jp
Jan 17 00:02:17 clip-c2-2 automount[2403]: dev_ioctl_send_fail:499: AUTOFS_DEV_IOCTL_FAIL: error Invalid argument
Jan 17 00:02:17 clip-c2-2 automount[2403]: do_mount_indirect: indirect trigger not valid or already mounted /cvmfs/belle.kek.jp
Jan 17 00:02:17 clip-c2-2 automount[2403]: dev_ioctl_send_fail:499: AUTOFS_DEV_IOCTL_FAIL: error Invalid argument
Jan 17 00:02:17 clip-c2-2 automount[2403]: do_mount_indirect: indirect trigger not valid or already mounted /cvmfs/belle.kek.jp
Jan 17 00:02:17 clip-c2-2 automount[2403]: dev_ioctl_send_fail:499: AUTOFS_DEV_IOCTL_FAIL: error Invalid argument
[repeated many times]
This happens sporadically to repos belle.kek.jp and unpacked.cern.ch.
Noteworthy to mention:
i know that other repos are also in use, i.e. grid.cern.ch, oasis.opensciencegrid.org, sft.cern.ch but I have not gotten any reports on these.
curios is also: grid pilot jobs don’t appear to hit this issue (we run pilots for alice, belle, cms)
users only seem to be affected when running in a shared partition of slurm, where they submit directly with slurm
as mentioned: this happens sporadically and is hard to reproduce. It also looks like that if the first attempt to access those cvmfs repos, it fails, but another attempt a few seconds later would work.
Does this error occur on multiple nodes, or a single one?
You mention that the issue occurs on the first mount attempt, but subsequent attempts succeed. Would it be possible to disable autofs and mount manually (Client Configuration — CernVM-FS 2.9.0 documentation), to see if this is makes any difference?
Hi @rapopesc
Thanks for the reply.
I will try this on a testing node. However, in production I will need to retain the ability to automount, as I don’t know all the repos that will ever be in use. On a side note: I have not seen this re-occur in the last 2 days.
Because of the log line
Jan 17 00:02:16 clip-c2-2 cvmfs2: (belle.kek.jp) looks like cvmfs has been crashed previously
I was speculating this being related to a timeout or unclean auto-unmount issue.
I’ll get back with some more results on the direct, non-autofs mounting.
I encountered these errors when using autofs and the slurm job_container feature that put the job in a private file name-space with /dev/shm and /tmp private to the job. The potential solution is that the mount directory to have shared or rshared property but this is reset by autofs at every invocation or rescan of the mount point.
Because the dynamic behavior of cvmfs is requested (experiments can, and will use various repositories and one cannot predict what those are) in the end i gave up the job_container even if it is hurting the cluster (i have leftovers in the /dev/shm so from time to time i reboot the nodes).
Given the possibilities of systemd.mount and systemd.automount (in the mount unit one can specify mount options so i can add shared or rshared) now i investigate if and how i could make a systemd generator to simulate the dynamic behavior of autofs (auto mounting of unspecified repository) but so far i did not find a way to catch the request to an non-existent directory
@rapopesc I’ve run the a script with 1000x locally unmounting/mounting cvmfs without autofs, never triggering the issue.
@asevcenc that is really curious, we have in fact switched to job_container recently, I will further investigate this. I found it hard to debug as it does only happen seldomly. Still, I’ve never seen it on the login nodes where there are no slurm jobs running (and no mount namespacing)
I’m not at all surprised the issue didn’t happen without automounter, it is definitely an automount issue. The problem happens with namespaces and attempts to access a mount point within a namespace without shared/rshared as @asevcenc said, and as a result the newly mounted repository outside the namespace does not get shared inside the namespace. I have seen it under various conditions with singularity as well, although I think most of them are now fixed. If the repository is already mounted at the time the namespace is created, the problem doesn’t happen.
Yeah, I can reproduce it reliably with Slurm job_container+autofs.
Have you found any workarounds besides disabling job_container (namespaces)?
Using mount --make-rshared /cvmfs does not seem to do the trick for me.
Surely, pre-mounting all cvmfs repos is an option, but’s that’s basically the same as getting rid of automounter, i.e. we loose the flexibility of discovering cvmfs repos on demand.
So, i had to disable the job_container as even with all “known” repos mounted i begun to have failed jobs just because some ALICE jobs were using atlas.cern.ch which was outside of my expectations. And yes, somehow autofs reset the shared attribute each time it “looks” at the mount point (i tried to add a needed service to autofs to make the /cvmfs rshared at each autofs restart and it did not work… even with timeout=0 after some time the property is lost).
At this moment i’m thinking about some systemd generator but the problem is that i need to catch somehow requests to a non-existent path to be able to react and automount with rshared properties, see linux - How to catch access/request to an (non-existent) directory under a base path? - Server Fault