Hi,
We have CernVM-FS version 2.8.1 on nodes running CentOS7.8. A run time error caused (logs pasted below) caused only fermilab.opensciencegrid.org to get dismounted. See the logs pasted below. Other cvmfs mounts are ok.
Few lines below, there is another clue about “no space left on device”. What device could it be? Did the “error 28” cause the disk getting filled or the other way round?
The node is a grid site worker node and the error caused a large number of jobs to get error out. Any help is appreciated!
cvmfs2: (fermilab.opensciencegrid.org) --#012Signal: 6, errno: 28, version: 2.8.1, PID: 94084#012Executable path: /usr/bin/cvmfs2#012failed to start gdb/lldb (-1 bytes written, errno 32)#012 (failed to open crash dump file ./stacktrace.fermilab.opensciencegrid.org)
cvmfs2: (singularity.opensciencegrid.org) failed to open inode: 1903038, CAS key d0b425514b8f553e06f15aad68bfef1d7c928b72, error code 28
cvmfs2: (singularity.opensciencegrid.org) failed to open inode: 1903038, CAS key d0b425514b8f553e06f15aad68bfef1d7c928b72, error code 2
cvmfs2: (larsoft.opensciencegrid.org) failed to open inode: 300621, CAS key 9ac9367414cf487a7f940766ef52a8a33f514cfb, error code 28
cvmfs2: (larsoft.opensciencegrid.org) failed to open inode: 255918, CAS key 35e96600c4cc71e00732d0520a3c366b8eec1cfc, error code 28
cvmfs2: (singularity.opensciencegrid.org) failed to open inode: 1903038, CAS key d0b425514b8f553e06f15aad68bfef1d7c928b72, error code 28
cvmfs2: (larsoft.opensciencegrid.org) failed to open inode: 255746, CAS key 9c7468a28b6c6bdf2d831102a2c651587d0e8c40, error code 28
rsyslogd: imjournal: fopen() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': No space left on device [v8.24.0-57.el7_9 try http://www.rsyslog.com/e/2013 ]
cvmfs2: (larsoft.opensciencegrid.org) failed to open inode: 255918, CAS key 35e96600c4cc71e00732d0520a3c366b8eec1cfc, error code 28
cvmfs2: (singularity.opensciencegrid.org) failed to open inode: 1903038, CAS key d0b425514b8f553e06f15aad68bfef1d7c928b72, error code 28