SingCVMFS: Transport endpoint is not connected

Hello,

I am trying to use cvmfsexec, and more specifically singcvmfs in a cluster where CVMFS is not mounted on the nodes.
I installed the package using the following commands:

git clone https://github.com/cvmfs/cvmfsexec.git

# The /tmp space is not available
export TMPDIR=/path/to/a/tmp/directory

cvmfsexec/makedist -s default
cvmfsexec/makedist -s -o /path/to/singcvmfs

Then I have used the package such as:

export SINGCVMFS_REPOSITORIES="<repo1>,<repo2>"
export SINGCVMFS_IMAGE="docker://centos:7"
export SINGCVMFS_LOGLEVEL="debug"
./singcvmfs ls /cvmfs/<repo1>

The command fails with the following error message: cannot access /cvmfs/<repo1>: Transport endpoint is not connected.

In the logs (/var/log/cvmfs/<repo1>.log), I can read the following message: (<repo1>) could not acquire workspace lock (9) (9 - cache directory/plugin problem)
I have also tried to use another cache directory such as:

truncate -s 6G scratch.img
mkfs.ext3 -F -O ^has_journal scratch.img

export SINGCVMFS_CACHEIMAGE=scratch.img
./singcvmfs ls /cvmfs/<repo1>

In this case <repo1>.log provided a different message: cannot create workspace directory /var/lib/cvmfs/shared.
Setting SINGCVMFS_CACHEDIR with values such as /test does not help: the container does not start: FATAL: container creation failed: hook function for tag prelayer returns error: /test doesn't exist in image /path/to/scratch.img

Here are some details about the environment:

  • OS: CentOS Linux release 7.9.2009 (Core)
  • Singularity version: 3.8.1 (allow setuid = yes)
  • user.max_user_namespaces = 0

Have you already encountered this issue? Any pointers that could help?
Should you need any further details, please let me know.

Thanks

That was quite insightful of you to discover the SINGCVMFS_CACHEIMAGE option since it was added less than a month ago and I see it is not mentioned at all in the documentation. I think you’re the first one to try it besides me. I don’t think I tried it with setuid mode.

I see that the problem is that the root of the filesystem is owned by the root user and you’re not allowed to create a directory there. The singularity overlay create command gets around their similar problem by using the -d option to mkfs.ext3, which is unfortunately not yet supported in the el7 version. You can however compile it from source code or run it on a newer machine. Then create a scratch directory containing “shared” and pass it to the mkfs.ext3 command:

$ mkdir -p tmp/shared
$ truncate -s 6G scratch.img
$ mkfs.ext3 -F -O ^has_journal -d tmp scratch.img

On a machine with unprivileged user namespaces this is not necessary.

I will update the singcvmfs documentation.

Dave

Thanks a lot for the details.
I have followed your instructions, I created the scratch.img in a newer machine and exported it to the cluster.
Unfortunately, I still have this issue in /var/log/cvmfs/<repo1>.log (the one I got when I was not using the SINGCVMFS_CACHEIMAGE):

(<repo1>) could not acquire workspace lock (13) (9 - cache directory/plugin problem)

The debug mode does not provide more details. Any idea of how I could debug this?

Thanks

I am assuming the underlying problem is that the filesystem you’re using does not support locking. Where do you have the cvmfsexec files, given that there’s no /tmp? On a networked filesystem? If so, what type? Is there any local disk you could use?

Otherwise you could put /var/log/cvmfs also onto a bind-mounted ext3 filesystem. I can’t offhand think of a way to use the same .img file, but you could make another one called log.img with a writable “log” directory and set SINGULARITY_BINDPATH=log.img:/var/log/cvmfs:image-src=log. That results in a warning because singcvmfs tries to separately bind that directory, so if that’s what you want to do permanently then there should probably be a modification made to the singcvmfs script to explicitly support a different bind for the logs.

Dave

Indeed, seems like there is an issue with the shared file system.
I exported the singcvmfs executable and scratch.img on my local machine to have a local disk space:

  • it turns out that I have the same issue on a local disk space when I use SINGCVMFS_CACHEIMAGE=scratch.img, namely:

    cannot access /cvmfs/<repo1>: Transport endpoint is not connected.
    

    And in /var/log/cvmfs/<repo1>.log I have:

    (<repo1>) could not acquire workspace lock (9) (9 - cache directory/plugin problem)
    
  • If I remove SINGCVMFS_CACHEIMAGE, I have no issue, it works well.

I managed to get access to /tmp, a local disk space, on the HPC cluster.

  • I have the same issue if I use SINGCVMFS_CACHEIMAGE
  • If I unset the variable, then I have a different issue:
    In /var/log/cvmfs/<repo1> I can find the following logs:
    Fri Nov 25 03:07:46 2022 (<repo1>) GeoAPI request http://cernvmfs.gridpp.rl.ac.uk:8000/cvmfs/lhcb.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1.hpc.swin.edu.au,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch,cvmfs-stratum-one.ihep.ac.cn,cvmfsrep.grid.sinica.edu.tw failed with error 15 [host serving data too slowly]
    Fri Nov 25 03:08:07 2022 (<repo1>) GeoAPI request http://cvmfs-s1.hpc.swin.edu.au:8000/cvmfs/lhcb.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1.hpc.swin.edu.au,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch,cvmfs-stratum-one.ihep.ac.cn,cvmfsrep.grid.sinica.edu.tw failed with error 15 [host serving data too slowly]
    Fri Nov 25 03:08:22 2022 (<repo1>) GeoAPI request http://cvmfs-s1bnl.opensciencegrid.org:8000/cvmfs/lhcb.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1.hpc.swin.edu.au,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.opensciencegrid.org,cvmfs-stratum-one.cern.ch,cvmfs-stratum-one.ihep.ac.cn,cvmfsrep.grid.sinica.edu.tw failed with error 15 [host serving data too slowly]
    Fri Nov 25 03:08:22 2022 (<repo1>) failed to retrieve geographic order from stratum 1 servers
    

Hence, I have a few more questions:

  • Even if I don’t use this option eventually, have you noticed the same issue with the scratch.img process?
  • Is there a way to fix, or at least bypass, the host serving data too slowly error? A timeout option for instance.

Thanks for your help

I’m sorry, I was misinterpreting the “could not acquire workspace lock” message. I was thinking it was the log directory it was complaining about, but it’s actually the cache directory. The only thing I can think of is maybe it was improperly unmounted? It works for me. You can try running e2fsck on scratch.img.

Regarding the GeoAPI request failures, can you do those requests with curl or wget? Replace @proxy@ with the name or IP address of a local machine.