CVMFS paths are hidden from client nodes

Client
1

Hi all,

If I am running test scripts even 400 or 500 condor jobs at one time. they are working well, but when trying to run condor jobs on my data-files, only 64 jobs are running at one time because CVMFS paths are hidden from client nodes. Could any please suggest some way to get out of it?

Best regards,
Manisha

2

Hi Manisha,

welcome to the forum! In order to help you we’ll need a little more information. First, which repository are you trying to access, and which paths exactly? Where are these jobs running (the CERN batch farm, I assume?)? What is the exact error you get?
Cheers,
Valentin

3

Hi Valentine,

Thank you for replying back.
I am a researcher in IceCube, we use data-files in I3 format. The version of CVMFS that we are using is following:
eval /cvmfs/icecube.opensciencegrid.org/py3-v4.1.1/setup.sh

the jobs which I am running those need to access the data files in one repository of our cluster only.
Until few months back, everything was working well, but now do not know what happened. CVMFS paths are only visible in head node and hidden from client nodes. So, at one time - only 64 jobs are running.
the details of cluster infrastructure are following:

1 Master Node: http://icin.tifr.res.in/ installed with cvmfs, ntp, HT-condor (scheduler, submiter and execute)
2 Client nodes: installed with cvmfs, ntp and HT-condor (execute)
installed with cvmfs, ntp and HT-condor (execute)
1 Storage server: /Storage area from this server is mounted to all the above servers.- We are able to run test jobs on all the nodes, these test jobs do some calculations with some basic inputs.

  • We can execute the physics jobs on head node with all the files either in “home” or the mounter “Storage”
  • Same physics jobs can not be executed on the client machines.

Please let me know if you need some more details from my side.

Best regards,
Manisha

4

Hi Manisha,

thanks for the additional information. It’s not yet clear to me what the actual problem with CVMFS is. What do you mean by “the paths are hidden from the client nodes”? Is the particular path not accessible, or the whole repository? Do you get an error when you do a directory listing, or is it just empty?

If you can login to a client node, please also run cvmfs_config bugreport and attach here the tarball it produces.
Cheers,
Valentin

5

It would be best to contact first the Icecube grid operations people. They may already know the answers, and they can contact the CVMFS developers with more details if they can’t figure it out.

Dave

6

Hi Valentine,

If we are submitting condor jobs from head node, then jobs are only running in head node not in the client nodes.
Sure, will try to run: cvmfs_config bugreport in the client node let you know the output.

Best regards,
Manisha

7

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.