Anything to change in configuration to deal with slow stashcache server?

Town Square
1

Hi,

I am not 100% sure what is the appropriate forum, so I decided to post in Town Square.

I have been told that the Cardiff’s stashcache server is slow, making jobs to stay idle for long period of times.
I don’t have experience with stashcache, so I am not sure how it works. First, is the client supposed to detect slow transfers and automatically switch to another server? Or the server is picked by GeoIP and sticks to that decision?

Would changing the setup in repository /cvmfs/config-egi.egi.eu help with this issue?
Current setup is like this:

[root@lcg2400 ~]# grep -r gravity /cvmfs/config-egi.egi.eu/etc/cvmfs/*
/cvmfs/config-egi.egi.eu/etc/cvmfs/config.d/ligo.osgstorage.org.conf:CVMFS_EXTERNAL_URL="$CVMFS_EXTERNAL_URL;https://fiona.uvalight.net:8444//user/ligo/;https://stashcache.gravity.cf.ac.uk:8444//user/ligo/;https://xcache.cr.cnaf.infn.it:8443//user/ligo/;https://xcachevirgo.pic.es:8443//user/ligo/"
/cvmfs/config-egi.egi.eu/etc/cvmfs/domain.d/osgstorage.org.conf:CVMFS_EXTERNAL_URL="$CVMFS_EXTERNAL_URL;http://fiona.uvalight.net:8000/;http://stashcache.gravity.cf.ac.uk:8000/;http://xcache.cr.cnaf.infn.it:8000/;http://xcachevirgo.pic.es:8000/"

Would changing the order and moving Cardiff to the end of the list help?

Or this is something that can only be fixed at the site?

Thanks a lot in advance.
Cheers,
Jose

2

If data transfer is slower than a set amount, the server will get an error and move on to the next. The parameter to control it is CVMFS_LOW_SPEED_LIMIT which is set in domain.d/osgstorage.org.conf to be 512 bytes per second. It is averaged over 10 seconds. The order of the servers is set by GeoIP.

No, because the servers get ordered by GeoIP.

A site could fix this by editing CVMFS_EXTERNAL_URL in the configuration of every worker node.

I also got a little more information about this by email from Steve Timm, and he told me that Cardiff was set up by LIGO collaborators and that there’s another server being set up at Edinburgh. An option then for solving this centrally could be to drop Cardiff from the general list in osgstorage.org and leaving them only in ligo.osgstorage.org, and adding Edinburgh in its place for the rest of the osgstorage.org repositories.

Dave