I’m evaluating the Compute Canada software stack via CVMFS, and my first proof of concept test has resulted in constant failures with Input/Output errors when trying to load and run many different modules, from R to Python to applications in the root like curl.
I am currently using 1 single machine and no forward Squid proxy, so I expected performance to be bad but I did not expect applications to fail to work. If I’m able to successfully get applications to work, I plan to deploy a squid proxy before adding more than a single client.
I am on Ubuntu 24.04 and following Accessing CVMFS - Alliance Doc . If there’s a more appropriate avenue for me to ask questions about the Compute Canada stack, I’d appreciation direction towards it.
My current setup is:
Thanks for the assistance! Even with my broken configuration, it was working after retrying commands and trying cvmfs_config reload, just slowly and inconsistently.
Removing CVMFS_HTTP_PROXY=DIRECT and falling back on the default value for CVMFS_HTTP_PROXY seems to have helped things, but I’m still seeing Input/Output errors as I test more applications.
Here is my whole cvmfs_config showconfig:
Running /usr/bin/cvmfs_config soft.computecanada.ca:
CVMFS_REPOSITORY_NAME=soft.computecanada.ca
CERNVM_GRID_UI_VERSION=
CVMFS_ALIEN_CACHE=
CVMFS_ALT_ROOT_PATH=
CVMFS_AUTHZ_HELPER=
CVMFS_AUTHZ_SEARCH_PATH=
CVMFS_AUTO_UPDATE=
CVMFS_BACKOFF_INIT=2 # from /etc/cvmfs/default.conf
CVMFS_BACKOFF_MAX=10 # from /etc/cvmfs/default.conf
CVMFS_BASE_ENV=1 # from /etc/cvmfs/default.conf
CVMFS_CACHE_BASE=/var/lib/cvmfs # from /etc/cvmfs/default.conf
CVMFS_CACHE_DIR=/var/lib/cvmfs/shared
CVMFS_CACHE_PRIMARY=
CVMFS_CACHE_REFCOUNT=true # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf
CVMFS_CHECK_PERMISSIONS=yes # from /etc/cvmfs/default.conf
CVMFS_CLAIM_OWNERSHIP=yes # from /etc/cvmfs/default.conf
CVMFS_CLIENT_PROFILE=single # from /etc/cvmfs/default.local
CVMFS_CONFIG_REPOSITORY=cvmfs-config.cern.ch # from /etc/cvmfs/default.d/50-cern-debian.conf
CVMFS_CONFIG_REPO_DEFAULT_ENV=1 # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf
CVMFS_CONFIG_REPO_REQUIRED=
CVMFS_DEBUGLOG=/tmp/cvmfs.log # from /etc/cvmfs/default.local
CVMFS_DEFAULT_DOMAIN=cern.ch # from /etc/cvmfs/default.d/50-cern-debian.conf
CVMFS_DNS_RETRIES=
CVMFS_DNS_TIMEOUT=
CVMFS_EXTERNAL_FALLBACK_PROXY=
CVMFS_EXTERNAL_HTTP_PROXY=
CVMFS_EXTERNAL_SERVER_URL=
CVMFS_EXTERNAL_TIMEOUT=
CVMFS_EXTERNAL_TIMEOUT_DIRECT=
CVMFS_FALLBACK_PROXY= # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/computecanada.ca.conf
CVMFS_FOLLOW_REDIRECTS=
CVMFS_HIDE_MAGIC_XATTRS=yes # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf
CVMFS_HOST_RESET_AFTER=1800 # from /etc/cvmfs/default.conf
CVMFS_HTTP_PROXY='auto;DIRECT' # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/computecanada.ca.conf
CVMFS_IGNORE_SIGNATURE=
CVMFS_INITIAL_GENERATION=
CVMFS_IPFAMILY_PREFER=
CVMFS_KCACHE_TIMEOUT=
CVMFS_KEYS_DIR=/cvmfs/cvmfs-config.cern.ch/etc/cvmfs/keys/computecanada.ca # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/computecanada.ca.conf
CVMFS_LOW_SPEED_LIMIT=1024 # from /etc/cvmfs/default.conf
CVMFS_MAGIC_XATTRS_VISIBILITY=rootonly # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf
CVMFS_MAX_IPADDR_PER_PROXY=
CVMFS_MAX_RETRIES=1 # from /etc/cvmfs/default.conf
CVMFS_MAX_TTL=
CVMFS_MEMCACHE_SIZE=
CVMFS_MOUNT_DIR=/cvmfs # from /etc/cvmfs/default.conf
CVMFS_MOUNT_RW=
CVMFS_NFILES=131072 # from /etc/cvmfs/default.conf
CVMFS_NFS_SHARED=
CVMFS_NFS_SOURCE=
CVMFS_OOM_SCORE_ADJ=
CVMFS_PAC_URLS='http://grid-wpad/wpad.dat;http://wpad/wpad.dat;http://cernvm-wpad.cern.ch/wpad.dat;http://cernvm-wpad.fnal.gov/wpad.dat' # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf
CVMFS_PROXY_RESET_AFTER=300 # from /etc/cvmfs/default.conf
CVMFS_PROXY_TEMPLATE=
CVMFS_PUBLIC_KEY=
CVMFS_QUOTA_LIMIT=4000 # from /etc/cvmfs/default.conf
CVMFS_RELOAD_SOCKETS=/var/run/cvmfs # from /etc/cvmfs/default.conf
CVMFS_REPOSITORIES=soft.computecanada.ca # from /etc/cvmfs/default.local
CVMFS_REPOSITORY_DATE=
CVMFS_REPOSITORY_TAG=
CVMFS_ROOT_HASH=
CVMFS_SEND_INFO_HEADER=yes # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/computecanada.ca.conf
CVMFS_SERVER_CACHE_MODE=
CVMFS_SERVER_URL=http://cvmfs-s1.computecanada.net/cvmfs/soft.computecanada.ca # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/computecanada.ca.conf
CVMFS_SHARED_CACHE=yes # from /etc/cvmfs/default.conf
CVMFS_STRICT_MOUNT=no # from /etc/cvmfs/default.conf
CVMFS_SYSLOG_FACILITY=
CVMFS_SYSLOG_LEVEL=
CVMFS_SYSTEMD_NOKILL=
CVMFS_TIMEOUT=5 # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT_DIRECT=10 # from /etc/cvmfs/default.conf
CVMFS_TRACEFILE=
CVMFS_TRUSTED_CERTS=
CVMFS_USER=cvmfs # from /etc/cvmfs/default.conf
CVMFS_USE_CDN=
CVMFS_USE_GEOAPI=yes # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/computecanada.ca.conf
CVMFS_WORKSPACE=
Could you please provide more debug logs? In particular each one of them:
when it is working
when it is working but took a long time
when it is failing
I think you have 2 issues.
Your config of the server and proxies is suboptimal - but this should not be the failing reason.
You normally either set CVMFS_CLIENT_PROFILE or CVMFS_HTTP_PROXY but only in rare cases both. If you use DIRECT you cannot use GEOAPI because there is only one server for compute canada if you set DIRECT.
I think your main issue is with the cache. Could you please let us know which kind of errors you find? Is it EIO (01)? If yes you need to increase CVMFS_QUOTA_LIMIT= to something larger than the default CVMFS_QUOTA_LIMIT=4000 (= 4GB)
My suggestion for your minimal config in /etc/cvmfs/default.local:
CVMFS_CLIENT_PROFILE=single
CVMFS_QUOTA_LIMIT=10000
CVMFS_DEBUGLOG=/tmp/cvmfs-@fqrn@.log # this makes a separate log for each repo with reponame @fqrn@
Let me know if that helps
Cheers
Laura
PS. CVMFS_REPOSITORIES is not needed. This is only useful if: you want to auto-run the cvmfs config commands on the list repositories OR if you want to enforce that users only mount those repositories with CVMFS_STRICT_MOUNT=on
Also make sure there’s enough disk space to hold CVMFS_QUOTA_LIMIT worth of cache.
It doesn’t really make sense that you’ll have better success leaving CVMFS_HTTP_PROXY unset than to set it to DIRECT, because the default setting is auto;DIRECT which means it will try to first read http://cernvm-wpad.cern.ch/wpad.dat and that probably does not return anything helpful. It doesn’t know that computecanada.net is Cloudflare so it returns NONE. In fact that’s probably where that http://NULL is coming from.