Turning off default proxy for a specific domain

christo · March 6, 2025, 12:14pm

Hi,

I’m trying to setup our cvmfs clients to setup to not use a proxy for our local cvmfs servers.
In /etc/cvmfs/default.local I have

CVMFS_HTTP_PROXY="http://proxy.example.com:8080"

and in /etc/cvmfs/domain.d/example.org.conf:

CVMFS_HTTP_PROXY="DIRECT"

This does not work. syslog says:

2025-03-06T10:51:55.907538+00:00 client cvmfs2: (repo.example.org) (manager 'standard') skipping DIRECT proxy to use fallback proxy
2025-03-06T10:51:55.908245+00:00 client cvmfs2: (repo.example.org) (manager 'standard') switching proxy from (none) to http://128.142.35.143:3126 (cmsextproxy.cern.ch). Reason: set random start proxy from the first proxy group [Current host: http://cvmfs-s1.example.org/cvmfs/repo.example.org)
2025-03-06T10:51:55.908396+00:00 client cvmfs2: (repo.example.org) (manager 'external') switching proxy from (none) to http://128.142.35.143:3126 (cmsextproxy.cern.ch). Reason: cloned [Current host: http://cvmfs-s1.example.org/cvmfs/repo.example.org]
2025-03-06T10:51:55.908457+00:00 client cvmfs2: (repo.example.org) (manager 'external') switching proxy from http://128.142.35.143:3126 (cmsextproxy.cern.ch) to DIRECT. Reason: set random start proxy from the first proxy group [Current host: ]
2025-03-06T10:51:55.947339+00:00 client cvmfs2: (repo.example.org) (manager 'standard') switching proxy from http://128.142.35.143:3126 (cmsextproxy.cern.ch) to http://128.142.248.156:3126 (cmsextproxy.cern.ch). Reason: failed proxy [Current host: http://cvmfs-s1.example.org/cvmfs/repo.example.org]
2025-03-06T10:51:55.986677+00:00 client cvmfs2: (repo.example.org) (manager 'standard') switching proxy from http://128.142.248.156:3126 (cmsextproxy.cern.ch) to http://131.225.188.245:3126 (cmsextproxy.fnal.gov). Reason: failed proxy [Current host: http://cvmfs-s1.example.org/cvmfs/repo.example.org]
2025-03-06T10:51:56.216331+00:00 client cvmfs2: (repo.example.org) (manager 'standard') switching proxy from http://131.225.188.245:3126 (cmsextproxy.fnal.gov) to http://131.225.188.246:3126 (cmsextproxy.fnal.gov). Reason: failed proxy [Current host: http://cvmfs-s1.example.org/cvmfs/repo.example.org]
2025-03-06T10:51:56.436515+00:00 client cvmfs2: (repo.example.org) (manager 'standard') switching proxy from http://131.225.188.246:3126 (cmsextproxy.fnal.gov) to http://128.142.35.143:3126 (cmsextproxy.cern.ch). Reason: reset proxies for host failover [Current host: http://cvmfs-s1.example.org/cvmfs/repo.example.org]
2025-03-06T10:51:56.436567+00:00 client cvmfs2: (repo.example.org)  - id 0) switching host from http://cvmfs-s1.example.org/cvmfs/repo.example.org to http://cvmfs-s0.example.org/cvmfs/repo.example.org (all proxies failed, trying host fail-over)
2025-03-06T10:51:56.537637+00:00 client cvmfs2: (repo.example.org) (manager 'standard') switching proxy from http://128.142.35.143:3126 (cmsextproxy.cern.ch) to http://128.142.248.156:3126 (cmsextproxy.cern.ch). Reason: failed proxy [Current host: http://cvmfs-s0.example.org/cvmfs/repo.example.org]
2025-03-06T10:51:56.629378+00:00 client cvmfs2: (repo.example.org) (manager 'standard') switching proxy from http://128.142.248.156:3126 (cmsextproxy.cern.ch) to http://131.225.188.245:3126 (cmsextproxy.fnal.gov). Reason: failed proxy [Current host: http://cvmfs-s0.example.org/cvmfs/repo.example.org]
2025-03-06T10:51:56.861172+00:00 client cvmfs2: (repo.example.org) (manager 'standard') switching proxy from http://131.225.188.245:3126 (cmsextproxy.fnal.gov) to http://131.225.188.246:3126 (cmsextproxy.fnal.gov). Reason: failed proxy [Current host: http://cvmfs-s0.example.org/cvmfs/repo.example.org]
2025-03-06T10:51:57.069197+00:00 client cvmfs2: (repo.example.org) failed to download repository manifest (8 - proxy returned HTTP error)
2025-03-06T10:51:57.069239+00:00 client cvmfs2: (repo.example.org) Failed to initialize root file catalog (16 - file catalog failure)

What am I missing here?

Best

Christopher

vavolkl · March 6, 2025, 2:10pm

Hi Christopher,

can you post the output of cvmfs_config showconfig repo.example.org?

It seems like the DIRECT proxy is set but the connection to the Stratum-1 server is unsuccessful. Are there any fallback proxies defined?

Cheers,
Valentin

vavolkl · March 6, 2025, 2:32pm

Oh, I think I had your question backwards. I thought you wanted it to be DIRECT. If you are trying to set CVMFS_HTTP_PROXY="http://proxy.example.com:8080" for repo.example.org , you should set CVMFS_HTTP_PROXY="http://proxy.example.com:8080" in /etc/cvmfs/domain.d/example.org.local. As a rule of thumb, never change the *.conf file, but override them with the corresponding .local ones. This is the full hierarchy:

# Don't edit here.  Create /etc/cvmfs/domain.d/cern.ch.local.
# As a rule of thumb, overwrite only parameters you find in here.
# If you look for any other parameter, check /etc/cvmfs/default.(conf|local)
# and /etc/cvmfs/config.d/<your_repository>.(conf|local)
#
# Parameter files are sourced in the following order
#
# /etc/cvmfs/default.conf
# /etc/cvmfs/default.d/*.conf (in alphabetical order)
# $CVMFS_CONFIG_REPOSITORY/etc/cvmfs/default.conf (if config repository is set)
# /etc/cvmfs/default.local
#
# $CVMFS_CONFIG_REPOSITORY/etc/cvmfs/domain.d/<your_domain>.conf (if config repository is set)
# /etc/cvmfs/domain.d/<your_domain>.conf
# /etc/cvmfs/domain.d/<your_domain>.local
#
# $CVMFS_CONFIG_REPOSITORY/etc/cvmfs/config.d/<your_repository>.conf (if config repository is set)
# /etc/cvmfs/config.d/<your_repository>.conf
# /etc/cvmfs/config.d/<your_repository>.local
#
# Use cvmfs_config showconfig to get the effective parameters.

christo · March 6, 2025, 3:08pm

OK, this is the output (without empty defaults):

CVMFS_REPOSITORY_NAME=repo.example.org
CVMFS_BACKOFF_INIT=2    # from /etc/cvmfs/default.conf
CVMFS_BACKOFF_MAX=10    # from /etc/cvmfs/default.conf
CVMFS_BASE_ENV=1    # from /etc/cvmfs/default.conf
CVMFS_CACHE_BASE=/var/cache/cvmfs    # from /etc/cvmfs/default.local
CVMFS_CACHE_DIR=/var/cache/cvmfs/shared
CVMFS_CHECK_PERMISSIONS=yes    # from /etc/cvmfs/default.conf
CVMFS_CLAIM_OWNERSHIP=yes    # from /etc/cvmfs/default.conf
CVMFS_CLIENT_PROFILE=    # from /etc/cvmfs/default.conf
CVMFS_CONFIG_REPOSITORY=cvmfs-config.cern.ch    # from /etc/cvmfs/default.d/50-cern-debian.conf
CVMFS_CONFIG_REPO_DEFAULT_ENV=1    # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf
CVMFS_DEFAULT_DOMAIN=vagrantup.com    # from /etc/cvmfs/default.local
CVMFS_FALLBACK_PROXY='http://cvmfsbproxy.cern.ch:3126;http://cvmfsbproxy.fnal.gov:3126'    # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/example.org.conf
CVMFS_HIDE_MAGIC_XATTRS=yes    # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf
CVMFS_HOST_RESET_AFTER=1800    # from /etc/cvmfs/default.conf
CVMFS_HTTP_PROXY=DIRECT    # from /etc/cvmfs/domain.d/example.org.conf
CVMFS_KEYS_DIR=/cvmfs/cvmfs-config.cern.ch/etc/cvmfs/keys/example.org    # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/gsi.de.conf
CVMFS_LOW_SPEED_LIMIT=1024    # from /etc/cvmfs/default.conf
CVMFS_MAGIC_XATTRS_VISIBILITY=rootonly    # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf
CVMFS_MAX_RETRIES=1    # from /etc/cvmfs/default.conf
CVMFS_MAX_TTL=30    # from /etc/cvmfs/default.local
CVMFS_MOUNT_DIR=/cvmfs    # from /etc/cvmfs/default.conf
CVMFS_NFILES=131072    # from /etc/cvmfs/default.conf
CVMFS_PAC_URLS='http://grid-wpad/wpad.dat;http://wpad/wpad.dat;http://cernvm-wpad.cern.ch/wpad.dat;http://cernvm-wpad.fnal.gov/wpad.dat'    # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/default.conf
CVMFS_PROXY_RESET_AFTER=300    # from /etc/cvmfs/default.conf
CVMFS_QUOTA_LIMIT=20000    # from /etc/cvmfs/default.local
CVMFS_RELOAD_SOCKETS=/var/run/cvmfs    # from /etc/cvmfs/default.conf
CVMFS_REPOSITORIES=    # from /etc/cvmfs/default.local
CVMFS_SEND_INFO_HEADER=yes    # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/example.org.conf
CVMFS_SERVER_URL='http://cvmfs-s1-int.example.org/cvmfs/repo.gsi.de;http://cvmfs-s0.gsi.de/cvmfs/vae.gsi.de'    # from /etc/cvmfs/domain.d/gsi.de.conf
CVMFS_SHARED_CACHE=yes    # from /etc/cvmfs/default.conf
CVMFS_STRICT_MOUNT=no    # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT=5    # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT_DIRECT=10    # from /etc/cvmfs/default.conf
CVMFS_USER=cvmfs    # from /etc/cvmfs/default.conf
CVMFS_USE_GEOAPI=no    # from /cvmfs/cvmfs-config.cern.ch/etc/cvmfs/domain.d/example.org.conf

CVMFS_HTTP_PROXY=DIRECT seems to be ignored and CVMFS_FALLBACK_PROXY comes into effect.

Is DIRECT the wrong identifier?

I’m using example.org.conf instead of ….local because the file is managed by us anyway.

vavolkl · March 6, 2025, 4:11pm

Right, makes sense of course.

The FALLBACK_PROXY list actually seems to be the issue, from the docs:

In addition to the regular proxy list set by CVMFS_HTTP_PROXY, a fallback proxy list is supported in CVMFS_FALLBACK_PROXY. The syntax of both lists is the same. The fallback proxy list is appended to the regular proxy list, and if the fallback proxy list is set, any DIRECT is removed from both lists.

So you’ll need to override CVMFS_FALLBACK_PROXY with an empty string in order to use DIRECT to download from the Stratum-1 without any proxy

christo · March 7, 2025, 10:15am

That works indeed.

Thanks for your help and hava a nice weekend

Christopher