Client unable to mount any repository: status=219/CGROUP

Client
1

Hi,

I am having problems with one client, unable to mount any repository. I am not sure how to troubleshoot it. Any tip or suggestion is more than welcome.

[root@lcg2615 ~]# uname -a
Linux lcg2615.gridpp.rl.ac.uk 5.4.224-1.el8.elrepo.x86_64 #1 SMP Tue Nov 8 17:27:46 EST 2022 x86_64 x86_64 x86_64 GNU/Linux

[root@lcg2615 ~]# rpm -qa | grep cvmfs
cvmfs-2.9.2-1.el8.x86_64
cvmfs-x509-helper-2.2-2.29.obs.el8.x86_64
cvmfs-config-egi-2.6-1.7.obs.el8.noarch
egi-cvmfs-4-3.19.obs.el8.noarch
tier1-cvmfs-fixer-1.0-1.noarch

[root@lcg2615 ~]# ls /cvmfs/
[root@lcg2615 ~]#

[root@lcg2615 ~]# ls /cvmfs/atlas.cern.ch
ls: cannot access '/cvmfs/atlas.cern.ch': No such file or directory

[root@lcg2615 ~]# mount -t cvmfs atlas.cern.ch /cvmfs/atlas.cern.ch
required configuration repository directory does not exist: /cvmfs/config-egi.egi.eu/etc/cvmfs

[root@lcg2615 ~]# mount -t cvmfs config-egi.egi.eu /cvmfs/config-egi.egi.eu
Moint point /cvmfs/config-egi.egi.eu does not exist

[root@lcg2615 ~]# systemctl stop autofs.service
[root@lcg2615 ~]# systemctl start autofs.service
Job for autofs.service failed because the control process exited with error code.
See "systemctl status autofs.service" and "journalctl -xe" for details.

[root@lcg2615 ~]# systemctl status autofs.service
â—Ź autofs.service - Automounts filesystems on demand
   Loaded: loaded (/usr/lib/systemd/system/autofs.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/autofs.service.d
           └─50-cvmfs.conf
           /etc/systemd/system/autofs.service.d
           └─fuse.conf
   Active: failed (Result: exit-code) since Mon 2023-01-16 09:21:24 GMT; 5s ago
  Process: 3408823 ExecStart=/usr/sbin/automount $OPTIONS --systemd-service --dont-check-daemon (code=exited, status=219/CGROUP)
 Main PID: 3408823 (code=exited, status=219/CGROUP)

Jan 16 09:21:24 lcg2615.gridpp.rl.ac.uk systemd[1]: Starting Automounts filesystems on demand...
Jan 16 09:21:24 lcg2615.gridpp.rl.ac.uk systemd[1]: autofs.service: Main process exited, code=exited, status=219/CGROUP
Jan 16 09:21:24 lcg2615.gridpp.rl.ac.uk systemd[1]: autofs.service: Failed with result 'exit-code'.
Jan 16 09:21:24 lcg2615.gridpp.rl.ac.uk systemd[1]: Failed to start Automounts filesystems on demand.

Let me know if there is some extra piece of data I should provide for.

Thanks a lot in advance.
Jose

2

Hi Jose,

thanks for the report! The debug log may be useful - could you create it by setting the configuration parameter CVMFS_DEBUGLOG=/tmp/cvmfs.log and attach it? There are also a few general tips in Getting Started — CernVM-FS 2.10.0 documentation that may help, but let me take a look at the log first.
Cheers,
Valentin

3

Hmm. I have added CVMFS_DEBUGLOG=/tmp/cvmfs.log to default.local, but I don’t see any file being created in /tmp/.
I only see entries in /var/log/messages:

[root@lcg2615 ~]# mount -t cvmfs config-egi.egi.eu /cvmfs/config-egi.egi.eu
[root@lcg2615 ~]# grep cvmfs2 /var/log/messages | tail -1
Jan 18 11:45:57 lcg2615 cvmfs2[4156912]: (config-egi.egi.eu) Moint point /cvmfs/config-egi.egi.eu does not exist
4

Hi Jose,

From the things you have posted it seems like it is not a cvmfs problem but an autofs problem.
Not sure why CVMFS_DEBUGLOG writting to /tmp/ does not work for you (maybe some other conf overwrites it to some invalid value?).

For direct debugging on the console I recommend to remove CVMFS_DEBUGLOG from the conf and use this command

export CVMFS_REPO=<myrepo>
sudo /usr/bin/cvmfs2  -d -f -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=`id -u cvmfs`,gid=`id -g cvmfs` $CVMFS_REPO /mnt/test

with /mnt/test being the directory to mount it on

this should put out all the debug log onto the console

Furthermore, short google for error 219 of autofs gave me following result:
([solved] systemd 219: automount doesn't work anymore / [testing] Repo Forum / Arch Linux Forums)
(Race in simultaneous systemd starts in docker containers results in 219/CGROUP errors · Issue #20939 · systemd/systemd · GitHub)

Autofs is notoriously fragile. For me my “normal” routine when something breaks because of that is

sudo cvmfs_config wipecache
sudo cvmfs_config killall

#if the commands hang: check if it is due to autofs hanging during umount. 
# --> try manually calling "umount -lf" and then retry above + stop autofs
#.. and if all this does not help: restart of the server
5

Jose, you’re not seeing anything in /tmp/cvmfs.log because it is not getting as far as starting cvmfs2. When not using autofs, you have to first mkdir the mountpoint. When using autofs, you cannot manually mount.

6

Hi,

the cvmfs_config commands worked fine. But cvmfs2 doesn’t seem to work. At some point it gets stuck. Logs attached.

However, I believe at this point it is not a problem with CVMFS. It seems to me that cgroups is totally broken on the host. The directory /sys/fs/cgroup/cpu/ should have a lot of stuff in it, but it is empty.
I am going to reboot the host.

Logs:

[root@lcg2615 ~]# mkdir /mnt/test
[root@lcg2615 ~]# export CVMFS_REPO=cms.cern.ch
[root@lcg2615 ~]# sudo /usr/bin/cvmfs2  -d -f -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=`id -u cvmfs`,gid=`id -g cvmfs` $CVMFS_REPO /mnt/test
Debug: using library /usr/lib64/libcvmfs_fuse_stub.so
required configuration repository directory does not exist: /cvmfs/config-egi.egi.eu/etc/cvmfs

[root@lcg2615 ~]# export CVMFS_REPO=config-egi.egi.eu
[root@lcg2615 ~]# sudo /usr/bin/cvmfs2  -d -f -o rw,system_mount,fsname=cvmfs2,allow_other,grab_mountpoint,uid=`id -u cvmfs`,gid=`id -g cvmfs` $CVMFS_REPO /mnt/test
Debug: using library /usr/lib64/libcvmfs_fuse_stub.so
CernVM-FS: running with credentials 993:987
CernVM-FS: running in debug mode
CernVM-FS: loading Fuse module... (cvmfs) Parsing config file /etc/cvmfs/default.conf    [01-24-2023 07:57:15 GMT]
(cvmfs) execve'd /bin/sh (PID: 1757759)    [01-24-2023 07:57:15 GMT]
(cvmfs) Parsing config file /etc/cvmfs/default.d/60-egi.conf    [01-24-2023 07:57:15 GMT]
(cvmfs) execve'd /bin/sh (PID: 1757761)    [01-24-2023 07:57:15 GMT]
(cvmfs) Parsing config file /etc/cvmfs/default.local    [01-24-2023 07:57:15 GMT]
(cvmfs) execve'd /bin/sh (PID: 1757763)    [01-24-2023 07:57:15 GMT]
(cvmfs) Parsing config file /etc/cvmfs/domain.d/egi.eu.conf    [01-24-2023 07:57:15 GMT]
(cvmfs) Parsing config file /etc/cvmfs/domain.d/egi.eu.local    [01-24-2023 07:57:15 GMT]
(cvmfs) Parsing config file /etc/cvmfs/config.d/config-egi.egi.eu.conf    [01-24-2023 07:57:15 GMT]
(cvmfs) execve'd /bin/sh (PID: 1757765)    [01-24-2023 07:57:15 GMT]
(cvmfs) Parsing config file /etc/cvmfs/config.d/config-egi.egi.eu.local    [01-24-2023 07:57:15 GMT]
(cvmfs) Options:
CVMFS_BACKOFF_INIT=2    # from /etc/cvmfs/default.conf
CVMFS_BACKOFF_MAX=10    # from /etc/cvmfs/default.conf
CVMFS_BASE_ENV=1    # from /etc/cvmfs/default.conf
CVMFS_CACHE_BASE=/pool/cache/cvmfs2/    # from /etc/cvmfs/default.local
CVMFS_CHECK_PERMISSIONS=yes    # from /etc/cvmfs/default.conf
CVMFS_CLAIM_OWNERSHIP=yes    # from /etc/cvmfs/default.conf
CVMFS_CLIENT_PROFILE=    # from /etc/cvmfs/default.conf
CVMFS_CONFIG_REPOSITORY=config-egi.egi.eu    # from /etc/cvmfs/default.d/60-egi.conf
CVMFS_CONFIG_REPO_REQUIRED=yes    # from /etc/cvmfs/default.d/60-egi.conf
CVMFS_FALLBACK_PROXY='http://cvmfsbproxy.cern.ch:3126;http://cvmfsbproxy.fnal.gov:3126'    # from /etc/cvmfs/config.d/config-egi.egi.eu.conf
CVMFS_HOST_RESET_AFTER=1800    # from /etc/cvmfs/default.conf
CVMFS_HTTP_PROXY=http://cvmfs-squid.gridpp.rl.ac.uk:3128    # from /etc/cvmfs/config.d/config-egi.egi.eu.conf
CVMFS_KEYS_DIR=/etc/cvmfs/keys/egi.eu    # from /etc/cvmfs/config.d/config-egi.egi.eu.conf
CVMFS_LOW_SPEED_LIMIT=1024    # from /etc/cvmfs/default.conf
CVMFS_MAX_RETRIES=1    # from /etc/cvmfs/default.conf
CVMFS_MEMCACHE_SIZE=64    # from /etc/cvmfs/default.local
CVMFS_MOUNT_DIR=/cvmfs    # from /etc/cvmfs/default.conf
CVMFS_NFILES=131072    # from /etc/cvmfs/default.conf
CVMFS_PAC_URLS='http://grid-wpad/wpad.dat;http://wpad/wpad.dat;http://cernvm-wpad.fnal.gov/wpad.dat;http://cernvm-wpad.cern.ch/wpad.dat'    # from /etc/cvmfs/config.d/config-egi.egi.eu.conf
CVMFS_PROXY_RESET_AFTER=300    # from /etc/cvmfs/default.conf
CVMFS_QUOTA_LIMIT=80000    # from /etc/cvmfs/default.local
CVMFS_RELOAD_SOCKETS=/var/run/cvmfs    # from /etc/cvmfs/default.conf
CVMFS_SEND_INFO_HEADER=yes    # from /etc/cvmfs/default.d/60-egi.conf
CVMFS_SERVER_URL='http://cvmfs-egi.gridpp.rl.ac.uk:8000/cvmfs/config-egi.egi.eu;http://cvmfs01.nikhef.nl:8000/cvmfs/config-egi.egi.eu;http://cvmfsrepo.lcg.triumf.ca:8000/cvmfs/config-egi.egi.eu;http://cvmfsrep.grid.sinica.edu.tw:8000/cvmfs/config-egi.egi.eu;http://cvmfs-stratum-one.ihep.ac.cn:8000/cvmfs/config-egi.egi.eu'    # from /etc/cvmfs/config.d/config-egi.egi.eu.conf
CVMFS_SHARED_CACHE=yes    # from /etc/cvmfs/default.conf
CVMFS_STRICT_MOUNT=no    # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT=5    # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT_DIRECT=10    # from /etc/cvmfs/default.conf
CVMFS_USER=cvmfs    # from /etc/cvmfs/default.conf
CVMFS_USE_CDN=    # from /etc/cvmfs/config.d/config-egi.egi.eu.conf
CVMFS_USE_GEOAPI=yes    # from /etc/cvmfs/default.d/60-egi.conf
    [01-24-2023 07:57:15 GMT]
(cvmfs) setting up cache manager instance default    [01-24-2023 07:57:15 GMT]
(quota) trying to connect to existing pipe    [01-24-2023 07:57:15 GMT]
(cvmfs) execve'd /usr/bin/cvmfs2 (PID: 1757767)    [01-24-2023 07:57:15 GMT]
(quota) starting quota manager    [01-24-2023 07:57:15 GMT]
(monitor) starting watchdog listener    [01-24-2023 07:57:15 GMT]
(quota) free space: 6538988 MB    [01-24-2023 07:57:15 GMT]
(quota) shared cache manager listening    [01-24-2023 07:57:15 GMT]
(quota) shared cache manager handshake done    [01-24-2023 07:57:15 GMT]
(quota) connected to a new cache manager    [01-24-2023 07:57:15 GMT]
(quota) starting quota manager    [01-24-2023 07:57:15 GMT]
(quota) received command 11    [01-24-2023 07:57:15 GMT]
(quota) received limit 83886080000, threshold 41943040000    [01-24-2023 07:57:15 GMT]
(quota) received command 10    [01-24-2023 07:57:15 GMT]
(quota) received command 10    [01-24-2023 07:57:15 GMT]
(cvmfs) CernVM-FS: quota initialized, current size 76440MB    [01-24-2023 07:57:15 GMT]
(cvmfs) CernVM-FS: using public key(s) /etc/cvmfs/keys/egi.eu/egi.eu.pub    [01-24-2023 07:57:15 GMT]
(download) first fallback proxy group 1    [01-24-2023 07:57:15 GMT]
(download) full proxy list http://cvmfs-squid.gridpp.rl.ac.uk:3128;http://cvmfsbproxy.cern.ch:3126;http://cvmfsbproxy.fnal.gov:3126    [01-24-2023 07:57:15 GMT]
(download) resolving 3 proxy addresses    [01-24-2023 07:57:15 GMT]
(dns) add address cvmfs-squid.gridpp.rl.ac.uk -> 130.246.183.247    [01-24-2023 07:57:15 GMT]
(dns) add address cvmfs-squid.gridpp.rl.ac.uk -> 130.246.183.248    [01-24-2023 07:57:15 GMT]
(dns) add address cvmfs-squid.gridpp.rl.ac.uk -> 130.246.183.246    [01-24-2023 07:57:15 GMT]
(dns) add address cvmfs-squid.gridpp.rl.ac.uk -> 130.246.183.244    [01-24-2023 07:57:15 GMT]
(dns) add address cvmfs-squid.gridpp.rl.ac.uk -> 130.246.183.211    [01-24-2023 07:57:15 GMT]
(dns) add address cvmfs-squid.gridpp.rl.ac.uk -> 2001:630:58:1800::82f6:b7f7    [01-24-2023 07:57:15 GMT]
(dns) add address cvmfs-squid.gridpp.rl.ac.uk -> 2001:630:58:1800::82f6:b7f4    [01-24-2023 07:57:15 GMT]
(dns) add address cvmfs-squid.gridpp.rl.ac.uk -> 2001:630:58:1800::82f6:b7d3    [01-24-2023 07:57:15 GMT]
(dns) add address cvmfs-squid.gridpp.rl.ac.uk -> 2001:630:58:1800::82f6:b7f6    [01-24-2023 07:57:15 GMT]
(dns) add address cvmfs-squid.gridpp.rl.ac.uk -> 2001:630:58:1800::82f6:b7f8    [01-24-2023 07:57:15 GMT]
(dns) add address cvmfsbproxy.cern.ch -> 128.142.161.84    [01-24-2023 07:57:15 GMT]
(dns) add address cvmfsbproxy.cern.ch -> 128.142.248.156    [01-24-2023 07:57:15 GMT]
(dns) add address cvmfsbproxy.cern.ch -> 2001:1458:301:b8::100:12    [01-24-2023 07:57:15 GMT]
(dns) add address cvmfsbproxy.cern.ch -> 2001:1458:301:73::100:9b    [01-24-2023 07:57:15 GMT]
(dns) add address cvmfsbproxy.fnal.gov -> 131.225.188.245    [01-24-2023 07:57:15 GMT]
(dns) add address cvmfsbproxy.fnal.gov -> 131.225.188.246    [01-24-2023 07:57:15 GMT]
(dns) add address cvmfsbproxy.fnal.gov -> 2620:6a:0:8421::92    [01-24-2023 07:57:15 GMT]
(dns) add address cvmfsbproxy.fnal.gov -> 2620:6a:0:8421::91    [01-24-2023 07:57:15 GMT]
(download) installed 9 proxies in 3 load-balance groups    [01-24-2023 07:57:15 GMT]
(download) requesting ordered server list from http://cvmfs-egi.gridpp.rl.ac.uk:8000/cvmfs/config-egi.egi.eu/api/v1.0/geo/@proxy@/cvmfs-egi.gridpp.rl.ac.uk,cvmfs-stratum-one.ihep.ac.cn,cvmfs01.nikhef.nl,cvmfsrep.grid.sinica.edu.tw,cvmfsrepo.lcg.triumf.ca,cmsextproxy.cern.ch,cmsextproxy.fnal.gov    [01-24-2023 07:57:15 GMT]
(download) replacing @proxy@ by cvmfs-squid.gridpp.rl.ac.uk    [01-24-2023 07:57:15 GMT]
(download) escaped http://cvmfs-egi.gridpp.rl.ac.uk:8000/cvmfs/config-egi.egi.eu/api/v1.0/geo/cvmfs-squid.gridpp.rl.ac.uk/cvmfs-egi.gridpp.rl.ac.uk,cvmfs-stratum-one.ihep.ac.cn,cvmfs01.nikhef.nl,cvmfsrep.grid.sinica.edu.tw,cvmfsrepo.lcg.triumf.ca,cmsextproxy.cern.ch,cmsextproxy.fnal.gov to http://cvmfs-egi.gridpp.rl.ac.uk:8000/cvmfs/config-egi.egi.eu/api/v1.0/geo/cvmfs-squid.gridpp.rl.ac.uk/cvmfs-egi.gridpp.rl.ac.uk,cvmfs-stratum-one.ihep.ac.cn,cvmfs01.nikhef.nl,cvmfsrep.grid.sinica.edu.tw,cvmfsrepo.lcg.triumf.ca,cmsextproxy.cern.ch,cmsextproxy.fnal.gov    [01-24-2023 07:57:15 GMT]
(download) Verify downloaded url http://cvmfs-egi.gridpp.rl.ac.uk:8000/cvmfs/config-egi.egi.eu/api/v1.0/geo/@proxy@/cvmfs-egi.gridpp.rl.ac.uk,cvmfs-stratum-one.ihep.ac.cn,cvmfs01.nikhef.nl,cvmfsrep.grid.sinica.edu.tw,cvmfsrepo.lcg.triumf.ca,cmsextproxy.cern.ch,cmsextproxy.fnal.gov, proxy http://130.246.183.248:3128 (curl error 0)    [01-24-2023 07:57:15 GMT]
(download) geographic order of servers retrieved from cvmfs-egi.gridpp.rl.ac.uk    [01-24-2023 07:57:15 GMT]
(download) order is 1,3,6,7,5,2,4
    [01-24-2023 07:57:15 GMT]
(download) first fallback proxy group 1    [01-24-2023 07:57:15 GMT]
(download) full proxy list DIRECT    [01-24-2023 07:57:15 GMT]
(download) resolving 1 proxy addresses    [01-24-2023 07:57:15 GMT]
(dns) empty hostname    [01-24-2023 07:57:15 GMT]
(download) installed 1 proxies in 1 load-balance groups    [01-24-2023 07:57:15 GMT]
(cvmfs) DNS roaming is disabled for this repository.    [01-24-2023 07:57:15 GMT]
(catalog) constructing client catalog manager    [01-24-2023 07:57:15 GMT]
(catalog) Initialize catalog    [01-24-2023 07:57:15 GMT]
(cache) cached copy publish date 3 Nov 2022 15:49:03 (hash 5a71e8e80802b3d17cecdbaef03021fb2b2d55e3)    [01-24-2023 07:57:15 GMT]
(download) escaped http://cvmfs-egi.gridpp.rl.ac.uk:8000/cvmfs/config-egi.egi.eu/.cvmfspublished to http://cvmfs-egi.gridpp.rl.ac.uk:8000/cvmfs/config-egi.egi.eu/.cvmfspublished    [01-24-2023 07:57:15 GMT]
(download) Verify downloaded url /.cvmfspublished, proxy http://130.246.183.248:3128 (curl error 0)    [01-24-2023 07:57:15 GMT]
(cache) remote checksum is 5a71e8e80802b3d17cecdbaef03021fb2b2d55e3    [01-24-2023 07:57:15 GMT]
(cache) hit ./5a/71e8e80802b3d17cecdbaef03021fb2b2d55e3    [01-24-2023 07:57:15 GMT]
(quota) pin into lru 5a71e8e80802b3d17cecdbaef03021fb2b2d55e3, path file catalog at config-egi.egi.eu:/ (5a71e8e80802b3d17cecdbaef03021fb2b2d55e3)    [01-24-2023 07:57:15 GMT]
(quota) received command 0    [01-24-2023 07:57:15 GMT]
(quota) received command 2    [01-24-2023 07:57:15 GMT]
(quota) reserve 29696 bytes for 5a71e8e80802b3d17cecdbaef03021fb2b2d55e3    [01-24-2023 07:57:15 GMT]
(quota) insert into lru 5a71e8e80802b3d17cecdbaef03021fb2b2d55e3, path file catalog at config-egi.egi.eu:/ (5a71e8e80802b3d17cecdbaef03021fb2b2d55e3), method 3    [01-24-2023 07:57:15 GMT]
(cache) hit: file catalog at config-egi.egi.eu:/ (5a71e8e80802b3d17cecdbaef03021fb2b2d55e3)    [01-24-2023 07:57:15 GMT]
(quota) received command 3    [01-24-2023 07:57:15 GMT]
(catalog) attaching catalog file @16    [01-24-2023 07:57:15 GMT]
(sql) opening database file @16    [01-24-2023 07:57:15 GMT]
(cache) read-ahead 16, 29696    [01-24-2023 07:57:15 GMT]
(sql) open sqlite3 catalog on fd 16, size 29696    [01-24-2023 07:57:15 GMT]
(sql) successfully prepared statement 'PRAGMA temp_store=2;'    [01-24-2023 07:57:15 GMT]
(sql) successfully prepared statement 'PRAGMA locking_mode=EXCLUSIVE;'    [01-24-2023 07:57:15 GMT]
(sql) successfully finalized statement    [01-24-2023 07:57:15 GMT]
(sql) successfully finalized statement    [01-24-2023 07:57:15 GMT]
(sql) successfully prepared statement 'BEGIN;'    [01-24-2023 07:57:15 GMT]
(sql) successfully prepared statement 'COMMIT;'    [01-24-2023 07:57:15 GMT]
(sql) successfully prepared statement 'SELECT count(*) FROM properties WHERE key = :key;'    [01-24-2023 07:57:15 GMT]
(sql) successfully prepared statement 'SELECT value FROM properties WHERE key = :key;'    [01-24-2023 07:57:15 GMT]
(sql) successfully prepared statement 'INSERT OR REPLACE INTO properties (key, value) VALUES (:key, :value);'    [01-24-2023 07:57:15 GMT]
(sql) opened database with schema version 2.500000 and revision 6    [01-24-2023 07:57:15 GMT]
(sql) successfully prepared statement 'SELECT DISTINCT hash, CASE WHEN flags & 4 THEN 0 WHEN flags & 1 THEN 76 END AS chunk_type,  ((flags&1792) >> 8)+1 AS hash_algorithm , ((flags&14336) >> 11) AS compression_algorithm FROM catalog WHERE (hash IS NOT NULL) AND (flags & 128 = 0) UNION SELECT DISTINCT chunks.hash, 80,  ((flags&1792) >> 8)+1 AS hash_algorithm , ((flags&14336) >> 11) AS compression_algorithm FROM chunks, catalog WHERE chunks.md5path_1=catalog.md5path_1 AND chunks.md5path_2=catalog.md5path_2 AND (catalog.flags & 128 = 0);'    [01-24-2023 07:57:15 GMT]
(sql) successfully prepared statement 'SELECT MAX(rowid) FROM catalog;'    [01-24-2023 07:57:15 GMT]
(catalog) no root prefix for root catalog file @16    [01-24-2023 07:57:15 GMT]
(sql) successfully prepared statement 'SELECT value from statistics WHERE counter = :counter;'    [01-24-2023 07:57:15 GMT]
(sql) successfully finalized statement    [01-24-2023 07:57:15 GMT]
(sql) successfully finalized statement    [01-24-2023 07:57:15 GMT]
(catalog) allocating inodes from 256 to 346.    [01-24-2023 07:57:15 GMT]
(cache) checking if config-egi.egi.eu revision 70 is blacklisted    [01-24-2023 07:57:15 GMT]
(cvmfs) kernel caches expire after 60 seconds    [01-24-2023 07:57:15 GMT]
(cvmfs) fuse inode size is 64 bits    [01-24-2023 07:57:15 GMT]
(cvmfs) root inode is 256    [01-24-2023 07:57:15 GMT]
(talk) socket created at ./cvmfs_io.config-egi.egi.eu (fd 19)    [01-24-2023 07:57:15 GMT]
done
FUSE library version: 2.9.7
CernVM-FS: mounted cvmfs on /mnt/test
(monitor) starting watchdog listener    [01-24-2023 07:57:15 GMT]
(cvmfs) starting dentry invalidator thread    [01-24-2023 07:57:15 GMT]
(cvmfs) starting remount trigger    [01-24-2023 07:57:15 GMT]
(cvmfs) starting negative entry cache cleaner    [01-24-2023 07:57:15 GMT]
(download) download I/O thread started    [01-24-2023 07:57:15 GMT]
(download) download I/O thread started    [01-24-2023 07:57:15 GMT]
(quota) received command 14    [01-24-2023 07:57:15 GMT]
(quota) register back channel 5c26a8b3f2fb6c2d975add471f9649d3 on fd 8    [01-24-2023 07:57:15 GMT]
(quota) received command 14    [01-24-2023 07:57:15 GMT]
(quota) register back channel 1783da726ea831091c4cfda82525690f on fd 9    [01-24-2023 07:57:15 GMT]
(quota) starting cache manager watchdog for fbb1cd19-5989-4c41-279d-6f332ff1b97f-watchdog    [01-24-2023 07:57:15 GMT]
(quota) starting unpin listener for fbb1cd19-5989-4c41-279d-6f332ff1b97f-unpin    [01-24-2023 07:57:15 GMT]
(talk) talk thread started    [01-24-2023 07:57:15 GMT]
(talk) accepting connections on socketfd 19    [01-24-2023 07:57:15 GMT]
unique: 2, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0
INIT: 7.31
flags=0x03fffffb
max_readahead=0x00020000
(cvmfs) cvmfs_init    [01-24-2023 07:57:15 GMT]
   INIT: 7.19
   flags=0x00000001
   max_readahead=0x00020000
   max_write=0x00020000
   max_background=0
   congestion_threshold=0
   unique: 2, success, outsize: 40
(cvmfs) negative entry cleaner: pruning    [01-24-2023 07:58:15 GMT]
(cvmfs) negative entry cleaner: pruning    [01-24-2023 07:59:15 GMT]

           ... it seems to be stuck here. I killed it ...

^C(talk) terminating talk thread (fd -1, errno 22)    [01-24-2023 07:59:23 GMT]
(talk) talk thread stopped    [01-24-2023 07:59:23 GMT]
(cvmfs) stopping remount trigger    [01-24-2023 07:59:23 GMT]
(cvmfs) stopping dentry invalidator thread    [01-24-2023 07:59:23 GMT]
(quota) stopping unpin listener for fbb1cd19-5989-4c41-279d-6f332ff1b97f-unpin    [01-24-2023 07:59:23 GMT]
(quota) stopping cache manager watchdog for fbb1cd19-5989-4c41-279d-6f332ff1b97f-watchdog    [01-24-2023 07:59:23 GMT]
(quota) received command 15    [01-24-2023 07:59:23 GMT]
(quota) closing back channel 1783da726ea831091c4cfda82525690f    [01-24-2023 07:59:23 GMT]
(cvmfs) stopping negative entry cache cleaner    [01-24-2023 07:59:23 GMT]
(quota) received command 15    [01-24-2023 07:59:23 GMT]
(quota) closing back channel 5c26a8b3f2fb6c2d975add471f9649d3    [01-24-2023 07:59:23 GMT]
(cache) unpinning / unloading all catalogs    [01-24-2023 07:59:23 GMT]
(quota) Unpin 5a71e8e80802b3d17cecdbaef03021fb2b2d55e3    [01-24-2023 07:59:23 GMT]
(sql) successfully finalized statement    [01-24-2023 07:59:23 GMT]
(quota) received command 4    [01-24-2023 07:59:23 GMT]
(sql) successfully finalized statement    [01-24-2023 07:59:23 GMT]
(sql) successfully finalized statement    [01-24-2023 07:59:23 GMT]
(sql) successfully finalized statement    [01-24-2023 07:59:23 GMT]
(sql) successfully finalized statement    [01-24-2023 07:59:23 GMT]
(sql) successfully finalized statement    [01-24-2023 07:59:23 GMT]
(sql) successfully finalized statement    [01-24-2023 07:59:23 GMT]
(sql) successfully finalized statement    [01-24-2023 07:59:23 GMT]
(sql) successfully finalized statement    [01-24-2023 07:59:23 GMT]
(sql) successfully finalized statement    [01-24-2023 07:59:23 GMT]
(sql) successfully finalized statement    [01-24-2023 07:59:23 GMT]
(sql) successfully finalized statement    [01-24-2023 07:59:23 GMT]
(sql) successfully finalized statement    [01-24-2023 07:59:23 GMT]
(sql) closing SQLite database '@16' (unlink: no)    [01-24-2023 07:59:23 GMT]
(download) download I/O thread terminated    [01-24-2023 07:59:23 GMT]
(download) download I/O thread terminated    [01-24-2023 07:59:23 GMT]
(quota) stopping cache manager (2)    [01-24-2023 07:59:23 GMT]
(quota) processing 5a71e8e80802b3d17cecdbaef03021fb2b2d55e3 (0)    [01-24-2023 07:59:23 GMT]
(quota) touching 5a71e8e80802b3d17cecdbaef03021fb2b2d55e3 (3112029671): 101    [01-24-2023 07:59:23 GMT]
(quota) processing 5a71e8e80802b3d17cecdbaef03021fb2b2d55e3 (3)    [01-24-2023 07:59:23 GMT]
(quota) contains 5a71e8e80802b3d17cecdbaef03021fb2b2d55e3 returns 1    [01-24-2023 07:59:23 GMT]
(quota) insert or replace 5a71e8e80802b3d17cecdbaef03021fb2b2d55e3, method 3: 101    [01-24-2023 07:59:23 GMT]
(monitor) stopping watchdog listener    [01-24-2023 07:59:23 GMT]
(quota) processing 5a71e8e80802b3d17cecdbaef03021fb2b2d55e3 (4)    [01-24-2023 07:59:23 GMT]
(quota) unpinning 5a71e8e80802b3d17cecdbaef03021fb2b2d55e3: 101    [01-24-2023 07:59:23 GMT]
(monitor) monitor stopped    [01-24-2023 07:59:23 GMT]
(cvmfs) cvmfs_destroy    [01-24-2023 07:59:23 GMT]
(monitor) stopping watchdog listener    [01-24-2023 07:59:23 GMT]
(monitor) monitor stopped    [01-24-2023 07:59:23 GMT]
7

Hi Jose,

hmm, I cannot say for sure, but it could indeed be that the problem is outside of cvmfs. Do let us know if the problem persists!

8

That seems to be case. I believe the problem was that cgroups was corrupted on the host. Rebooting it fixed it.
Thanks a lot for everything.

1 Like