Garbage collection failed (9)

jcaballe · June 19, 2023, 12:00pm

Hi,

after modifying the configuration for the repositories at our Stratum-0 to enable GC, and re-publishing them, all the repos with the same domain name have failed.

This is how the config looks [1]
This is what I get in the console [2]
And this is what I get in the logs [3]

The fact that only affects a single domain name, and all the repos in that domain, is quite interesting. Does that suggest that there is something wrong with the public key?

Any comment/question/suggestion/idea is more than welcome.

Cheers,
Jose

[1]

[root@stratum-0 ~]# cat /etc/cvmfs/repositories.d/myrepo.domain.name/server.conf
# Created by cvmfs_server.
CVMFS_CREATOR_VERSION=143
CVMFS_REPOSITORY_NAME=myrepo.domain.name
CVMFS_REPOSITORY_TYPE=stratum0
CVMFS_USER=cvmfs
CVMFS_UNION_DIR=/cvmfs/myrepo.domain.name
CVMFS_SPOOL_DIR=/var/spool/cvmfs/myrepo.domain.name
CVMFS_STRATUM0=http://localhost/cvmfs/myrepo.domain.name
CVMFS_UPSTREAM_STORAGE=local,/srv/cvmfs/myrepo.domain.name/data/txn,/srv/cvmfs/myrepo.domain.name
CVMFS_USE_FILE_CHUNKING=true
CVMFS_MIN_CHUNK_SIZE=4194304
CVMFS_AVG_CHUNK_SIZE=8388608
CVMFS_MAX_CHUNK_SIZE=16777216
CVMFS_UNION_FS_TYPE=overlayfs
CVMFS_HASH_ALGORITHM=sha1
CVMFS_COMPRESSION_ALGORITHM=default
CVMFS_EXTERNAL_DATA=false
CVMFS_AUTO_TAG=false
CVMFS_AUTO_TAG_TIMESPAN=""
CVMFS_GARBAGE_COLLECTION=true
CVMFS_AUTO_REPAIR_MOUNTPOINT=true
CVMFS_AUTOCATALOGS=true
CVMFS_ASYNC_SCRATCH_CLEANUP=true
CVMFS_PRINT_STATISTICS=false
CVMFS_IGNORE_XDIR_HARDLINKS=true
CVMFS_AUTO_GC=true

[2]

[root@stratum-0 ~]# cvmfs_server transaction myrepo.domain.name
[root@stratum-0 ~]# cvmfs_server publish myrepo.domain.name
Processing changes...
Waiting for upload of files before committing...
Committing file catalogs...
Wait for all uploads to finish
Exporting repository manifest
Statistics stored at: /var/spool/cvmfs/myrepo.domain.name/stats.db
Tagging myrepo.domain.name
Flushing file system buffers
Signing new manifest
Running automatic garbage collection
Garbage collection failed (9)

[3]

Jun 19 12:47:31 stratum-0 cvmfs_publish: (myrepo.domain.name) opened transaction
Jun 19 12:47:31 stratum-0 cvmfs_publish: Trying to remount /cvmfs/myrepo.domain.name read/write... success
Jun 19 12:47:41 stratum-0 cvmfs_server: (myrepo.domain.name) started publishing
Jun 19 12:47:41 stratum-0 cvmfs_swissknife: Parameter CVMFS_STATISTICS_DB was not set in the repository configuration file. Using default value: /var/spool/cvmfs/myrepo.domain.name/stats.db
Jun 19 12:47:41 stratum-0 cvmfs_swissknife: Parameter CVMFS_STATS_DB_DAYS_TO_KEEP was not set in the repository configuration file. Using default value: 365
Jun 19 12:47:42 stratum-0 cvmfs_server: (myrepo.domain.name) failed to publish
Jun 19 12:47:53 stratum-0 cvmfs_publish: (myrepo.domain.name) aborting transaction
Jun 19 12:47:53 stratum-0 cvmfs_publish: (myrepo.domain.name) attempting mountpoint repair (2)
Jun 19 12:47:53 stratum-0 cvmfs_publish: Trying to unmount /cvmfs/myrepo.domain.name... success
Jun 19 12:47:53 stratum-0 cvmfs_publish: Trying to unmount /var/spool/cvmfs/myrepo.domain.name/rdonly... success
Jun 19 12:47:53 stratum-0 cvmfs2: (myrepo.domain.name) CernVM-FS: unmounted /var/spool/cvmfs/myrepo.domain.name/rdonly (myrepo.domain.name)
Jun 19 12:47:53 stratum-0 systemd: Mounting /var/spool/cvmfs/myrepo.domain.name/rdonly...
Jun 19 12:47:53 stratum-0 mount: CernVM-FS: running with suid support
Jun 19 12:47:53 stratum-0 mount: CernVM-FS: loading Fuse module... done
Jun 19 12:47:53 stratum-0 systemd: Mounted /var/spool/cvmfs/myrepo.domain.name/rdonly.
Jun 19 12:47:53 stratum-0 cvmfs_publish: Trying to mount /var/spool/cvmfs/myrepo.domain.name/rdonly... success
Jun 19 12:47:53 stratum-0 mount: CernVM-FS: mounted cvmfs on /var/spool/cvmfs/myrepo.domain.name/rdonly
Jun 19 12:47:53 stratum-0 cvmfs2: (myrepo.domain.name) CernVM-FS: linking /var/spool/cvmfs/myrepo.domain.name/rdonly to repository myrepo.domain.name
Jun 19 12:47:53 stratum-0 systemd: Mounting /cvmfs/myrepo.domain.name...
Jun 19 12:47:53 stratum-0 systemd: Mounted /cvmfs/myrepo.domain.name.
Jun 19 12:47:53 stratum-0 cvmfs_publish: Trying to mount /cvmfs/myrepo.domain.name... success
Jun 19 12:47:53 stratum-0 cvmfs_publish: Trying to remount /cvmfs/myrepo.domain.name read/write... success
Jun 19 12:47:53 stratum-0 cvmfs_publish: finished mountpoint repair (0)
Jun 19 12:47:53 stratum-0 cvmfs_publish: Trying to unmount /cvmfs/myrepo.domain.name... success
Jun 19 12:47:53 stratum-0 cvmfs_publish: Trying to unmount /var/spool/cvmfs/myrepo.domain.name/rdonly... success
Jun 19 12:47:53 stratum-0 cvmfs2: (myrepo.domain.name) CernVM-FS: unmounted /var/spool/cvmfs/myrepo.domain.name/rdonly (myrepo.domain.name)
Jun 19 12:47:53 stratum-0 cvmfs_publish: Trying to wipe out /var/spool/cvmfs/myrepo.domain.name/scratch/current (async cleanup)... success
Jun 19 12:47:53 stratum-0 systemd: Mounting /var/spool/cvmfs/myrepo.domain.name/rdonly...
Jun 19 12:47:53 stratum-0 mount: CernVM-FS: running with suid support
Jun 19 12:47:53 stratum-0 mount: CernVM-FS: loading Fuse module... done
Jun 19 12:47:53 stratum-0 systemd: Mounted /var/spool/cvmfs/myrepo.domain.name/rdonly.
Jun 19 12:47:53 stratum-0 cvmfs_publish: Trying to mount /var/spool/cvmfs/myrepo.domain.name/rdonly... success
Jun 19 12:47:53 stratum-0 mount: CernVM-FS: mounted cvmfs on /var/spool/cvmfs/myrepo.domain.name/rdonly
Jun 19 12:47:53 stratum-0 cvmfs2: (myrepo.domain.name) CernVM-FS: linking /var/spool/cvmfs/myrepo.domain.name/rdonly to repository myrepo.domain.name
Jun 19 12:47:53 stratum-0 systemd: Mounting /cvmfs/myrepo.domain.name...
Jun 19 12:47:53 stratum-0 systemd: Mounted /cvmfs/myrepo.domain.name.
Jun 19 12:47:53 stratum-0 cvmfs_publish: Trying to mount /cvmfs/myrepo.domain.name... success
Jun 19 12:48:01 stratum-0 systemd: Started Session 48001 of user root.

vavolkl · June 20, 2023, 9:35am

Hi Jose,

indeed there could be one issue that’s not very intuitive - Could you try disabling the automatic garbage collection and run it manually for the first few publications? Also, running the publication in debug mode could be helpful:

CVMFS_SERVER_DEBUG=3 cvmfs_server publish

jcaballe · June 20, 2023, 11:53am

I will try and let you know.

jcaballe · June 20, 2023, 2:36pm

Hi,

First, running transaction/publish with the old configuration (w/o GC) works fine.

I gave it another try with GC on, on DEBUG LEVEL 3.
If you need it, I could copy and paste the entire output (200+ lines). But the very end of it is the same, no extra info:

# CVMFS_SERVER_DEBUG=3 cvmfs_server publish myrepo.domain.name
...
...
(sql) successfully finalized statement    [06-20-2023 15:21:46 BST]
(sql) successfully finalized statement    [06-20-2023 15:21:46 BST]
(sql) closing SQLite database '/var/spool/cvmfs/myrepo.domain.name/tmp/history.mJG6jJ' (unlink: no)    [06-20-2023 15:21:46 BST]
(spooler) File '/var/spool/cvmfs/myrepo.domain.name/tmp/history.mJG6jJ' processed (bulk hash: de09a4d072d8b672bfa2728306191334e7ccb33e suffix: H)
(cvmfs) exported manifest (14) with new history 'de09a4d072d8b672bfa2728306191334e7ccb33e'
Flushing file system buffers
Signing new manifest
Running automatic garbage collection
Garbage collection failed (9)

jcaballe · June 20, 2023, 3:02pm

Interesting… I have just removed the repository and created it again (with -g flag). Now everything seems to work fine.

dwd · June 21, 2023, 6:37pm

Jose,

First, I hope you have made sure that these repositories actually delete files frequently. Otherwise it is not usually worth the resources to run it, especially if there are a lot of files in the repository.

Secondly, I always recommend disabling automatic garbage collection on stratum zeros, because it can take a very long time to run and while it is running, no further publishes can happen. That can be very annoying to the repository owners. Instead, I ask the repository owners to run it themselves at times that are convenient to them, at least once a week. Then I run cvmfs-servermon on the stratum zero and watch for warnings about it having been too long since gc had been run, and when
that happens bug the repository owners to set up a process to make sure it happens regularly.

Dave

system · July 6, 2023, 6:37pm

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.