Cache not cleaned

Client
1

Hi,
I presume this situation should not arise if everything is working correctly, right?

CVMFS_CACHE_BASE=/cvmfscache
CVMFS_QUOTA_LIMIT=10240
$ df -h /cvmfscache
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p1 15G 14G 12K 100% /cvmfscache

This follows the 20%+1GB grace amount of space, but somehow the cache is not flushed. The cache is shared.

I don`t have root access to the machine, so investigation is very limited. What happens if there is a very big file used, e.g.
-rw-r–r–. 1 cvmfs cvmfs 9.4G Aug 18 2020 /cvmfs/unpacked.cern.ch/registry.hub.docker.com/atlas/athena:21.0.15.sw6-0/usr/database/DBRelease/current/sqlite200/ALLP200.db
?

Another question: on a different node we had (I think bad) config
CVMFS_SHARED_CACHE=no
CVMFS_QUOTA_LIMIT=40000
Does this try to give 40GB to each repo and thus not clean the cache when full? Here

df -h /cvmfscache/
Filesystem Size Used Avail Use% Mounted on
/dev/md0p2 46G 44G 2,5M 100% /cvmfscache

Cheers,
Rod.

2

Hi Rod,

somehow the cache is not flushed

no, this can unfortunately happen if there are long-running processes keeping the files open. In that case we will have removed the cache file already, but the disk space is not released until the file is closed. There is no easy solution here except sizing the cache according to workloads. Maybe the 20% grace amount is not the right recommendation, in particular when the cache size is small overall.

What happens if there is a very big file used

CVMFS splits large files into smaller chunks internally, so this is equivalent to reading many small files (if the whole file is read, for a partial read, only the required chunks are read).

Trying to do this on a too small cache will usually just result in cache thrashing and bad performance. When the partition gets full due to non-reclaimed space there will be I/O errors on reads. If the file catalogs can no longer be loaded, also directory listings will fail.

CVMFS_SHARED_CACHE=no
CVMFS_QUOTA_LIMIT=40000
Does this try to give 40GB to each repo and thus not clean the cache when full?

Yes, the quota limit applies to each cache instance individually. Here I’d recommend to use a shared cache.

Cheers,
Valentin

3

The cache does not clean even with shared cache and more grace amount of space.

df -h /cvmfscache/
Filesystem Size Used Avail Use% Mounted on
/dev/md0p2 46G 44G 2,1M 100% /cvmfscache
CVMFS_CACHE_BASE=/cvmfscache/
CVMFS_SHARED_CACHE=yes
CVMFS_QUOTA_LIMIT=35000

As I mentioned I have no root access, so would need to give the admin some more or less cut’n’paste debug instructions if at all possible.

Cheers,
Rod.

4

This command would probably help:

sudo lsof|grep /var/lib/cvmfs.*deleted|grep -v FIFO

If they have set CVMFS_CACHE_BASE use that value instead of /var/lib/cvmfs.