How to estimate cache turnover time and detect cache thrashing

Hello,

It would be useful to know the age of the least recently used file in a client cache, as an estimator of how much turnover is occurring and potential indicator of cache thrashing (given the steady state operating assumption that a cache will always be full).

Some kind of find /var/lib/cvmfs/shared to look for the files with oldest access time could work, assuming that CVMFS evicts the LRU file (?). However if that is the case, I would think CVMFS must have some internal tracking of the last accessed time of files in order to evict the LRU ones? I had a quick look at extended attributes and the sqlite cachedb but nothing stood out to me.

Thanks!

Hi Ryan,

cvmfs evicts according to LRU, so access time should work (if it’s not disabled in the mount options of the file system hosting the cache). Cvmfs itself does not keep timestamps but the access sequence only in the sqlite cachedb.

In order to check the cache turnorver, you can also use

cvmfs_talk -i <reponame> cleanup rate <minutes>

That number tells you how many times in the last x minutes the cache ran full and triggered the eviction of half of it.

Cheers,
Jakob

There’s also the ncleanup24 extended attribute which can be read unprivileged and lists the number of cache cleanups in the last 24 hours.

Dave

Thanks Jakob and Dave!
So IIUC those counters are for the “aggressive” (synchronous) cleanups required to immediately make space for a new chunk to avoid an IO error for the client?
If I recall correctly there is also asynchronous background cleanup?

Looks like it must be the 3rd field of the cachedb sqlite3 file which gives the access sequence:

Yes, the 3rd field is the access sequence number.

The counters are for the regular cleanup, too. The regular cleanup mechanism kicks in when the cache runs full and it cleans it until 50% are free again.