CVMFS client stops after a few minutes

Client
1

Hello,

We want to install CVMFS on a cluster, nodes are running Ubuntu 20.04 LTS.
Currently, we are making some preliminary tests on a single node.

We have installed CVMFS as explained in the documentation.
We are able to get the repositories we need when we configure it and execute cvmfs_probe.
But after 5 minutes, repositories are unmounted for unknown reasons.

We set the CVMFS_DEBUGLOG option to get further details.
Here are the latest lines of the log that draw our attention:

(download) Verify downloaded url /.cvmfspublished, proxy DIRECT (curl error 0)    [12-07-2022 05:48:26 PST]
(cache) remote checksum is 1edb3e45aa512b75784c3c76cf7a9006b9785f3e    [12-07-2022 05:48:26 PST]
(cvmfs) catalog up to date (could be offline mode)    [12-07-2022 05:48:26 PST]
(cvmfs) negative entry cleaner: pruning    [12-07-2022 05:49:24 PST]
(cvmfs) negative entry cleaner: pruning    [12-07-2022 05:49:25 PST]
(cvmfs) negative entry cleaner: pruning    [12-07-2022 05:50:24 PST]
(cvmfs) negative entry cleaner: pruning    [12-07-2022 05:50:25 PST]
(cvmfs) negative entry cleaner: pruning    [12-07-2022 05:51:24 PST]
(cvmfs) negative entry cleaner: pruning    [12-07-2022 05:51:26 PST]
(cvmfs) cvmfs_getattr (stat) for inode: 256    [12-07-2022 05:52:08 PST]
(lru) lookup inode --> dirent: 256 (hit)    [12-07-2022 05:52:08 PST]
(talk) terminating talk thread (fd -1, errno 22)    [12-07-2022 05:52:08 PST]
(talk) talk thread stopped    [12-07-2022 05:52:08 PST]
(cvmfs) stopping remount trigger    [12-07-2022 05:52:08 PST]
(cvmfs) stopping dentry invalidator thread    [12-07-2022 05:52:08 PST]
(quota) stopping unpin listener for 364c202d-8e2b-1b49-e08a-63142a271b5f-unpin    [12-07-2022 05:52:08 PST]
(quota) stopping cache manager watchdog for 364c202d-8e2b-1b49-e08a-63142a271b5f-watchdog    [12-07-2022 05:52:08 PST]
(cvmfs) stopping negative entry cache cleaner    [12-07-2022 05:52:08 PST]
(cache) unpinning / unloading all catalogs    [12-07-2022 05:52:08 PST]
(quota) Unpin 1edb3e45aa512b75784c3c76cf7a9006b9785f3e    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) closing SQLite database '@20' (unlink: no)    [12-07-2022 05:52:08 PST]
(download) download I/O thread terminated    [12-07-2022 05:52:08 PST]
(download) download I/O thread terminated    [12-07-2022 05:52:08 PST]
(cvmfs) cvmfs_getattr (stat) for inode: 256    [12-07-2022 05:52:08 PST]
(lru) lookup inode --> dirent: 256 (hit)    [12-07-2022 05:52:08 PST]
(talk) terminating talk thread (fd -1, errno 22)    [12-07-2022 05:52:08 PST]
(talk) talk thread stopped    [12-07-2022 05:52:08 PST]
(cvmfs) stopping remount trigger    [12-07-2022 05:52:08 PST]
(cvmfs) stopping dentry invalidator thread    [12-07-2022 05:52:08 PST]
(quota) stopping unpin listener for cb23a38e-fd56-0842-b59a-35042a6c4884-unpin    [12-07-2022 05:52:08 PST]
(quota) stopping cache manager watchdog for cb23a38e-fd56-0842-b59a-35042a6c4884-watchdog    [12-07-2022 05:52:08 PST]
(cvmfs) stopping negative entry cache cleaner    [12-07-2022 05:52:08 PST]
(cache) unpinning / unloading all catalogs    [12-07-2022 05:52:08 PST]
(quota) Unpin cd503e6ba67b96c15c95255722307a534d89cdab    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) successfully finalized statement    [12-07-2022 05:52:08 PST]
(sql) closing SQLite database '@18' (unlink: no)    [12-07-2022 05:52:08 PST]
(download) download I/O thread terminated    [12-07-2022 05:52:08 PST]
(download) download I/O thread terminated    [12-07-2022 05:52:08 PST]

It seems like an unknown error occurs and forces CVMFS to stop at some point:

# here?
(download) Verify downloaded url /.cvmfspublished, proxy DIRECT (curl error 0)    [12-07-2022 05:48:26 PST]
# or here?
(talk) terminating talk thread (fd -1, errno 22)    [12-07-2022 05:52:08 PST]

Have you ever encountered such an issue?
Should you need any more details, please let me know.

Thanks

2

I am not sure this is an issue: repositories are routinely unmounted when not in use. It could be an issue if they are not re-mounted when used again.

3

Oops indeed, I forgot to mention this point: repositories are not re-mounted when used again.
After 5 minutes, if we try to list repositories, we get no result:

$ ls -l /cvmfs/
total 0
4

That is normal automount behavior. You need to try to access a specific repository name and it will appear.

5

Indeed, accessing /cvmfs/<repo1> triggers the automount feature.
Thanks a lot and sorry for the disturbance, there is no issue.