Hello,
We are trying to install our own CVMFS Stratum 0 server on Ubuntu 20.04
When we start a transaction we have a crash:
# dpkg -s cvmfs-server | grep Version:
Version: 2.10.0~1+ubuntu20.04
# cvmfs_server transaction
(unexpected termination) Autofs on /cvmfs has to be disabled
Stacktrace:
/usr/bin/cvmfs_publish(+0x43bb3) [0x56325962fbb3]
/usr/bin/cvmfs_publish(+0x3045d) [0x56325961c45d]
/usr/bin/cvmfs_publish(+0x4069d) [0x56325962c69d]
/usr/bin/cvmfs_publish(+0x1522d) [0x56325960122d]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7ff9a7fb5083]
/usr/bin/cvmfs_publish(+0x1543e) [0x56325960143e]
We enabled logs with CVMFS_DEBUGLOG but there is nothing
Here is the result with CVMFS_SERVER_DEBUG:
# CVMFS_SERVER_DEBUG=3 cvmfs_server transaction
(cvmfs) Parsing config file /etc/cvmfs/server.local [02-07-2023 15:10:05 UTC]
(cvmfs) Parsing config file /etc/cvmfs/repositories.d/ReproVIP.egi.eu/server.conf [02-07-2023 15:10:05 UTC]
(cvmfs) execve'd /bin/sh (PID: 20137) [02-07-2023 15:10:05 UTC]
(cvmfs) Parsing config file /etc/cvmfs/repositories.d/ReproVIP.egi.eu/replica.conf [02-07-2023 15:10:05 UTC]
(cvmfs) current credentials uid 0 gid 0 euid 0 egid 0, switching to 0 0 (temp: 0) [02-07-2023 15:10:05 UTC]
(cvmfs) switch credentials result 0 (0) [02-07-2023 15:10:05 UTC]
(unexpected termination) Autofs on /cvmfs has to be disabled
Stacktrace:
/usr/bin/cvmfs_publish_debug(+0xac119) [0x558c99cc4119]
/usr/bin/cvmfs_publish_debug(+0x84248) [0x558c99c9c248]
/usr/bin/cvmfs_publish_debug(+0xa42f6) [0x558c99cbc2f6]
/usr/bin/cvmfs_publish_debug(+0xad470) [0x558c99cc5470]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f3690c92083]
/usr/bin/cvmfs_publish_debug(+0x3ddee) [0x558c99c55dee]
Thanks in advance for your help
Best regards,
Hi Emmanuel,
thanks for the report! Just to exclude this first: Is autofs actually running on your system / what’s the output of sudo systemctl status autofs?
Note that running server commands when autofs is enabled can get you in a broken state which can only be resolved by restarting the entire machine.
Hi Valentin,
Thanks for your reply. We restarted our server, autofs is running:
● autofs.service - Automounts filesystems on demand
Loaded: loaded (/lib/systemd/system/autofs.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-02-08 07:07:58 UTC; 4min 36s ago
Docs: man:autofs(8)
Process: 631 ExecStart=/usr/sbin/automount $OPTIONS --pid-file /var/run/autofs.pid (code=exited, status=0/SUCCESS)
Main PID: 740 (automount)
Tasks: 4 (limit: 4676)
Memory: 6.9M
CGroup: /system.slice/autofs.service
└─740 /usr/sbin/automount --pid-file /var/run/autofs.pid
Feb 08 07:07:56 cvmfs-reprovip systemd[1]: Starting Automounts filesystems on demand...
Feb 08 07:07:58 cvmfs-reprovip systemd[1]: Started Automounts filesystems on demand.
# cvmfs_config probe
Probing /cvmfs/ReproVIP.egi.eu... OK
# cvmfs_server transaction
(unexpected termination) Autofs on /cvmfs has to be disabled
Stacktrace:
/usr/bin/cvmfs_publish(+0x43bb3) [0x564b4013abb3]
/usr/bin/cvmfs_publish(+0x3045d) [0x564b4012745d]
/usr/bin/cvmfs_publish(+0x4069d) [0x564b4013769d]
/usr/bin/cvmfs_publish(+0x1522d) [0x564b4010c22d]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7ff84f743083]
/usr/bin/cvmfs_publish(+0x1543e) [0x564b4010c43e]
It crashes also when we stop autofs. Do you mean we have to stop autofs when doing a transaction ?
autofs is inactive(dead) after the crash and we get this:
# systemctl --no-pager status autofs
● autofs.service - Automounts filesystems on demand
Loaded: loaded (/lib/systemd/system/autofs.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Wed 2023-02-08 07:15:52 UTC; 6min ago
Docs: man:autofs(8)
Process: 631 ExecStart=/usr/sbin/automount $OPTIONS --pid-file /var/run/autofs.pid (code=exited, status=0/SUCCESS)
Main PID: 740 (code=exited, status=0/SUCCESS)
Feb 08 07:15:48 cvmfs-reprovip cvmfs2[1261]: (cvmfs-config.cern.ch) PANIC: /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/ubuntu2004/b…istener.cc : 105
cache manager disappeared, aborting
Feb 08 07:15:48 cvmfs-reprovip systemd[1]: Stopping Automounts filesystems on demand...
Feb 08 07:15:48 cvmfs-reprovip cvmfs2[1365]: (ReproVIP.egi.eu) watchdog: received unexpected signal 15 from PID 1 / UID 0
Feb 08 07:15:48 cvmfs-reprovip cvmfs2[1261]: (cvmfs-config.cern.ch) watchdog disappeared, disabling stack trace reporting (revents: 16 / 8|16|32)
Feb 08 07:15:48 cvmfs-reprovip cvmfs2[1261]: (cvmfs-config.cern.ch) PANIC: /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/ubuntu2004/b…monitor.cc : 522
watchdog disappeared, aborting
Feb 08 07:15:48 cvmfs-reprovip cvmfs2[1361]: (ReproVIP.egi.eu) watchdog disappeared, disabling stack trace reporting (revents: 16 / 8|16|32)
Feb 08 07:15:48 cvmfs-reprovip cvmfs2[1361]: (ReproVIP.egi.eu) PANIC: /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/ubuntu2004/build/…monitor.cc : 522
watchdog disappeared, aborting
Feb 08 07:15:51 cvmfs-reprovip automount[740]: umount_autofs_indirect: ask umount returned busy /cvmfs
Feb 08 07:15:52 cvmfs-reprovip systemd[1]: autofs.service: Succeeded.
Feb 08 07:15:52 cvmfs-reprovip systemd[1]: Stopped Automounts filesystems on demand.
Cheers,
After another reboot this time it works when we stop autofs before a transaction.
Cheers,
OK! On the publisher autofs should be disabled altogether, not just for the transactions. Let me confirm this and review the documentation on setting up a statum 0, I see that this is not mentioned there but probably should be.
Ok thanks ! The server works when autofs is stopped and disabled. And the client is able to receive files.
But there is this warning on the server:
# cvmfs_config chksetup
Warning: autofs service is not running
Warning: debug mode is on for ReproVIP.egi.eu
Cheers,
cvmfs_config chksetup checks the setup for the client, not the server I think, so this can be ignored.
Ok Thanks a lot for your help !
dwd
February 8, 2023, 7:20pm
9
It’s OK to have autofs enabled on a server, as long as doesn’t control /cvmfs.
vavolkl
February 8, 2023, 8:58pm
10
Thanks Dave for the clarification!