Cvmfs 2.8.1 client crash

Client
1

Hi,
We have intermittent crash of cvmfs client 2.8.1.
We have been able to gather the following stack trace:





Signal: 11, errno: 2, version: 2.8.1, PID: 16228

Executable path: /usr/bin/cvmfs2


Thread 21 (Thread 0x7fa064b47700 (LWP 16245)):

#0  0x00007fa06e0fb9dd in accept () from /lib64/libpthread.so.0

#1  0x00007fa06d2c2b70 in loader::loader_talk::MainTalk (data=)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/loader_talk.cc:60

#2  0x00007fa06e0f4ea5 in start_thread () from /lib64/libpthread.so.0

#3  0x00007fa06d5fd9fd in clone () from /lib64/libc.so.6


Thread 20 (Thread 0x7fa068b4f700 (LWP 36231)):

#0  0x00007fa06d5f2ccd in poll () from /lib64/libc.so.6

#1  0x00007fa06aebcd3f in poll (__timeout=-1, __nfds=2, __fds=0x7fa068b4eeb8)

at /usr/include/bits/poll2.h:46

#2  Watchdog::MainWatchdogListener (data=0x7f9ffc2f6040)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/monitor.cc:486

#3  0x00007fa06e0f4ea5 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fa06d5fd9fd in clone () from /lib64/libc.so.6


Thread 19 (Thread 0x7fa066b4b700 (LWP 36232)):

#0  0x00007fa06e0fb75d in read () from /lib64/libpthread.so.0

#1  0x00007fa06aee1e5a in read (__nbytes=1, __buf=0x7fa066b4ae8f, __fd=17)

at /usr/include/bits/unistd.h:44

#2  ReadPipe (fd=17, buf=buf@entry=0x7fa066b4ae8f, nbyte=nbyte@entry=1)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/util/posix.cc:533

#3  0x00007fa06aefbf36 in FuseInvalidator::MainInvalidator (

data=0x7f9ffc2f5ee0)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/fuse_evict.cc:111

#4  0x00007fa06e0f4ea5 in start_thread () from /lib64/libpthread.so.0

#5  0x00007fa06d5fd9fd in clone () from /lib64/libc.so.6


Thread 18 (Thread 0x7fa05e34a700 (LWP 36233)):

#0  0x00007fa06d5f2ccd in poll () from /lib64/libc.so.6

#1  0x00007fa06aefcd97 in poll (__timeout=, __nfds=1,

__fds=0x7fa05e349ed0) at /usr/include/bits/poll2.h:46

#2  FuseRemounter::MainRemountTrigger (data=0x7f9ffc2f5e60)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/fuse_remount.cc:210

#3  0x00007fa06e0f4ea5 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fa06d5fd9fd in clone () from /lib64/libc.so.6


Thread 17 (Thread 0x7fa06734c700 (LWP 36234)):

#0  0x00007fa06d5f2ccd in poll () from /lib64/libc.so.6

#1  0x00007fa06aea21c7 in poll (__timeout=60000, __nfds=1,

__fds=0x7fa06734bed0) at /usr/include/bits/poll2.h:46

#2  glue::NentryTracker::MainCleaner (data=0x7f9ffc2f1670)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/glue_buffer.cc:187

#3  0x00007fa06e0f4ea5 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fa06d5fd9fd in clone () from /lib64/libc.so.6


Thread 16 (Thread 0x7fa06834e700 (LWP 36235)):

#0  0x00007fa06d5f2ccd in poll () from /lib64/libc.so.6

#1  0x00007fa06ae983ea in poll (__timeout=,

__nfds=, __fds=)

at /usr/include/bits/poll2.h:46

#2  download::DownloadManager::MainDownload (data=0x7f9ffc02b4e0)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/download.cc:513

#3  0x00007fa06e0f4ea5 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fa06d5fd9fd in clone () from /lib64/libc.so.6


Thread 15 (Thread 0x7fa067b4d700 (LWP 36236)):

#0  0x00007fa06d5f2ccd in poll () from /lib64/libc.so.6

#1  0x00007fa06ae983ea in poll (__timeout=,

__nfds=, __fds=)

at /usr/include/bits/poll2.h:46

#2  download::DownloadManager::MainDownload (data=0x7f9ffc02fb70)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/download.cc:513

#3  0x00007fa06e0f4ea5 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fa06d5fd9fd in clone () from /lib64/libc.so.6


Thread 14 (Thread 0x7fa06634a700 (LWP 36237)):

#0  0x00007fa06d5f2ccd in poll () from /lib64/libc.so.6

#1  0x00007fa06af01bce in poll (__timeout=-1, __nfds=2, __fds=0x7fa0500008c0)

at /usr/include/bits/poll2.h:46

#2  quota::MainWatchdogListener (data=0x7f9ffc001060)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/quota_listener.cc:90

#3  0x00007fa06e0f4ea5 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fa06d5fd9fd in clone () from /lib64/libc.so.6


Thread 13 (Thread 0x7fa065b49700 (LWP 36238)):

#0  0x00007fa06d5f2ccd in poll () from /lib64/libc.so.6

#1  0x00007fa06af020eb in poll (__timeout=-1, __nfds=2, __fds=0x7fa0580008c0)

at /usr/include/bits/poll2.h:46

#2  quota::MainUnpinListener (data=0x7f9ffc01da40)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/quota_listener.cc:48

#3  0x00007fa06e0f4ea5 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fa06d5fd9fd in clone () from /lib64/libc.so.6


Thread 12 (Thread 0x7fa065348700 (LWP 36239)):

#0  0x00007fa06e0fb9dd in accept () from /lib64/libpthread.so.0

#1  0x00007fa06af03f27 in TalkManager::MainResponder (data=0x7f9ffc2f6110)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/talk.cc:173

#2  0x00007fa06e0f4ea5 in start_thread () from /lib64/libpthread.so.0

#3  0x00007fa06d5fd9fd in clone () from /lib64/libc.so.6


Thread 11 (Thread 0x7fa05d348700 (LWP 32390)):

#0  0x00007fa06e0fb75d in read () from /lib64/libpthread.so.0

#1  0x00007fa06c9b5d62 in fuse_kern_chan_receive () from /lib64/libfuse.so.2

#2  0x00007fa06c9b6d59 in fuse_ll_receive_buf () from /lib64/libfuse.so.2

#3  0x00007fa06c9b637e in fuse_do_work () from /lib64/libfuse.so.2

#4  0x00007fa06e0f4ea5 in start_thread () from /lib64/libpthread.so.0

#5  0x00007fa06d5fd9fd in clone () from /lib64/libc.so.6


Thread 10 (Thread 0x7f9fea7fc700 (LWP 34719)):

#0  0x00007fa06e0fb75d in read () from /lib64/libpthread.so.0

#1  0x00007fa06c9b5d62 in fuse_kern_chan_receive () from /lib64/libfuse.so.2

#2  0x00007fa06c9b6d59 in fuse_ll_receive_buf () from /lib64/libfuse.so.2

#3  0x00007fa06c9b637e in fuse_do_work () from /lib64/libfuse.so.2

#4  0x00007fa06e0f4ea5 in start_thread () from /lib64/libpthread.so.0

#5  0x00007fa06d5fd9fd in clone () from /lib64/libc.so.6


Thread 9 (Thread 0x7f9feaffd700 (LWP 45624)):

#0  0x00007fa06e0fb75d in read () from /lib64/libpthread.so.0

#1  0x00007fa06c9b5d62 in fuse_kern_chan_receive () from /lib64/libfuse.so.2

#2  0x00007fa06c9b6d59 in fuse_ll_receive_buf () from /lib64/libfuse.so.2

#3  0x00007fa06c9b637e in fuse_do_work () from /lib64/libfuse.so.2

#4  0x00007fa06e0f4ea5 in start_thread () from /lib64/libpthread.so.0

#5  0x00007fa06d5fd9fd in clone () from /lib64/libc.so.6


Thread 8 (Thread 0x7fa02ffff700 (LWP 18711)):

#0  0x00007fa06e0fb75d in read () from /lib64/libpthread.so.0

#1  0x00007fa06c9b5d62 in fuse_kern_chan_receive () from /lib64/libfuse.so.2

#2  0x00007fa06c9b6d59 in fuse_ll_receive_buf () from /lib64/libfuse.so.2

#3  0x00007fa06c9b637e in fuse_do_work () from /lib64/libfuse.so.2

#4  0x00007fa06e0f4ea5 in start_thread () from /lib64/libpthread.so.0

#5  0x00007fa06d5fd9fd in clone () from /lib64/libc.so.6


Thread 7 (Thread 0x7fa00effd700 (LWP 18714)):

#0  0x00007fa06e0fb75d in read () from /lib64/libpthread.so.0

#1  0x00007fa06c9b5d62 in fuse_kern_chan_receive () from /lib64/libfuse.so.2

#2  0x00007fa06c9b6d59 in fuse_ll_receive_buf () from /lib64/libfuse.so.2

#3  0x00007fa06c9b637e in fuse_do_work () from /lib64/libfuse.so.2

#4  0x00007fa06e0f4ea5 in start_thread () from /lib64/libpthread.so.0

#5  0x00007fa06d5fd9fd in clone () from /lib64/libc.so.6


Thread 6 (Thread 0x7fa05ffff700 (LWP 18715)):

#0  0x00007fa06d5f4a33 in select () from /lib64/libc.so.6

#1  0x00007fa06aee3233 in SafeSleepMs (ms=ms@entry=100)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/util/posix.cc:1730

#2  0x00007fa06aebca5b in Watchdog::SendTrace (sig=11,

siginfo=, context=)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/monitor.cc:317

#3  

#4  0x00007fa06d655596 in __memcpy_ssse3_back () from /lib64/libc.so.6

#5  0x00007fa06dea3910 in std::string::_Rep::_M_clone(std::allocator const&, unsigned long) () from /lib64/libstdc++.so.6

#6  0x00007fa06dea3994 in std::string::reserve(unsigned long) ()

from /lib64/libstdc++.so.6

#7  0x00007fa06dea3a58 in std::string::append(std::string const&) ()

from /lib64/libstdc++.so.6

#8  0x00007fa06aeb661e in ChunkListMagicXattr::PrepareValueFenced (

this=0x7f9ffc2f4690)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/magic_xattr.cc:174

#9  0x00007fa06aef745f in cvmfs::cvmfs_getxattr (req=0x7fa03c000dd0,

ino=, name=, size=128)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/cvmfs.cc:1366

#10 0x00007fa06d2bf086 in loader::stub_getxattr (req=0x7fa03c000dd0,

ino=54122196, name=0x7f9ff8017480 “user.chunk_list”, size=128)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/loader.cc:303

#11 0x00007fa06c9b9b6b in fuse_ll_process_buf () from /lib64/libfuse.so.2

#12 0x00007fa06c9b6401 in fuse_do_work () from /lib64/libfuse.so.2

#13 0x00007fa06e0f4ea5 in start_thread () from /lib64/libpthread.so.0

#14 0x00007fa06d5fd9fd in clone () from /lib64/libc.so.6


Thread 5 (Thread 0x7fa02dffb700 (LWP 24015)):

#0  0x00007fa06e0fb75d in read () from /lib64/libpthread.so.0

#1  0x00007fa06aee1e5a in read (__nbytes=4, __buf=0x7fa02dff9654, __fd=81)

at /usr/include/bits/unistd.h:44

#2  ReadPipe (fd=81, buf=buf@entry=0x7fa02dff9654, nbyte=nbyte@entry=4)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/util/posix.cc:533

#3  0x00007fa06ae988a7 in download::DownloadManager::Fetch (

this=0x7f9ffc02b4e0, info=info@entry=0x7fa028009018)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/download.cc:1756

#4  0x00007fa06ae9db70 in cvmfs::Fetcher::Fetch (

this=this@entry=0x7f9ffc089e90, id=…, size=1242,

name="/CentOS7/EDEN-3.0/lib/python3.9/site-packages/skimage/draw/pycache/_polygon2mask.cpython-39.pyc",

compression_algorithm=compression_algorithm@entry=zlib::kZlibDefault,

object_type=object_type@entry=CacheManager::kTypeRegular, alt_url="",

range_offset=-1)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/fetch.cc:165

#5  0x00007fa06aef6a7f in cvmfs::cvmfs_open (req=0x7fa02800d730,

ino=54121800, fi=0x7fa02dffacf0)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/cvmfs.cc:1019

#6  0x00007fa06d2bf45d in loader::stub_open (req=0x7fa02800d730,

ino=54121800, fi=0x7fa02dffacf0)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/loader.cc:265

#7  0x00007fa06c9ba52c in do_open () from /lib64/libfuse.so.2

#8  0x00007fa06c9b9b6b in fuse_ll_process_buf () from /lib64/libfuse.so.2

#9  0x00007fa06c9b6401 in fuse_do_work () from /lib64/libfuse.so.2

#10 0x00007fa06e0f4ea5 in start_thread () from /lib64/libpthread.so.0

#11 0x00007fa06d5fd9fd in clone () from /lib64/libc.so.6


Thread 4 (Thread 0x7fa05cb47700 (LWP 24016)):

#0  0x00007fa06e0fb75d in read () from /lib64/libpthread.so.0

#1  0x00007fa06c9b5d62 in fuse_kern_chan_receive () from /lib64/libfuse.so.2

#2  0x00007fa06c9b6d59 in fuse_ll_receive_buf () from /lib64/libfuse.so.2

#3  0x00007fa06c9b637e in fuse_do_work () from /lib64/libfuse.so.2

#4  0x00007fa06e0f4ea5 in start_thread () from /lib64/libpthread.so.0

#5  0x00007fa06d5fd9fd in clone () from /lib64/libc.so.6


Thread 3 (Thread 0x7fa02e7fc700 (LWP 40586)):

#0  0x00007fa06e0fb75d in read () from /lib64/libpthread.so.0

#1  0x00007fa06c9b5d62 in fuse_kern_chan_receive () from /lib64/libfuse.so.2

#2  0x00007fa06c9b6d59 in fuse_ll_receive_buf () from /lib64/libfuse.so.2

#3  0x00007fa06c9b637e in fuse_do_work () from /lib64/libfuse.so.2

#4  0x00007fa06e0f4ea5 in start_thread () from /lib64/libpthread.so.0

#5  0x00007fa06d5fd9fd in clone () from /lib64/libc.so.6


Thread 2 (Thread 0x7fa02cff9700 (LWP 4762)):

#0  0x00007fa06e0fb75d in read () from /lib64/libpthread.so.0

#1  0x00007fa06c9b5d62 in fuse_kern_chan_receive () from /lib64/libfuse.so.2

#2  0x00007fa06c9b6d59 in fuse_ll_receive_buf () from /lib64/libfuse.so.2

#3  0x00007fa06c9b637e in fuse_do_work () from /lib64/libfuse.so.2

#4  0x00007fa06e0f4ea5 in start_thread () from /lib64/libpthread.so.0

#5  0x00007fa06d5fd9fd in clone () from /lib64/libc.so.6


Thread 1 (Thread 0x7fa06e6f2740 (LWP 16228)):

#0  0x00007fa06e0fab3b in do_futex_wait.constprop.1 ()

from /lib64/libpthread.so.0

#1  0x00007fa06e0fabcf in __new_sem_wait_slow.constprop.0 ()

from /lib64/libpthread.so.0

#2  0x00007fa06e0fac6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0

#3  0x00007fa06c9b65c8 in fuse_session_loop_mt () from /lib64/libfuse.so.2

#4  0x00007fa06d2c1fa9 in FuseMain (argc=,

argv=)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/loader.cc:1073

#5  0x0000000000404a3e in main (argc=5, argv=0x7ffd837a32e8)

at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/fuse_main.cc:144

(gdb)


Timestamp: Wed Feb  2 16:27:56 2022




How can we debug further the issue? Can it be linked to the file that is accessed by the client?


Thanks

            


            

              
              
              
            


            

                
                
              


          

          

            

              



              
                  
                  
              2
              
            

            

              
Hi Pierre,


Thanks for the report.


It looks like it’s trying to download a Python pyc file (thread 5):


#4 0x00007fa06ae9db70 in cvmfs::Fetcher::Fetch (
this=this@entry=0x7f9ffc089e90, id=…, size=1242,
name="/CentOS7/EDEN-3.0/lib/python3.9/site-packages/skimage/draw/pycache/_polygon2mask.cpython-39.pyc",
compression_algorithm=compression_algorithm@entry=zlib::kZlibDefault,
object_type=object_type@entry=CacheManager::kTypeRegular, alt_url="",
range_offset=-1)
at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/fetch.cc:165



What node is this?

Is the network reliable?

How is the cache configured for the node?

Could you provide the output of cvmfs_config showconfig <REPOSITORY_NAME>?


Thanks,

Radu

            


            

              
              
              
            


            

                
                
              


          

          

            

              



              
                  
                  
              3
              
            

            

              
Thanks for your reply.

Yes the network is reliable.

The cvmfs cache is on /cvmfscache and is shared between all the repositories mounted.

Bellow the output requested:


CVMFS_REPOSITORY_NAME=euclid-dev.in2p3.fr
CERNVM_GRID_UI_VERSION=
CVMFS_ALIEN_CACHE=
CVMFS_ALT_ROOT_PATH=
CVMFS_AUTHZ_HELPER=
CVMFS_AUTHZ_SEARCH_PATH=
CVMFS_AUTO_UPDATE=
CVMFS_BACKOFF_INIT=2    # from /etc/cvmfs/default.conf
CVMFS_BACKOFF_MAX=10    # from /etc/cvmfs/default.conf
CVMFS_BASE_ENV=1    # from /etc/cvmfs/default.conf
CVMFS_CACHE_BASE=/cvmfscache/cache    # from /etc/cvmfs/default.local
CVMFS_CACHE_DIR=/cvmfscache/cache/shared
CVMFS_CACHE_PRIMARY=
CVMFS_CHECK_PERMISSIONS=yes    # from /etc/cvmfs/default.conf
CVMFS_CLAIM_OWNERSHIP=yes    # from /etc/cvmfs/default.conf
CVMFS_CLIENT_PROFILE=    # from /etc/cvmfs/default.conf
CVMFS_CONFIG_REPO_REQUIRED=yes    # from /etc/cvmfs/default.d/60-egi.conf
CVMFS_CONFIG_REPOSITORY=config-egi.egi.eu    # from /etc/cvmfs/default.d/60-egi.conf
CVMFS_DEBUGLOG=
CVMFS_DEFAULT_DOMAIN=
CVMFS_DNS_RETRIES=
CVMFS_DNS_TIMEOUT=
CVMFS_EXTERNAL_FALLBACK_PROXY=
CVMFS_EXTERNAL_HTTP_PROXY=
CVMFS_EXTERNAL_SERVER_URL=
CVMFS_EXTERNAL_TIMEOUT=
CVMFS_EXTERNAL_TIMEOUT_DIRECT=
CVMFS_FALLBACK_PROXY=
CVMFS_FOLLOW_REDIRECTS=
CVMFS_HIDE_MAGIC_XATTRS=
CVMFS_HOST_RESET_AFTER=1800    # from /etc/cvmfs/default.conf
CVMFS_HTTP_PROXY='http://ccsquidli01.in2p3.fr:3128|http://ccsquidli02.in2p3.fr:3128|http://ccsquidli03.in2p3.fr:3128|http://ccsquidli04.in2p3.fr:3128'    # from /etc/cvmfs/default.local
CVMFS_IGNORE_SIGNATURE=
CVMFS_INITIAL_GENERATION=
CVMFS_IPFAMILY_PREFER=
CVMFS_KCACHE_TIMEOUT=
CVMFS_KEYS_DIR=/etc/cvmfs/keys    # from /etc/cvmfs/default.conf
CVMFS_LOW_SPEED_LIMIT=1024    # from /etc/cvmfs/default.conf
CVMFS_MAX_IPADDR_PER_PROXY=
CVMFS_MAX_RETRIES=1    # from /etc/cvmfs/default.conf
CVMFS_MAX_TTL=
CVMFS_MEMCACHE_SIZE=
CVMFS_MOUNT_DIR=/cvmfs    # from /etc/cvmfs/default.conf
CVMFS_MOUNT_RW=
CVMFS_NFILES=131072    # from /etc/cvmfs/default.conf
CVMFS_NFS_SHARED=
CVMFS_NFS_SOURCE=
CVMFS_OOM_SCORE_ADJ=
CVMFS_PAC_URLS='http://grid-wpad/wpad.dat;http://wpad/wpad.dat;http://cernvm-wpad.fnal.gov/wpad.dat;http://cernvm-wpad.cern.ch/wpad.dat'    # from /cvmfs/config-egi.egi.eu/etc/cvmfs/default.conf
CVMFS_PROXY_RESET_AFTER=300    # from /etc/cvmfs/default.conf
CVMFS_PROXY_TEMPLATE=
CVMFS_PUBLIC_KEY=/etc/cvmfs/keys/in2p3.fr/euclid-dev.in2p3.fr.pub    # from /etc/cvmfs/config.d/euclid-dev.in2p3.fr.conf
CVMFS_QUOTA_LIMIT=128000    # from /etc/cvmfs/default.local
CVMFS_RELOAD_SOCKETS=/var/run/cvmfs    # from /etc/cvmfs/default.conf
CVMFS_REPOSITORIES=config-egi.egi.eu,atlas.cern.ch,atlas-condb.cern.ch,lhcb.cern.ch,cms.cern.ch,alice.cern.ch,sft.cern.ch,ams.cern.ch,ilc.desy.de,auger.egi.eu,dirac.egi.eu,km3net.egi.eu,alice-ocdb.cern.ch,euclid.in2p3.fr,oasis.opensciencegrid.org,atlas-nightlies.cern.ch,francegrilles.in2p3.fr,euclid-dev.in2p3.fr,sw.lsst.eu,singularity.opensciencegrid.org,grid.cern.ch,singularity.in2p3.fr,glast.cern.ch,xenon.opensciencegrid.org,fermilab.opensciencegrid.org,lhcb-condb.cern.ch,larsoft.opensciencegrid.org,dune.opensciencegrid.org,dune.osgstorage.org,ligo.osgstorage.org,juno.ihep.ac.cn,belle.cern.ch,t2k.egi.eu,hyperk.egi.eu,cernvm-prod.cern.ch,gridpp.egi.eu,geant4.cern.ch,unpacked.cern.ch,sft-nightlies.cern.ch,cta.in2p3.fr,belle.kek.jp,sw.cta-observatory.org,compass.cern.ch,compass-condb.cern.ch,compass-mc.cern.ch,juno_nightlies.ihep.ac.cn,sw.hess-experiment.eu    # from /etc/cvmfs/default.local
CVMFS_REPOSITORY_DATE=
CVMFS_REPOSITORY_TAG=
CVMFS_ROOT_HASH=
CVMFS_SEND_INFO_HEADER=yes    # from /etc/cvmfs/default.d/60-egi.conf
CVMFS_SERVER_CACHE_MODE=
CVMFS_SERVER_URL='http://cceuclids1.in2p3.fr/cvmfs/euclid-dev.in2p3.fr;http://cernfs.roe.ac.uk/cvmfs/euclid-dev.in2p3.fr'    # from /etc/cvmfs/config.d/euclid-dev.in2p3.fr.conf
CVMFS_SHARED_CACHE=yes    # from /etc/cvmfs/default.local
CVMFS_STRICT_MOUNT=no    # from /etc/cvmfs/default.conf
CVMFS_SYSLOG_FACILITY=
CVMFS_SYSLOG_LEVEL=
CVMFS_SYSTEMD_NOKILL=
CVMFS_TIMEOUT=5    # from /etc/cvmfs/default.conf
CVMFS_TIMEOUT_DIRECT=10    # from /etc/cvmfs/default.conf
CVMFS_TRACEFILE=
CVMFS_TRUSTED_CERTS=
CVMFS_USE_CDN=
CVMFS_USE_GEOAPI=yes    # from /etc/cvmfs/default.d/60-egi.conf
CVMFS_USER=cvmfs    # from /etc/cvmfs/default.conf
CVMFS_WORKSPACE=


            


            

              
              
              
            


            

                
                
              


          

          

            

              



              
                  
                  
              4
              
            

            

              
We had three crashes impacting three different worker nodes.

Here the two missing stack traces.


Workernode 1330


--
Signal: 6, errno: 12, version: 2.8.2, PID: 8783
Executable path: /usr/bin/cvmfs2;614d97bc (deleted)

Thread 1 (process 8783):
(gdb) 


Error output:
/usr/bin/cvmfs2;614d97bc (deleted): No such file or directory.
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0xc4af2680: 


Timestamp: Tue Feb  1 01:39:05 2022




Workernode 0453


--
Signal: 6, errno: 6, version: 2.8.1, PID: 16478
Executable path: /usr/bin/cvmfs2

Thread 21 (Thread 0x7f8d21c82700 (LWP 16494)):
#0  0x00007f8d2ba7b9dd in accept () from /lib64/libpthread.so.0
#1  0x00007f8d2ac42b70 in loader::loader_talk::MainTalk (data=<optimized out>)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/loader_talk.cc:60
#2  0x00007f8d2ba74ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f8d2af7d9fd in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7f8d2648b700 (LWP 21741)):
#0  0x00007f8d2af72ccd in poll () from /lib64/libc.so.6
#1  0x00007f8d2883cd3f in poll (__timeout=-1, __nfds=2, __fds=0x7f8d2648aeb8)
    at /usr/include/bits/poll2.h:46
#2  Watchdog::MainWatchdogListener (data=0x7f8cd0317160)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/monitor.cc:486
#3  0x00007f8d2ba74ea5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f8d2af7d9fd in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7f8d24487700 (LWP 21742)):
#0  0x00007f8d2ba7b75d in read () from /lib64/libpthread.so.0
#1  0x00007f8d28861e5a in read (__nbytes=1, __buf=0x7f8d24486e8f, __fd=17)
    at /usr/include/bits/unistd.h:44
#2  ReadPipe (fd=17, buf=buf@entry=0x7f8d24486e8f, nbyte=nbyte@entry=1)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/util/posix.cc:533
#3  0x00007f8d2887bf36 in FuseInvalidator::MainInvalidator (
    data=0x7f8cd0316fe0)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/fuse_evict.cc:111
#4  0x00007f8d2ba74ea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8d2af7d9fd in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f8d23c86700 (LWP 21743)):
#0  0x00007f8d2af72ccd in poll () from /lib64/libc.so.6
#1  0x00007f8d2887cd97 in poll (__timeout=<optimized out>, __nfds=1, 
    __fds=0x7f8d23c85ed0) at /usr/include/bits/poll2.h:46
#2  FuseRemounter::MainRemountTrigger (data=0x7f8cd0316f60)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/fuse_remount.cc:210
#3  0x00007f8d2ba74ea5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f8d2af7d9fd in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f8d24c88700 (LWP 21744)):
#0  0x00007f8d2af72ccd in poll () from /lib64/libc.so.6
#1  0x00007f8d288221c7 in poll (__timeout=60000, __nfds=1, 
    __fds=0x7f8d24c87ed0) at /usr/include/bits/poll2.h:46
#2  glue::NentryTracker::MainCleaner (data=0x7f8cd03127b0)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/glue_buffer.cc:187
#3  0x00007f8d2ba74ea5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f8d2af7d9fd in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f8d25c8a700 (LWP 21745)):
#0  0x00007f8d289baa69 in inflate_fast () from /usr/lib64/libcvmfs_fuse.so
#1  0x00007f8d289b58c3 in inflate () from /usr/lib64/libcvmfs_fuse.so
#2  0x00007f8d2880b7d2 in zlib::DecompressZStream2Sink (
    buf=buf@entry=0x7f8d14005cd0, size=size@entry=16384, 
    strm=strm@entry=0x7f8cd40268f0, sink=0x7f8cddffa798)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/compression.cc:252
#3  0x00007f8d28812d52 in download::CallbackCurlData (ptr=0x7f8d14005cd0, 
    size=<optimized out>, nmemb=<optimized out>, info_link=0x7f8cd4026858)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/download.cc:285
#4  0x00007f8d2889e64d in Curl_client_write ()
   from /usr/lib64/libcvmfs_fuse.so
#5  0x00007f8d288a58e6 in Curl_readwrite () from /usr/lib64/libcvmfs_fuse.so
#6  0x00007f8d2889a076 in multi_runsingle () from /usr/lib64/libcvmfs_fuse.so
#7  0x00007f8d2889a891 in multi_socket () from /usr/lib64/libcvmfs_fuse.so
#8  0x00007f8d2889a973 in curl_multi_socket_action ()
   from /usr/lib64/libcvmfs_fuse.so
#9  0x00007f8d288184e1 in download::DownloadManager::MainDownload (
    data=0x7f8cd0021110)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/download.cc:572
#10 0x00007f8d2ba74ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f8d2af7d9fd in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f8d25489700 (LWP 21746)):
#0  0x00007f8d2af72ccd in poll () from /lib64/libc.so.6
#1  0x00007f8d288183ea in poll (__timeout=<optimized out>, 
    __nfds=<optimized out>, __fds=<optimized out>)
    at /usr/include/bits/poll2.h:46
#2  download::DownloadManager::MainDownload (data=0x7f8cd0050c50)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/download.cc:513
#3  0x00007f8d2ba74ea5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f8d2af7d9fd in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f8d23485700 (LWP 21747)):
#0  0x00007f8d2af72ccd in poll () from /lib64/libc.so.6
#1  0x00007f8d28881bce in poll (__timeout=-1, __nfds=2, __fds=0x7f8ce4014120)
    at /usr/include/bits/poll2.h:46
#2  quota::MainWatchdogListener (data=0x7f8cd002b600)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/quota_listener.cc:90
#3  0x00007f8d2ba74ea5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f8d2af7d9fd in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f8d22c84700 (LWP 21748)):
#0  0x00007f8d2af72ccd in poll () from /lib64/libc.so.6
#1  0x00007f8d288820eb in poll (__timeout=-1, __nfds=2, __fds=0x7f8d180008c0)
    at /usr/include/bits/poll2.h:46
#2  quota::MainUnpinListener (data=0x7f8cd0029c10)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/quota_listener.cc:48
#3  0x00007f8d2ba74ea5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f8d2af7d9fd in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f8d22483700 (LWP 21749)):
#0  0x00007f8d2ba7b9dd in accept () from /lib64/libpthread.so.0
#1  0x00007f8d28883f27 in TalkManager::MainResponder (data=0x7f8cd0317230)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/talk.cc:173
#2  0x00007f8d2ba74ea5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f8d2af7d9fd in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f8cdd7fa700 (LWP 16076)):
#0  0x00007f8d2ba7b75d in read () from /lib64/libpthread.so.0
#1  0x00007f8d2a335d62 in fuse_kern_chan_receive () from /lib64/libfuse.so.2
#2  0x00007f8d2a336d59 in fuse_ll_receive_buf () from /lib64/libfuse.so.2
#3  0x00007f8d2a33637e in fuse_do_work () from /lib64/libfuse.so.2
#4  0x00007f8d2ba74ea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8d2af7d9fd in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f8cb97fa700 (LWP 20024)):
#0  0x00007f8d2ba7b75d in read () from /lib64/libpthread.so.0
#1  0x00007f8d2a335d62 in fuse_kern_chan_receive () from /lib64/libfuse.so.2
#2  0x00007f8d2a336d59 in fuse_ll_receive_buf () from /lib64/libfuse.so.2
#3  0x00007f8d2a33637e in fuse_do_work () from /lib64/libfuse.so.2
#4  0x00007f8d2ba74ea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8d2af7d9fd in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f8cddffb700 (LWP 42769)):
#0  0x00007f8d2ba7b75d in read () from /lib64/libpthread.so.0
#1  0x00007f8d28861e5a in read (__nbytes=4, __buf=0x7f8cddff9654, __fd=67)
    at /usr/include/bits/unistd.h:44
#2  ReadPipe (fd=67, buf=buf@entry=0x7f8cddff9654, nbyte=nbyte@entry=4)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/util/posix.cc:533
#3  0x00007f8d288188a7 in download::DownloadManager::Fetch (
    this=0x7f8cd0021110, info=info@entry=0x7f8cd4026858)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/download.cc:1756
#4  0x00007f8d2881db70 in cvmfs::Fetcher::Fetch (
    this=this@entry=0x7f8cd00aafe0, id=..., size=364072, 
    name="/CentOS7/EDEN-3.0/lib/python3.9/site-packages/sklearn/neighbors/_dist_metrics.cpython-39-x86_64-linux-gnu.so", 
    compression_algorithm=compression_algorithm@entry=zlib::kZlibDefault, 
    object_type=object_type@entry=CacheManager::kTypeRegular, alt_url="", 
    range_offset=-1)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/fetch.cc:165
#5  0x00007f8d28876a7f in cvmfs::cvmfs_open (req=0x7f8cd4021c70, 
    ino=58157381, fi=0x7f8cddffacf0)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/cvmfs.cc:1019
#6  0x00007f8d2ac3f45d in loader::stub_open (req=0x7f8cd4021c70, 
    ino=58157381, fi=0x7f8cddffacf0)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/loader.cc:265
#7  0x00007f8d2a33a52c in do_open () from /lib64/libfuse.so.2
#8  0x00007f8d2a339b6b in fuse_ll_process_buf () from /lib64/libfuse.so.2
#9  0x00007f8d2a336401 in fuse_do_work () from /lib64/libfuse.so.2
#10 0x00007f8d2ba74ea5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f8d2af7d9fd in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f8d027fc700 (LWP 17841)):
#0  0x00007f8d2ba7b75d in read () from /lib64/libpthread.so.0
#1  0x00007f8d2a335d62 in fuse_kern_chan_receive () from /lib64/libfuse.so.2
#2  0x00007f8d2a336d59 in fuse_ll_receive_buf () from /lib64/libfuse.so.2
#3  0x00007f8d2a33637e in fuse_do_work () from /lib64/libfuse.so.2
#4  0x00007f8d2ba74ea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8d2af7d9fd in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f8d21186700 (LWP 39895)):
#0  0x00007f8d2af74a33 in select () from /lib64/libc.so.6
#1  0x00007f8d28863233 in SafeSleepMs (ms=ms@entry=100)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/util/posix.cc:1730
#2  0x00007f8d2883ca5b in Watchdog::SendTrace (sig=6, 
    siginfo=<optimized out>, context=<optimized out>)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/monitor.cc:317
#3  <signal handler called>
#4  0x00007f8d2aeb5387 in raise () from /lib64/libc.so.6
#5  0x00007f8d2aeb6a78 in abort () from /lib64/libc.so.6
#6  0x00007f8d2aef7f67 in __libc_message () from /lib64/libc.so.6
#7  0x00007f8d2af00329 in _int_free () from /lib64/libc.so.6
#8  0x00007f8d2b8239db in std::string::reserve(unsigned long) ()
   from /lib64/libstdc++.so.6
#9  0x00007f8d2b823a58 in std::string::append(std::string const&) ()
   from /lib64/libstdc++.so.6
#10 0x00007f8d2883661e in ChunkListMagicXattr::PrepareValueFenced (
    this=0x7f8cd03157d0)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/magic_xattr.cc:174
#11 0x00007f8d2887745f in cvmfs::cvmfs_getxattr (req=0x7f8cc0002c80, 
    ino=<optimized out>, name=<optimized out>, size=128)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/cvmfs.cc:1366
#12 0x00007f8d2ac3f086 in loader::stub_getxattr (req=0x7f8cc0002c80, 
    ino=58171173, name=0x7f8cf002be90 "user.chunk_list", size=128)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/loader.cc:303
#13 0x00007f8d2a339b6b in fuse_ll_process_buf () from /lib64/libfuse.so.2
#14 0x00007f8d2a336401 in fuse_do_work () from /lib64/libfuse.so.2
#15 0x00007f8d2ba74ea5 in start_thread () from /lib64/libpthread.so.0
#16 0x00007f8d2af7d9fd in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f8cde7fc700 (LWP 40028)):
#0  0x00007f8d2ba7b75d in read () from /lib64/libpthread.so.0
#1  0x00007f8d2a335d62 in fuse_kern_chan_receive () from /lib64/libfuse.so.2
#2  0x00007f8d2a336d59 in fuse_ll_receive_buf () from /lib64/libfuse.so.2
#3  0x00007f8d2a33637e in fuse_do_work () from /lib64/libfuse.so.2
#4  0x00007f8d2ba74ea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8d2af7d9fd in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f8cbbfff700 (LWP 40036)):
#0  0x00007f8d2ba7b75d in read () from /lib64/libpthread.so.0
#1  0x00007f8d2a335d62 in fuse_kern_chan_receive () from /lib64/libfuse.so.2
#2  0x00007f8d2a336d59 in fuse_ll_receive_buf () from /lib64/libfuse.so.2
#3  0x00007f8d2a33637e in fuse_do_work () from /lib64/libfuse.so.2
#4  0x00007f8d2ba74ea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8d2af7d9fd in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f8d20985700 (LWP 40038)):
#0  0x00007f8d2ba7b75d in read () from /lib64/libpthread.so.0
#1  0x00007f8d2a335d62 in fuse_kern_chan_receive () from /lib64/libfuse.so.2
#2  0x00007f8d2a336d59 in fuse_ll_receive_buf () from /lib64/libfuse.so.2
#3  0x00007f8d2a33637e in fuse_do_work () from /lib64/libfuse.so.2
#4  0x00007f8d2ba74ea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8d2af7d9fd in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f8cdeffd700 (LWP 15161)):
#0  0x00007f8d2ba7b75d in read () from /lib64/libpthread.so.0
#1  0x00007f8d2a335d62 in fuse_kern_chan_receive () from /lib64/libfuse.so.2
#2  0x00007f8d2a336d59 in fuse_ll_receive_buf () from /lib64/libfuse.so.2
#3  0x00007f8d2a33637e in fuse_do_work () from /lib64/libfuse.so.2
#4  0x00007f8d2ba74ea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8d2af7d9fd in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f8d037fe700 (LWP 15166)):
#0  0x00007f8d2ba7b75d in read () from /lib64/libpthread.so.0
#1  0x00007f8d2a335d62 in fuse_kern_chan_receive () from /lib64/libfuse.so.2
#2  0x00007f8d2a336d59 in fuse_ll_receive_buf () from /lib64/libfuse.so.2
#3  0x00007f8d2a33637e in fuse_do_work () from /lib64/libfuse.so.2
#4  0x00007f8d2ba74ea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8d2af7d9fd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f8d2c073740 (LWP 16478)):
#0  0x00007f8d2ba7ab3b in do_futex_wait.constprop.1 ()
   from /lib64/libpthread.so.0
#1  0x00007f8d2ba7abcf in __new_sem_wait_slow.constprop.0 ()
   from /lib64/libpthread.so.0
#2  0x00007f8d2ba7ac6b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x00007f8d2a3365c8 in fuse_session_loop_mt () from /lib64/libfuse.so.2
#4  0x00007f8d2ac41fa9 in FuseMain (argc=<optimized out>, 
    argv=<optimized out>)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/loader.cc:1073
#5  0x0000000000404a3e in main (argc=5, argv=0x7fffcf56d108)
    at /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.8.1/cvmfs/fuse_main.cc:144
(gdb) 


Timestamp: Tue Feb  1 01:39:10 2022



and the third is the one reported above.

            


            

              
              
              
            


            

                
                
              


          

          

            

              



              
                  
                  
              5
              
            

            

              
Thank you for the extre details.


Does the issue only happen under load?


Is there a lot of pressure on the CVMFS cache?

            


            

              
              
              
            


            

                
                
              


          

          

            

              



              
                  
                  
              6
              
            

            

              
I’m not sure I can answer but I can try to check with the cvmfs admins here.

But I would say no to both your questions:  we were running a simple Euclid pipeline and the worker nodes impacted were running one (or max two Euclid single-core jobs). Instead, I didn’t know how many jobs were on the same worker nodes from other projects accessing other CVMFS repo.

            


            

              
              
              
            


            

                
                
              


          

          

            

              



              
                  
                  
              7
              
            

            

              
I discussed with the admin here at CC-IN2P3 and I confirm: no load and no pressure on CVMFS cache when the crash happened.

            


            

              
              
              
            


            

                
                
              


          

          

            

              



              
                  
                  
              8
              
            

            

              
This looks to me like the problem with extended attributes described in CVM-2014, which was fixed in cvmfs 2.9.