Hello,
I faced a filesystem corruption Job for var-spool-cvmfs-...-rdonly.mount failed
I’ve deleted the filesystem to restart from scratch.
I did the cvmfs version update, and tried multiple times to recreate it from scratch, and sync our data without success.
It fails now every time it reaches an amount of data/files to share IMAS data. (~2TB/17M small files)
If I do a full rsync at some point I have a cvmfs crash. I’m now doing some rsync by subfolders (transaction/rsync/publish) but at some point during a publish it also crash.
Before restarting again from scratch, is there any obvious problem? I’ve tried different partition configurations, and also tried with CVMFS_AUTOCATALOGS true & false…
I am contacting you because data synchronization takes hours before each crash.
Thank you in advance for your help.
Regards
If I see correctly, you are trying to copy all the data from nfs (the 2 TB?) and publish it in a single transaction?
If that is the case, please try making multiple smaller transactions and see if that works.
Your /var/spool/cvmfs is only 700 GB large. During publication this is used as temporary staging area, where the data will be laying uncompressed. So a single publication of 2TB will not work.
I recommend checking the size of each subfolder with du -hs to make sure it’s well below 700GB. You can also watch the space remaining in /var/spool/cvmfs as you do the rsyncs.
[root@io-ls-cvmfs01:~]# cvmfs_server publish imas.iter.org
imas.iter.org is in a transaction but /cvmfs/imas.iter.org is not mounted read/write
Repository imas.iter.org is in a transaction and cannot be repaired.
--> Run `cvmfs_server abort imas.iter.org` to revert and repair.
[root@io-ls-cvmfs01:~]# cvmfs_server transaction imas.iter.org
imas.iter.org is in a transaction but /cvmfs/imas.iter.org is not mounted read/write
Repository imas.iter.org is in a transaction and cannot be repaired.
--> Run `cvmfs_server abort $name` to revert and repair.
(unexpected termination) cannot establish writable mountpoint
Stacktrace:
/lib64/libcvmfs_server.so.2.11.3(+0x79c96) [0x7f95ae48ac96]
/lib64/libcvmfs_server.so.2.11.3(+0x80b72) [0x7f95ae491b72]
/lib64/libcvmfs_server.so.2.11.3(_ZN7publish9Publisher16TransactionRetryEv+0x58) [0x7f95ae49a47c]
/lib64/libcvmfs_server.so.2.11.3(_ZN7publish9Publisher11TransactionEv+0xe) [0x7f95ae48b214]
/usr/bin/cvmfs_publish() [0x4396c9]
/usr/bin/cvmfs_publish() [0x40f785]
/lib64/libc.so.6(__libc_start_main+0xf3) [0x7f95abfeb493]
/usr/bin/cvmfs_publish() [0x40f98e]
/dev/sdb1 3.7T 338G 3.4T 10% /var/spool/cvmfs
/dev/sdc1 3.7T 273G 3.4T 8% /srv/cvmfs
imas.iter.org 4.0G 3.8G 130M 97% /var/spool/cvmfs/imas.iter.org/rdonly
overlay_imas.iter.org 3.7T 338G 3.4T 10% /cvmfs/imas.iter.org
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 93.2G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 92.2G 0 part
├─cl_io--ls--cvmfs01-root 253:0 0 55.7G 0 lvm /
└─cl_io--ls--cvmfs01-swap 253:1 0 9.3G 0 lvm [SWAP]
sdb 8:16 0 3.7T 0 disk
└─sdb1 8:17 0 3.7T 0 part /var/spool/cvmfs
sdc 8:32 0 3.7T 0 disk
└─sdc1 8:33 0 3.7T 0 part /srv/cvmfs
can you abort the transaction? It was not clear if you tried that. Please also retry setting the environment variable CVMFS_SERVER_DEBUG=3
Cheers,
Valentin