How to recover from filling the disk to 100% on a stratum-1 server?

Stratum 1
1

We recently manage to fill the drive to 100% on one of our stratum-1 servers. sudo cvmfs_server check runs correctly, but sudo cvmfs_server check -i our.local.cvmfs fails with mismatch of file name and content hash | at: /srv/cvmfs/path/data/89/hash.

I believe a cvmfs_server snapshot ran just as the file system filled to 100%, which caused the file with the mismatch to be zero bytes in size.

I have since corrected the lack of free space, but how do I get CVMFS to pull the correct contents from our stratum-0?

Thank you,
Omen

2

Hi Omen,

Yes, this is a feature that is really missing from CVMFS, we just discussed it recently: cvmfs_server check option to repair missing data chunks · Issue #3878 · cvmfs/cvmfs · GitHub

Let me see if I can quickly put in a fix. The manual procedure would be just to remove the corrupted file and download it manually from the stratum 0, so if there are no chunks missing, it should just be:


export hash1=89
export hash2=0123abcdefg...
mv /srv/cvmfs/path/data/$hash1/$hash2 /tmp/cvmfsfile.$hash1.$hash2.backup
curl -o /srv/cvmfs/path/data/$hash1/$hash2 http://my-stratum-0.tld/cvmfs/my.repo.name/data/$hash1/hash2
3

Thank you, that worked!

For people from the future, just a small correction that the final hash2 needs to be $hash2. This worked for me:

export hash1=89
export hash2=0123abcdefg...
export repo=my.repo.tld
export stratum0=my-stratum-0.tld

mv /srv/cvmfs/$repo/data/$hash1/$hash2 /tmp/cvmfsfile.$hash1.$hash2.backup
curl -o /srv/cvmfs/$repo/data/$hash1/$hash2 http://$stratum0/cvmfs/$repo/data/$hash1/$hash2

So no need for a quick fix for us, but it would be a fantastic feature. I missed that bug report and the linked information, so I appreciate you giving me details here to fix it.