we have the following problem: We had a running CVMFS instance in a Ubuntu 16.04 VM which still used AUFS. The repositories /srv/cvmfs and /var/spool/cvmfs were symlinks which pointed to folders which were mounted via NFS from outside the VM.
To switch to Ubuntu 20, we recreated the VM. Since OverlayFS complained about the NFS shares, we copied the content of the NFS share for /srv/cvmfs to the actual folder /srv/cvmfs inside the VM. In addition, we copied the keys from /etc/cvmfs/keys in the old VM to the new VM
Then, we performed cvmfs_server import -r [reponame] followed by cvmfs_server eliminate-hardlinks [reponame]. Now we expected our data to show up in /var/spool/cvmfs/[reponame]/rdonly. However, this directory is empty.
cvmfs_server check returns no problems found .
Is there anything else we can/have to do to retrieve our data and get the repository back up with our old data?
Thank you very much and please tell me if I can supply additional information.
If I understand your message correctly, you’re saying that /var/spool/cvmfs is still on NFS? I’m surprised that OverlayFS would have complained about /srv/cvmfs, but /var/spool/cvmfs is important. I don’t know of anybody doing that at this point although I used to use NFS on a cvmfs server years ago (probably it was AUFS in those days, not OverlayFS). I found a message from the OverlayFS author that indicates it should work except that NFS acls can get in the way. He suggests setting an NFS export option “noacl”.
As pointed out by Dave, the actual repository data consists of the /srv/cvmfs directory and the repository keys. The /var/spool/cvmfs directory should be local, i.e. not on a network drive. You could try creating a new Ubuntu 20 VM and repeat the import, this time only with /srv/cvmfs mounted through NFS.
For the size of the local disk of the publisher, keep in mind that the free space in /var/spool/cvmfs is the limit for a single transaction.
Ah sorry, I might have been imprecise. I maybe supplied too much information about the state of the installation before the move to the new VM.
Before moving to the new VM, we actually had /var/spool/cvmfs also mounted via NFS. However we couldn’t get that to work in the new VM with OverlayFS so we stopped doing this. Instead we stopped using both NFS shares and moved the content directly into a fresh VM (no more mounts). Then we did the import. However, /var/spool/cvmfs/[reponame]/rdonly is empty afterwards. Do we still need to do anything else?
I see, the /var/spool/cvmfs/[reponame]/rdonly directory is a cvmfs mountpoint. It should normally be mounted but perhaps it isn’t. Does cvmfs_server mount <reponame> help?
No, unfortunately this doesn’t help, the directory is still empty.
The linux mount command (no arguments) reports
overlay_[reponame] on /cvmfs/[reponame] type overlay (ro,nodev,relatime,lowerdir=/var/spool/cvmfs/[reponame]/rdonly,upperdir=/var/spool/cvmfs/[reponame]/scratch/current,workdir=/var/spool/cvmfs/[reponame]/ofs_workdir
Is there some way to check whether CVMFS can “see” our data in /srv/cvmfs ? Just to crosscheck that this is not a permissions thing or something like that.
Both the read-only mountpoint on /var/spool/cvmfs/[reponame]/rdonly and the union mount on /cvmfs/[reponame] should actually be registered in /etc/fstab. Could you paste the contents of /etc/fstab?
root@cvmfs:~# mount /var/spool/cvmfs/[reponame]/rdonly
CernVM-FS: running with suid support
CernVM-FS: loading Fuse module... done
CernVM-FS: mounted cvmfs on /var/spool/cvmfs/[reponame]/rdonly
/var/spool/cvmfs/[reponame]/rdonly is still completely empty.
That is quite strange. The mounting succeeds and it mounts indeed a completely empty repository (revision number 0). That is normally not an achievable state. Even if you do cvmfs_server mkfs there will be an initial transaction bumping the revision number to 1. My only explanation is that at some point a cvmfs_server mkfs command aborted prematurely.
Does the /srv/cvmfs/[reponame] directory have some content, i.e. does du -h /srv/cvmfs/[reponame] actually show some volume?
Do you have the original /srv/cvmfs directory from the NFS server still available? In this case, you could unmount both /var/spool/cvmfs/[reponame]/rdonly and /cvmfs/[reponame] and copy the data again into the new Ubuntu 20 machine. Perhaps the first time there was an issue with copying the .cvmfsXYZ files from the /srv/cvmfs/[reponame] directory.
In this instance of the VM I didn’t do any calls of cvmfs_server mkfs, just cvmfs_server import. Maybe previously some cvmfs_server mkfs failed and the old content was overwritten.
The directory /srv/cvmfs/[reponame] has 7GB so there is something in there.
However, there are no .cvmfsXYZ files in /srv/cvmfs/[reponame], just a directory called data. This directory contains consecutively numbered folders which in turn contain a bunch of files with names that look like hashes. We assumed that this is the data, I guess now that this is wrong?
But I just found out that 7GB is not enough for the things that are supposed to be in the directory so it seems like during the process of restoring the data a cvmfs_server mkfs was executed which maybe overwrote the data? We will check if the things in /srv/cvmfs agree with the cold-storage backup.
Okay, we have now restored a backup of /srv/cvmfs. However, there is no significant difference between this backup from the previously running instance and the backup we had used until now.
I diff’ed both directories (the /srv/cvmfs/ inside the VM and the backup and only the files .cvmfspublished, .cvmfsreflog and .cvmfswhitelist differ between both folders.
Additionally I also checked inside the old VM but couldn’t find any .cvmfsXYZ-files.
If the data is contained in these files, what do the “hash”-named files in /srv/cvmfs/[reponame]/data/ contain?
The hash-named files under the data directory contain the actual data, as well as catalogs (the files ending in C). The .cvmfs* files contain some small bits of meta information needed by cvmfs. This CHEP paper explains some of the details if you’d like to know more.
Ah sorry, I thought you meant files with some number instead of “XYZ”. I didn’t think about those metadata files.
Unfortunately restoring those files also didn’t work. However we have discovered a previously unknown location at which the repository contents (just as a directory, not as cvmfs-data) had been backed up and we’re now basically setting up a new repository with this. The data is also much larger than what we had at /srv/cvmfs so we suspect that some very strange configuration must have been present before (maybe some NFS mount from one server to another or something like that). We will try to investigate after restoring access to the cvmfs repo’s content for our users.
Anyway, thank you so much for the support and sorry for the late reply!