"cvmfs_server import" from local data?

Publisher / Stratum 0
1

Hello,

at present I’m doing a backup of /etc/cvmfs and /srv/cvmfs of our Stratum-0 server with the hope of using that data to recover our repositories into another machine in case that the Stratum-0 server dies or the data gets corrupted for some reason.

I was hoping to use “cvmfs_server import […]”, but I just realized that the documentation for this (Creating a Repository (Stratum 0) — CernVM-FS 2.11.3 documentation) is not very detailed, so I’m not sure how to proceed.

So, my question is: assuming that I have a backup copy of the directories above (/etc/cvmfs and /srv/cvmfs) and a new server without any CernVM-FS repositories in it, can I use “cvmfs_server import […]” to recreate the original server? (my guess is that maybe I just need to put all the data in those directories and then just call “cvmfs_server import […]” instead of “cvmfs_server mkfs”, but obviously I don’t want to run this in the production Stratum-0 server, and before I get to the pain of trying it in a new machine I wanted to know whether my guess is valid and if this is the right approach).

Any help, tips, examples, etc. very welcome.

Many thanks,

2

Hi Angel,

yes, that’s exactly the intended use of cvmfs_server import! cvmfs_server mkfs might overwrite the manifest, and import will also preserve the reflog for garbage collection.
The usage is similar to mkfs, something like this if you’d have s3 backend storage:

cvmfs_server import \
                        -o `whoami`  -w http://cvmfs-test-migration.s3.cern.ch/test.cern.ch \
                        -u S3,/var/spool/cvmfs/test.cern.ch/tmp,cvmfs/test.cern.ch@/etc/cvmfs/s3.conf   \
                           -r test.cern.ch

cvmfs_server import should be safe to do, but you can test this on a test repo by creating a test repository and removing it with cvmfs_server rmfs -p to preserve data and keys.
I hope this helps,
Valentin

3

Perfect, many thanks. I will try it in a few days and will report back.

4

I did try it today with a test repository, and after removing the directory completely (to simulate starting afresh in a new machine), I just had to recover from the backup directory the relevant files for that repo in /etc/cvmfs/keys/ plus the /srv/cvmfs/test_repo directory and a simple

cvmfs_server import test_repo.iac.es

did the trick. After that, I think that I would need to also recover from the backup the “server.conf” file in /etc/cvmfs/repositories.d/test_repo/

and the repository would be up and running again. Really cool, much faster than recreating the repository from the raw data.

5

Note that stratum 1s also have a copy of all the data in /srv/cvmfs, so it may be unnecessary to keep a backup copy of that. The only things that are really needed for backup are the files in /etc/cvmfs/keys and the settings in server.conf.

It is often advantageous to start out with a fresh replica from a stratum 0 or 1, by doing cvmfs_server add-replica and cvmfs_server snapshot because that gets rid of garbage files that have collected over time due to catalogs being regenerated or files getting deleted by users. Then that replica can be converted into a stratum 0 by doing cvmfs_server rmfs -p and cvmfs_server import.

If this is done on the same machine that is actively being used as a stratum 0 for a repository, you can use a temporary name for the repository with add-replica -n tempname and when ready have a short intervention to swap the fresh replica into the place of the old one under srv/cvmfs.

6

Oh, thanks for that. It sounds really interesting and I could get rid of my backup.

I didn’t know that you could create a replica out of a Stratum-1, so I was assuming that if my Stratum-0 died, I would have to manually copy the stuff in /srv and then do the “import”, but your approach sounds very neat. I’m now almost wishing that our Stratum-0 server dies so I can try the approach :slight_smile: