Best practices wrt backup (S3 backend)

Publisher / Stratum 0
1

Dear all

Here at INFN we are setting up a CVMFS infrastructure to host both personal and group repositories.

The Stratum 0 is using a S3 (ceph-rgw) backend.

In the CVMFS clients we specify as CVMFS_SERVER_URL the URL of this S3 endpoint: we are using site squids but, at least for the time being, we have not deployed any stratum-1 server.

Given this architecture, I’d like to better understand what are the best practices and experiences wrt backup of the data in the S3 backend.

I am not too worried about hardware failures since the ceph elements of the cluster are already properly replicated.

I am more worried about other possible incidents. E.g. in the past it happened that in some repos the .cvmfspublished and .cvmfsreflog files disappeared. We were not able to understand the root cause of the problem (maybe an admin deleted these files by errors ?) and we were not able to recover these repos: we had to delete and then recreate them. Fortunately they were just some test repos

So I would appreciate if you could share your experiences and your recommendations wrt backup

I’d also like to understand if having stratum-1 could in some way help to address this issue

Thanks a lot

Massimo

2

I don’t have any experience with S3, but in my experience with stratum 0s with local data it’s always possible to recover a stratum 0 if there’s a stratum-1 copy. In the worst case you can take a snapshot from the stratum 1, do cvmfs_server rmfs -fp to preserve the data, and cvmfs_server import to convert it into a stratum 0. You do need a copy of the key files from /etc/cvmfs/keys. If the .cvmfsreflog is a problem, it can be recreated with cvmfs_server check -r. If just .cvmfspublished is missing it can be copied from the stratum 1. It can also be resigned with cvmfs_server resign -p, and whitelists can be resigned with cvmfs_server resign.

Personally I don’t understand why people want to store stratum 0 data in S3. I haven’t measured it but it must be significantly slower than local disk.

Dave

3

Thanks a lot for your feedback !