Restoring Stratum0-PreProd : no backups

[paul.zakharov]

We have a little clone of this topic :

Failed to fetch repository manifest (2 - incomplete manifest) : due to a disk corruption

We have a very critical issue on our production CVMFS server. Due to some unattended maintenance the server had some issues and the filesystem contained cvmfs got corrupted. Normally (I was not here at the moment) the filesystem has been repaired. At least it seems to work well. But the cvmfs still has the issues. We have two repositories. One of them seems to work but probably not well (but we could discuss it later). Another repository is shown in “cvmfs_server list” > cvmfs_server list…

We discovered that our Pre-Prod Stratum0 also corrupted.

• as it’s the Pre-Prod server we are not really bothered about the data…
• but we don’t want the empty Stratum0-PreProd as we use Pre-Prod to test some things and we need some data there.

Also, as it’s the Pre-Prod :

• we have not backup for it ;
• we have not Stratum1-Pre-Prod ;
  So we have not any source to restore data.

We think that we could restore Pre-Prod using backup of Stratum0-Prod.
But there is one detail that is not clear.

Before we do

cvmfs_server import

should we put the keys from Stratum0-Prod (corresponding to the data) or from Stratum0-PreProd.

1. If we use keys from Stratum0-PreProd.

It seems that is doesn’t work.
Probably because the data doesn’t correspond to the keys ?
Am I right ? Or we did something wrong ?

2. I think we should not use the keys from Stratum0-Prod as in this case our Stratum0-Prod and our new Stratum0-PreProd will have the same keychain.
   Am I right ?

In the documentation I see :

“An import can either use a provided repository keychain placed into /etc/cvmfs/keys or generate a fresh repository key and certificate for the imported repository. The latter case requires an update of the repository’s whitelist to incorporate the newly generated repository key. To generate a fresh repository key add -t -r to cvmfs_server import.”

So, is it the only way that we can proceed ?
And in this case we will have to change the public keys on all clients which use Stratum0-PreProd (publishers, builders etc) ?

Sorry for another messy problem and thank you for your help.

[vavolkl]

Hi Paul, as long as the masterkey for the preprod survived, you should be able to also import the contents of the production repository and keep using the preprod repo with that key without any config changes on the client side. You should use the -t -r options to import for that, that will recreate any files that have a signature from the the production repo, so something like this:

# keys dir contains ONLY the Pre-Prod masterkey pair:
#   /etc/cvmfs/keys/pre/<repo>.masterkey  and  <repo>.pub
 cvmfs_server import -w http://<preprod-s0>/cvmfs/<repo> -k /etc/cvmfs/keys/pre -t -r [-o <owner> -g] <repo>

Note: I’m fairly confident in this but haven yet to test it.

Best,
Valentin

[paul.zakharov]

Thank you, Valentin

For the moment I didn’t do all the checks, but on first sight your command works perfectly.

I could import repo from stm0-prod backup and after import everything seems to work.

We need some time to finish the server before put it into Pre-Prod infrastructure.
I’ll confirm once we do it (and if everything is ok).

But I think the problem is probably solved.

Thank you very much again for your help.

Merci,
Paul Zakharov