But it was already 3 years ago and our memory is short, and more serious, the situation is little bit different now.
Now, we have stratum0 server connected with two publishers.
Sometimes, when we connect to our publishers and try to see the repository content it shows nothing :
ls /cvmfs/repo1
The repo seems to be empty.
If we open a transaction for this repository :
cvmfs_server transaction repo1
it works and the /cvmfs/repo1 directory become populated.
Also, after publishing (or aborting) this transaction, the repository directory stay populated, i.e. :
ls /cvmfs/repo1 show the content of repository.
We have some question about this.
Is it normal ?
If yes, how many time the repository stays populated and at which moment it become “empty” ?
The linked post still applies, I’d say, in particular this comment:
/cvmfs does not need to be backed up because it is generated from files in /srv. Do keep backing up /srv. Skipping /var/spool/cvmfs is good.
So on publishers (both stratum0 and publishers connected through a gateway) /cvmfs/repo1 is a actually an overlayfs - the repository is mounted just like a client would do it - in /var/spool/cvmfs/repo1/rdonly . Then there is a writeable scratch layer, also in /var/spool/cvmfs, and then both are overlayed on /cvmfs/repo1 to allow you to write there. The configuration of all of this mount points is in /etc/fstab.
So you could for example do
cd /var/spool/cvmfs/repo1
umount rdonly
mount rdonly
or
cd /var/spool/cvmfs/repo1
cvmfs_talk -p cvmfs_io remount
It’s normal that the transaction command will remount rdonly to get the latest revision, I’m not sure why it would ever be empty though. The rdonly mount should behave pretty similar to any other cvmfs client. Can you check the syslog of the publisher for messages from cvmfs?
This time it’s not about backing up.
We have another issue that is very different, strange and rare. So it’s not easy to explain, and so we are starting to check some basic points.
So, it’s not normal that repository’s directory (/cvmfs/repo1) is empty ?
For the moment we are pretty sure that we observe such behavior after some period of inactivity (probably more that one day).
Also, when we were in such situation (in our case /cvmfs/euclid-dev.in2p3.fr directory was empty) and we opened transaction we saw :
$ cvmfs_server transaction euclid-dev.in2p3.fr
/var/spool/cvmfs/euclid-dev.in2p3.fr/rdonly is not mounted properly
/cvmfs/euclid-dev.in2p3.fr is not mounted properly
Trying to mount /var/spool/cvmfs/euclid-dev.in2p3.fr/rdonly... success
Trying to mount /cvmfs/euclid-dev.in2p3.fr... success
Gateway reply: ok
Does the message
“/var/spool/cvmfs/euclid-dev.in2p3.fr/rdonly is not mounted properly”
(and “/cvmfs/euclid-dev.in2p3.fr is not mounted properly”)
is something that we should be worried ?
It seems that when directory is not empty it’s just show “Gateway reply: ok” for opening transaction.
P.S.: By the way, we use cvmfs-2.11.0 on our publishers.
Hmm, no, I wouldn’t expect the mountpoint /cvmfs/repo1 to be empty. I just checked with one testing publisher machine that has sat inactive for several days, and it was mounted correctly.
The printout you see also indicates that /var/spool/cvmfs/euclid-dev.in2p3.fr/rdonly was unmounted somehow.
I’d suggest to run mount -l and check the syslog whenever you see a publisher in this state the next time. You could also run the full report gathering of cvmfs_config bugreport
cvmfs-2.11.0 should be fine, but I’d recommend in any case to upgrade to the latest production version 2.12.7. There is an important fix for the timeout option of cvmfs_server transaction -t
Actually, since you mention another issue, updating to the latest version really helps for debugging to narrow down potential issues. Do feel free to let me know the details about that issue though!
Thank you @vavolkl
I took some time to describe our problem. But I think it’s better to create a separate topic, I’ll do it right after replying here.
But concerning the question of this topic.
As I said, in the situation when we observed that directory of our repository was empty (for exemaple /cvmfs/euclid-dev.in2p3.fr), if we open a transaction it will remount the CVMFS repository (comment #3).
$ cvmfs_server transaction euclid-dev.in2p3.fr
/var/spool/cvmfs/euclid-dev.in2p3.fr/rdonly is not mounted properly
/cvmfs/euclid-dev.in2p3.fr is not mounted properly
Trying to mount /var/spool/cvmfs/euclid-dev.in2p3.fr/rdonly... success
Trying to mount /cvmfs/euclid-dev.in2p3.fr... success
Gateway reply: ok
But when we abort or publish this transaction we have another strange message :
$ cvmfs_server abort euclid-dev.in2p3.fr
You are about to DISCARD ALL CHANGES OF THE CURRENT TRANSACTION for euclid-dev.in2p3.fr! Are you sure (y/N)? y
Gateway reply: ok
euclid-dev.in2p3.fr is not based on the newest published revision
Trying to unmount /cvmfs/euclid-dev.in2p3.fr... success
Trying to unmount /var/spool/cvmfs/euclid-dev.in2p3.fr/rdonly... success
Trying to mount /var/spool/cvmfs/euclid-dev.in2p3.fr/rdonly... success
Trying to mount /cvmfs/euclid-dev.in2p3.fr... success
Trying to remount /cvmfs/euclid-dev.in2p3.fr read/write... success
The strange message is “euclid-dev.in2p3.fr is not based on the newest published revision”.
Do you have any idea why our repository is not based on the latest revision ? Is it normal ?
P.S.: Thanks for suggestion to upgrade our CVMFS to 2.12.7. We are staring to think about it, but it could be not so easy in our situation. I’ll probably create another topic to ask some questions about upgrade to 2.12.7.
