OverlayFS on top of CephFS


has anybody tried to have the /cvmfs/ directory on the Stratum-0 mounted on CephFS?
I believe OverlayFS is “compatible” with CephFS, but not sure 100%. Any feedback if someone has every tried?


I’m not quite sure I understand the desired file system stack. On the stratum 0, the /cvmfs/$repo mountpoint is a virtual mountpoint combining
/var/spool/cvmfs/$repo/rdonly [read-only layer]
/var/spool/cvmfs/$repo/scratch/current [writable overlay]

Do you want to put the scratch directory on Ceph-FS? Or rather the /srv/cvmfs/$repo directory that contains the repository data in content-addressable form?


Hi Jakob,

sorry I was not clear. My bad.

Actually, I guess I am asking about all of them. But I was concerned about whatever mountpoint is made writable with the “cvmfs_server transaction” command. In our case, it is “/cvmfs/”, that’s why I mentioned it explicitly in my first post.
I am just wondering if anything would break if the underlying filesystem is CephFS, instead of xfs or zfs.

Makes sense?


Hi @jakob,

after a while, finally I got some time to keep investigating this.
So, I have tried to mount /var/spool/cvmfs on CephFS, now that I know that /cvmfs/ is just a virtual mountpoint.
Still no luck:

[root@host-172-16-112-130 ~]# mount -t ceph deneb-mon1.nubes.rl.ac.uk:/test/cvmfs_stratum0_test /mnt/ceph -o name=cvmfs_test,secret=AQA4fr...T7Q==

[root@host-172-16-112-130 ~]# mount -t ceph deneb-mon1.nubes.rl.ac.uk:/test/cvmfs_stratum0_test /var/spool/cvmfs -o name=cvmfs_test,secret=AQA4fr...T7Q==
[root@host-172-16-112-130 ~]# cvmfs_server mkfs -w http://host-172-16-112-130.nubes.stfc.ac.uk/cvmfs/test.iris.ac.uk -g -m -o cvmfs test.iris.ac.uk
Creating Configuration Files... done
Creating CernVM-FS Master Key and Self-Signed Certificate... done
Creating CernVM-FS Server Infrastructure... done
Creating Backend Storage... done
Signing 30 day whitelist with master key... done
Creating Initial Repository... done
Mounting CernVM-FS Storage... (overlayfs) mount: wrong fs type, bad option, bad superblock on overlay_test.iris.ac.uk,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
Allowing Replication of this Repository... done
/cvmfs/test.iris.ac.uk is not mounted properly.
fail! (health check after mount)

[root@host-172-16-112-130 ~]# ltr /mnt/ceph/test.iris.ac.uk/
total 1
drwxr-xr-x 1 cvmfs root   2 Aug 16 09:45 scratch
drwxr-xr-x 1 cvmfs root   0 Aug 16 09:45 rdonly
drwxr-xr-x 1 cvmfs root   0 Aug 16 09:45 ofs_workdir
-rw-r--r-- 1 cvmfs cvmfs 40 Aug 16 09:45 reflog.chksum
-rw-r--r-- 1 cvmfs cvmfs  0 Aug 16 09:45 client.local
drwxr-xr-x 1 cvmfs root   2 Aug 16 09:45 cache
srw-rw---- 1 cvmfs cvmfs  0 Aug 16 09:45 cvmfs_io
drwxr-xr-x 1 cvmfs root   1 Aug 16 09:45 tmp

[root@host-172-16-112-130 ~]# cvmfs_server transaction test.iris.ac.uk
/cvmfs/test.iris.ac.uk is not mounted properly
Job for cvmfs-test.iris.ac.uk.mount failed. See "systemctl status cvmfs-test.iris.ac.uk.mount" and "journalctl -xe" for details.
Trying to mount /cvmfs/test.iris.ac.uk... fail
(unexpected termination) Trying to mount /cvmfs/test.iris.ac.uk... fail

/lib64/libcvmfs_server.so.2.9.2(+0x78c93) [0x7fcbb4220c93]
/lib64/libcvmfs_server.so.2.9.2(+0x7fe4d) [0x7fcbb4227e4d]
/lib64/libcvmfs_server.so.2.9.2(_ZN7publish9Publisher11ManagedNode15AlterMountpointENS1_22EMountpointAlterationsEi+0x6fe) [0x7fcbb422cc14]
/lib64/libcvmfs_server.so.2.9.2(_ZN7publish9Publisher11ManagedNode5CheckEb+0x4d8) [0x7fcbb422d764]
/lib64/libcvmfs_server.so.2.9.2(_ZN7publish9Publisher16TransactionRetryEv+0x2b) [0x7fcbb42307c5]
/lib64/libcvmfs_server.so.2.9.2(_ZN7publish9Publisher11TransactionEv+0xe) [0x7fcbb4221174]
/usr/bin/cvmfs_publish() [0x438e55]
/usr/bin/cvmfs_publish() [0x40f505]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7fcbb26da555]
/usr/bin/cvmfs_publish() [0x40f723]
[root@host-172-16-112-130 ~]# vi /cvmfs/test.iris.ac.uk/test_1
[root@host-172-16-112-130 ~]# cvmfs_server publish test.iris.ac.uk
/cvmfs/test.iris.ac.uk is not mounted properly.
Note: Trying to mount /cvmfs/test.iris.ac.uk... Job for cvmfs-test.iris.ac.uk.mount failed. See "systemctl status cvmfs-test.iris.ac.uk.mount" and "journalctl -xe" for details.

It looks like the directory for the new repository is created in CephFS anyways, but clearly things didn’t work correctly. It is not possible to open a transaction.

At this point I am wondering if the problem is that what I am trying is not possible, or the lack of proper kernel packages on this testbed.
I see this in the journal:

Aug 16 12:42:39 host-172-16-112-130.nubes.stfc.ac.uk kernel: overlayfs: filesystem on '/var/spool/cvmfs/test.iris.ac.uk/scratch/current' not supported as upperdir

Any comment is more than welcome.

I’m afraid that this is an overlayfs limitation we cannot workaround and Ceph-FS is thus unsupported for the /var/spool/cvmfs area.

I see. Fair enough.
Thanks a lot.