Deleted files still exists during a transaction

Publisher / Stratum 0
1

Hi,
When opening a transaction and deleting some files, they still exist until the transaction is published. Is it possible to change this behavior?

It creates some weird things in the file system:

ls -l test/
ls: cannot access test/foo: No such file or directory
total 0
?????????? ? ? ? ?            ? foo

Thanks

2

That is a strange situation. Can you let us know the version of cvmfs you are using and the exact steps (or provide a small reproducer)?

3

cvmfs version :

cvmfs-2.9.4-1.el7.x86_64
cvmfs-server-2.9.0-1.el7.x86_64

Step to reproduce :

cvmfs_server transaction repo-test
rm /cvmfs/repo-test/foo
ls -l /cvmfs/repo-test/foo
ls: cannot access /cvmfs/repo-test/foo: No such file or directory
total 0
?????????? ? ? ? ?            ? foo

After a publish the file is completely gone.

4

Trying to reproduce… is the repository configured in any non-standard way? Or is it simply cvmfs_server mkfs ...? I guess it is also a standard CentOS 7?

5

It was created a long time ago, with mkfs command. Yes standard centos 7.

6

Apologies for returning to this issue only now!

I wasn’t able to reproduce what you see but I do notice that client and server have different versions. Could you update them both to version 2.9.4 (you can also wait for a few more days and update both to version 2.10). If that doesn’t help, could you paste the output of cat /proc/mounts and uname -a. It may be an overlayfs related issue.

7

No change with cvmfs-2.9.4-1.

cat /proc/mounts
repo /var/spool/cvmfs/repo/rdonly fuse ro,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other 0 0
overlay_repo /cvmfs/repo overlay ro,nodev,relatime,lowerdir=/var/spool/cvmfs/repo/rdonly,upperdir=/var/spool/cvmfs/repo/scratch/current,workdir=/var/spool/cvmfs/repo/ofs_workdir 0 0

uname -a
Linux stratum0 3.10.0-1160.59.1.el7.x86_64 #1 SMP Wed Feb 23 16:47:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
8

That looks ok.

It would be useful to also have the cat /proc/mounts entries from the file system hosting /var/spool/cvmfs. Overlayfs is sensitive to the exact file system settings of its layers. For XFS, it is important that the file system is create with ftype=1 option, which you can check by

xfs_info /var/spool/cvmfs/ | grep ftype
9 
repo /var/spool/cvmfs/repo/rdonly fuse ro,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other 0 0
overlay_repo /cvmfs/repo overlay ro,nodev,relatime,lowerdir=/var/spool/cvmf /repo/rdonly,upperdir=/var/spool/cvmfs/repo/scratch/current,workdir=/var/spool/cvmfs/repo/ofs_workdir 0 0

Indeed, ftype is set to 0 :

xfs_info /var/spool/cvmfs/ | grep ftype
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
10

That explains it. Overlayfs is not really supported on XFS with ftype=0. So you’d need to move /var/spool/cvmfs to a ext4 or xfs with ftype=1.