Data corruption in .cvmfsreflog

Publisher / Stratum 0
1

Dear all

I have a repo using a S3 backend implemented using OpenStack swift

I have a problem that I am not able to understand (and recover a working environment).

I start a transaction and I try to add a file to the repo:

[root@cvmfs2v209 centos]# cvmfs_server transaction gmalatesta.infn.it
[root@cvmfs2v209 centos]# ll /cvmfs/gmalatesta.infn.it/
total 1
-rw-r–r–. 1 root root 0 May 25 16:29 azqdd
[root@cvmfs2v209 centos]# touch /cvmfs/gmalatesta.infn.it/file2
[root@cvmfs2v209 centos]# ll /cvmfs/gmalatesta.infn.it/
total 1
-rw-r–r–. 1 root root 0 May 25 16:29 azqdd
-rw-r–r–. 1 root root 0 May 25 18:15 file2

The publish fails reporting data corruption with cvmfsreflog:

[root@cvmfs2v209 centos]# cvmfs_server publish gmalatesta.infn.it
Using auto tag ‘generic-2023-05-25T16:16:05Z’
Processing changes…
Waiting for upload of files before committing…
Committing file catalogs…
Wait for all uploads to finish
Exporting repository manifest
Statistics stored at: /var/spool/cvmfs/gmalatesta.infn.it/stats.db
Tagging gmalatesta.infn.it
Flushing file system buffers
Signing new manifest
data corruption in .cvmfsreflog for gmalatesta.infn.it, run cvmfs_server check -r to fix
reflog missing
Signing failed

I then try to abort the transaction:

[root@cvmfs2v209 centos]# cvmfs_server abort gmalatesta.infn.it
You are about to DISCARD ALL CHANGES OF THE CURRENT TRANSACTION for gmalatesta.infn.it! Are you sure (y/N)? y
[root@cvmfs2v209 centos]#

I am not able to I fix the data corruption using “cvmfs_server check -r” as suggested by the error message [*]

Any hint would be really appreciated :slight_smile:

In general: we are seeing a lot of problems with repos hosted on this S3 swift backend. I am trying to see if there are some problems with the infrastructure, but first of all I would like to undertand if there are known problems with swift

Thanks, Massimo

[*]

[root@cvmfs2v209 centos]# cvmfs_server check -r gmalatesta.infn.it
Verifying integrity of gmalatesta.infn.it…
Info: found https://stor.cloud.infn.it/v1/AUTH_79322ee743c74382ad9de8b1d895616a/cvmfsc7v209/gmalatesta.infn.it/.cvmfsreflog with content hash ca87040c78650a6ec85bcbde36106b9b6922c17d
Warning: restored reflog checksum as ca87040c78650a6ec85bcbde36106b9b6922c17d (was: 7c0e551e4b1bbbbaf2fccb690d4ddac649afb6c3)
Warning: a reflog hash is registered in the manifest, re-creating missing reflog
Certificate: 2a8d2b12974b126ab3af60d42f83624854a89294
Metainfo: 85e4f386da0d5d77c7f66cf2f58e10951f24b653
Catalog: cf258553004dc3a94fd02b36024528464043001a Revision: 61
Catalog: 51c6219e4c3409421a89bad4bd900c6ae254bc06 Revision: 60
Catalog: 8bd37a006d60575984bea719a02578b6c8903238 Revision: 59
Catalog: ac2f752c44ea14eeb02a00a12b070d1b63cd5c1e Revision: 58

…

…

History: 3483cb99e5c528933c4bdd6269ad6a2c6cafe263
History: 0cbf37e8c4c7efff23b8e1fd9fdca3f9cf67493b
found 212 entries
Inspecting log of references
The .cvmfsreflog has unexpected content hash ca87040c78650a6ec85bcbde36106b9b6922c17d (expected 495cea1516cd58ea64023f59065a1de9f247d7cd)
failed to verify reflog
[root@cvmfs2v209 centos]#

2

I retried this morning and I don’t see anymore the problem
However I see another issue with the same problem.

I start a transaction and I write a new file in that repo:

[root@cvmfs2v209 centos]# cvmfs_server transaction gmalatesta.infn.it
CernVM-FS transaction error: another transaction is already open

[root@cvmfs2v209 centos]# cvmfs_server publish gmalatesta.infn.it
Using auto tag ‘generic-2023-05-26T06:03:30Z’
Processing changes…
Waiting for upload of files before committing…
Committing file catalogs…
Wait for all uploads to finish
Exporting repository manifest
Statistics stored at: /var/spool/cvmfs/gmalatesta.infn.it/stats.db
Tagging gmalatesta.infn.it
Flushing file system buffers
Signing new manifest
Remounting newly created repository revision

[root@cvmfs2v209 centos]#
[root@cvmfs2v209 centos]# ll /cvmfs/gmalatesta.infn.it/
total 4
-rw-r–r–. 1 root root 0 May 25 16:29 azqdd
-rw-r–r–. 1 root root 3523 May 26 08:03 fstab

So far so good

If then start another transaction I am told that the repo is not based on the newest published revision:

[root@cvmfs2v209 centos]# cvmfs_server transaction gmalatesta.infn.it
gmalatesta.infn.it is not based on the newest published revision
Trying to unmount /cvmfs/gmalatesta.infn.it… success
Trying to unmount /var/spool/cvmfs/gmalatesta.infn.it/rdonly… success
Trying to mount /var/spool/cvmfs/gmalatesta.infn.it/rdonly… success
Trying to mount /cvmfs/gmalatesta.infn.it… success

and the content of the repo is empty:

[root@cvmfs2v209 centos]# ll /cvmfs/gmalatesta.infn.it/
total 1
-rw-r–r–. 1 root root 0 May 25 16:29 azqdd
[root@cvmfs2v209 centos]#

3

This is not unexpected, but still, you should see your updated show up in the repository. You also don’t see them in the client on a different machine?

If there a public stratum1 where this repository is replicated, could you send us the URL? Otherwise, would it be possible to get access to the publisher?