Data corruption in .cvmfsreflog

[sgaravat]

Dear all

I have a repo using a S3 backend implemented using OpenStack swift

I have a problem that I am not able to understand (and recover a working environment).

I start a transaction and I try to add a file to the repo:

[root@cvmfs2v209 centos]# cvmfs_server transaction gmalatesta.infn.it
[root@cvmfs2v209 centos]# ll /cvmfs/gmalatesta.infn.it/
total 1
-rw-r–r–. 1 root root 0 May 25 16:29 azqdd
[root@cvmfs2v209 centos]# touch /cvmfs/gmalatesta.infn.it/file2
[root@cvmfs2v209 centos]# ll /cvmfs/gmalatesta.infn.it/
total 1
-rw-r–r–. 1 root root 0 May 25 16:29 azqdd
-rw-r–r–. 1 root root 0 May 25 18:15 file2

The publish fails reporting data corruption with cvmfsreflog:

[root@cvmfs2v209 centos]# cvmfs_server publish gmalatesta.infn.it
Using auto tag ‘generic-2023-05-25T16:16:05Z’
Processing changes…
Waiting for upload of files before committing…
Committing file catalogs…
Wait for all uploads to finish
Exporting repository manifest
Statistics stored at: /var/spool/cvmfs/gmalatesta.infn.it/stats.db
Tagging gmalatesta.infn.it
Flushing file system buffers
Signing new manifest
data corruption in .cvmfsreflog for gmalatesta.infn.it, run cvmfs_server check -r to fix
reflog missing
Signing failed

I then try to abort the transaction:

[root@cvmfs2v209 centos]# cvmfs_server abort gmalatesta.infn.it
You are about to DISCARD ALL CHANGES OF THE CURRENT TRANSACTION for gmalatesta.infn.it! Are you sure (y/N)? y
[root@cvmfs2v209 centos]#

I am not able to I fix the data corruption using “cvmfs_server check -r” as suggested by the error message [*]

Any hint would be really appreciated

In general: we are seeing a lot of problems with repos hosted on this S3 swift backend. I am trying to see if there are some problems with the infrastructure, but first of all I would like to undertand if there are known problems with swift

Thanks, Massimo

[*]

[root@cvmfs2v209 centos]# cvmfs_server check -r gmalatesta.infn.it
Verifying integrity of gmalatesta.infn.it…
Info: found https://stor.cloud.infn.it/v1/AUTH_79322ee743c74382ad9de8b1d895616a/cvmfsc7v209/gmalatesta.infn.it/.cvmfsreflog with content hash ca87040c78650a6ec85bcbde36106b9b6922c17d
Warning: restored reflog checksum as ca87040c78650a6ec85bcbde36106b9b6922c17d (was: 7c0e551e4b1bbbbaf2fccb690d4ddac649afb6c3)
Warning: a reflog hash is registered in the manifest, re-creating missing reflog
Certificate: 2a8d2b12974b126ab3af60d42f83624854a89294
Metainfo: 85e4f386da0d5d77c7f66cf2f58e10951f24b653
Catalog: cf258553004dc3a94fd02b36024528464043001a Revision: 61
Catalog: 51c6219e4c3409421a89bad4bd900c6ae254bc06 Revision: 60
Catalog: 8bd37a006d60575984bea719a02578b6c8903238 Revision: 59
Catalog: ac2f752c44ea14eeb02a00a12b070d1b63cd5c1e Revision: 58

…

…

History: 3483cb99e5c528933c4bdd6269ad6a2c6cafe263
History: 0cbf37e8c4c7efff23b8e1fd9fdca3f9cf67493b
found 212 entries
Inspecting log of references
The .cvmfsreflog has unexpected content hash ca87040c78650a6ec85bcbde36106b9b6922c17d (expected 495cea1516cd58ea64023f59065a1de9f247d7cd)
failed to verify reflog
[root@cvmfs2v209 centos]#

[sgaravat]

I retried this morning and I don’t see anymore the problem
However I see another issue with the same problem.

I start a transaction and I write a new file in that repo:

[root@cvmfs2v209 centos]# cvmfs_server transaction gmalatesta.infn.it
CernVM-FS transaction error: another transaction is already open

[root@cvmfs2v209 centos]# cvmfs_server publish gmalatesta.infn.it
Using auto tag ‘generic-2023-05-26T06:03:30Z’
Processing changes…
Waiting for upload of files before committing…
Committing file catalogs…
Wait for all uploads to finish
Exporting repository manifest
Statistics stored at: /var/spool/cvmfs/gmalatesta.infn.it/stats.db
Tagging gmalatesta.infn.it
Flushing file system buffers
Signing new manifest
Remounting newly created repository revision

[root@cvmfs2v209 centos]#
[root@cvmfs2v209 centos]# ll /cvmfs/gmalatesta.infn.it/
total 4
-rw-r–r–. 1 root root 0 May 25 16:29 azqdd
-rw-r–r–. 1 root root 3523 May 26 08:03 fstab

So far so good

If then start another transaction I am told that the repo is not based on the newest published revision:

[root@cvmfs2v209 centos]# cvmfs_server transaction gmalatesta.infn.it
gmalatesta.infn.it is not based on the newest published revision
Trying to unmount /cvmfs/gmalatesta.infn.it… success
Trying to unmount /var/spool/cvmfs/gmalatesta.infn.it/rdonly… success
Trying to mount /var/spool/cvmfs/gmalatesta.infn.it/rdonly… success
Trying to mount /cvmfs/gmalatesta.infn.it… success

and the content of the repo is empty:

[root@cvmfs2v209 centos]# ll /cvmfs/gmalatesta.infn.it/
total 1
-rw-r–r–. 1 root root 0 May 25 16:29 azqdd
[root@cvmfs2v209 centos]#

[vavolkl]

@sgaravat

If then start another transaction I am told that the repo is not based on the newest published revision

This is not unexpected, but still, you should see your updated show up in the repository. You also don’t see them in the client on a different machine?

If there a public stratum1 where this repository is replicated, could you send us the URL? Otherwise, would it be possible to get access to the publisher?