Uploader error in publisher

Publisher / Stratum 0
1

When publishing a transaction from a publisher machine I get the following error:

# cvmfs_server publish herd.sw.common/centos7/x86_64-opt/
Using auto tag 'generic-2024-07-22T17:19:02Z'
Warning: CAP_DAC_READ_SEARCH cannot be obtained. It's not in the process's permitted set.
Processing changes...
Waiting for upload of files before committing...
Committing file catalogs...
Wait for all uploads to finish
SessionContext::DoUpload - error reply: {"reason":"uploader_error","status":"error"}
terminate called after throwing an instance of 'ECvmfsException'
  what():  PANIC: /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker-x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.11.3/cvmfs/session_context.cc : 416
SessionContext: could not submit payload. Aborting.
/usr/bin/cvmfs_server: line 4111:  4719 Killed                  $user_shell "$sync_command"
Synchronization failed

Executed Command:
cvmfs_swissknife sync -u /cvmfs/herd.sw.common -s /var/spool/cvmfs/herd.sw.common/scratch/current -c /var/spool/cvmfs/herd.sw.common/rdonly -t /var/spool/cvmfs/herd.sw.common/tmp -b 80482b57b2d6bd1d9736ea154a082550133131d9 -r gw,/srv/cvmfs/herd.sw.common/data/txn,http://gateway.131.154.99.234.myip.cloud.infn.it/api/v1 -w https://minio.herd.cloud.infn.it/cvmfs/herd.sw.common -o /var/spool/cvmfs/herd.sw.common/tmp/manifest -e sha1 -Z default -C /etc/cvmfs/repositories.d/herd.sw.common/trusted_certs -N herd.sw.common -K /etc/cvmfs/keys/herd.sw.common.pub -L -D generic-2024-07-22T17:19:02Z -H /etc/cvmfs/keys/herd.sw.common.gw -P /var/spool/cvmfs/herd.sw.common/session_token -f overlayfs -p -l 4194304 -a 8388608 -h 16777216 -i

Publisher version is 2.11.3, while server and gateway are 2.11.2. I don’t have a clue about what might cause it so any help is greatly appreciated.

2

Hi Nicola,

this is strange - did you just set up the publisher or were you able to publish before?

You could try running cvmfs-server fix-permissions. This should have been run as a post-script when installing the rpms though.
Cheers,
Valentin

3

I had a S3 backend secret containing an & character which turned out to screw up everything. Setting a new key pair without problematic characters fixed the issue. It took me a lot of time and pain since the error message is absolutely useless to understand the problem but I did it.

4

OK! I agree about the error message, uploader_error gives an indication, but we could certainly be more specific. Also the warning for the capabilities was a red herring - let me reproduce this and see what we can improve.