Unable to create a new repo with data stored in S3-compatible storage

Publisher / Stratum 0
21

Yes, that seems to work.

# curl -I https://s3.echo.stfc.ac.uk/
HTTP/1.1 200 OK
Transfer-Encoding: chunked
x-amz-request-id: tx0000000000000007d7717-00610ce22f-16db61985-default
Content-Type: application/xml
Date: Fri, 06 Aug 2021 07:18:07 GMT
22

It took a bit longer than expected, but now there is a new nightly build which should work thanks to this fix.

Cheers,
Jakob

23

Hi @jakob

Apparently, still not working for me:

[root@host-172-16-112-130 ~]# s3cmd --config creds ls s3://cvmfs_test_s3_egi_eu
                          DIR  s3://cvmfs_test_s3_egi_eu/test_s3.egi.eu/
2021-06-23 11:24           23  s3://cvmfs_test_s3_egi_eu/test
2021-06-24 13:55           23  s3://cvmfs_test_s3_egi_eu/test2

[root@host-172-16-112-130 ~]# rpm -qa | grep cvmfs
cvmfs-server-2.9.0-0.2994.e7c4465d2876a852git.el7.x86_64
cvmfs-2.9.0-0.2994.e7c4465d2876a852git.el7.x86_64
cvmfs-config-none-1.0-2.noarch

[root@host-172-16-112-130 ~]# cat s3.conf
CVMFS_S3_ACCESS_KEY=****
CVMFS_S3_SECRET_KEY=****
CVMFS_S3_HOST=s3.echo.stfc.ac.uk 
CVMFS_S3_BUCKET=cvmfs_test_s3_egi_eu
CVMFS_S3_USE_HTTPS=true
CVMFS_S3_DNS_BUCKETS=false 

[root@host-172-16-112-130 ~]# CVMFS_SERVER_DEBUG=3 cvmfs_server mkfs -s s3.conf -w 
https://s3.echo.stfc.ac.uk/cvmfs_test_s3_egi_eu -g -m -o cvmfs test_s3.egi.eu
Creating Configuration Files... done
Creating CernVM-FS Master Key and Self-Signed Certificate... done
Creating CernVM-FS Server Infrastructure... done
Signing 30 day whitelist with master key... done
Creating Initial Repository... Cannot find S3 config file at 's3.conf'
PANIC: /home/sftnight/jenkins/workspace/CvmfsFullBuildDocker/CVMFS_BUILD_ARCH/docker- 
x86_64/CVMFS_BUILD_PLATFORM/cc7/build/BUILD/cvmfs-2.9.0/cvmfs/upload_s3.cc : 51
Error in parsing the spooler definition
Aborted
fail! (cannot init repo)

Do you see something wrong with my s3.conf?

Thanks a lot.
Jose

24

Could it be that s3.conf is a symlink? If so, please try making it a real file. We’ll fix that issue shortly.

25

Nope, it is not a symlink.

26

Hm, please try with an absolute path name for s3.conf

27

That may be possible. Before seeing your suggestion, I moved the file to /tmp/, and using -s /tmp/s3.conf worked.

28

However, “-s /root/s3.conf” fails as before. The problem seems to be the file cannot be in the home directory of root.

29

That makes sense. The mkfs command assigns ownership of the repository to the cvmfs user (option -o cvmfs). So that user needs to be able to read the S3 config file.

30

I see.
In any case, with the config file in the right place, everything seems to work fine now, I believe:

[root@host-172-16-112-130 ~]# s3cmd --config creds ls s3://cvmfs_test_s3_egi_eu/test_s3.egi.eu/
                          DIR  s3://cvmfs_test_s3_egi_eu/test_s3.egi.eu/data/
2021-09-15 09:39            0  s3://cvmfs_test_s3_egi_eu/test_s3.egi.eu/.cvmfs_master_replica
2021-09-15 09:39          595  s3://cvmfs_test_s3_egi_eu/test_s3.egi.eu/.cvmfspublished
2021-09-15 09:39         5120  s3://cvmfs_test_s3_egi_eu/test_s3.egi.eu/.cvmfsreflog
2021-09-15 09:39          407  s3://cvmfs_test_s3_egi_eu/test_s3.egi.eu/.cvmfswhitelist

Thanks a lot !!!
Jose