CernVM4 and parrot_run: does parrot_run need an upgrade?

CernVM Appliance
1

Hi,

parrot_run installation on CernVM4 (5.4.19) seems to have a weird behaviour when used with certain applications.
I got this issue whith an lhcb-specific application (lb-run) and do not know if this is very specific to this software or a general issue.

How to reproduce it:

  • I instantiate a Singularity container of cernvm4, binding /cvmfs on it:
$ singularity exec --cleanenv --bind /cvmfs /cvmfs/cernvm-prod.cern.ch/cvm4/ bash --norc
  • I source LbEnv to get the appropriate environment to execute the command that I want to trace:
$ source /cvmfs/lhcb.cern.ch/lib/LbEnv.sh
  • According to the cvmfs documentation, I need to set up the environment to execute parrot_run:
$ export PARROT_ALLOW_SWITCHING_CVMFS_REPOSITORIES=yes
$ export PARROT_CVMFS_REPO="lhcb.cern.ch:url=http://cvmfs-stratum-one.cern.ch/cvmfs/lhcb.cern.ch,pubkey=/cvmfs/cvmfs-config.cern.ch/etc/cvmfs/keys/cern.ch/cern-it1.cern.ch.pub lhcb-condb.cern.ch:url=http://cvmfs-stratum-one.cern.ch/cvmfs/lhcb-condb.cern.ch,pubkey=/cvmfs/cvmfs-config.cern.ch/etc/cvmfs/keys/cern.ch/cern-it1.cern.ch.pub"
$ export HTTP_PROXY='<content of CVMFS_HTTP_PROXY available in /etc/cvmfs/config.d/lhcb.cern.ch.local>;DIRECT'
  • Finally I execute parrot_run:
$ parrot_run --name-list namelist --env-list envlist lb-run --siteroot=/cvmfs/lhcb.cern.ch/lib/ -c x86_64-slc6-gcc48-opt --use="AppConfig v3r395" --use="Gen/DecFiles v30r51" --use="ProdConf" Gauss/v49r18 gaudirun.py '$APPCONFIGOPTS/Gauss/Sim08-Beam4000GeV-mu100-2012-nu2.5.py' '$APPCONFIGOPTS/Gauss/DataType-2012.py' '$APPCONFIGOPTS/Gauss/RICHRandomHits.py' '$APPCONFIGOPTS/Gauss/NoPacking.py' '$DECFILES
ROOT/options/12877041.py' '$LBPYTHIA8ROOT/options/Pythia8.py' '$APPCONFIGOPTS/Gauss/G4PL_FTFP_BERT_EmNoCuts.py' '$APPCONFIGOPTS/Persistency/Compression-ZLIB-1.py' $PWD/prodConf.py

with prodConf.py being:

from ProdConf import ProdConf

ProdConf(
  NOfEvents=1,
  DDDBTag='dddb-20170721-3',
  AppVersion='v49r18',
  XMLSummaryFile='summaryGauss_00117610_00013443_1.xml',
  Application='Gauss',
  OutputFilePrefix='00117610_00013443_1',
  RunNumber=11774443,
  XMLFileCatalog='pool_xml_catalog.xml',
  FirstEventNumber=7877013,
  CondDBTag='sim-20170721-2-vc-mu100',
  OutputFileTypes=['sim'],
)

Most of the time, very randomly, I get the following issue:

2021/05/18 11:22:54.48 parrot_run[23981] <child:24468> fatal: cannot run application which does not allow tracing (CLONE_UNTRACED)
2021/05/18 11:22:54.48 parrot_run[23981] <child:24468> notice: received signal 15 (Terminated), killing all my children...
2021/05/18 11:22:54.48 parrot_run[23981] <child:24468> notice: sending myself 15 (Terminated), goodbye!

Sometimes, the application is able to run, this seems completely random.

I tried to run the same application with a newer version of parrot_run (7.1.11) that I downloaded as follows:

curl -O ccl.cse.nd.edu/software/files/cctools-7.1.11-x86_64-centos7.tar.gz
tar -zxf cctools-7.1.11-x86_64-centos7.tar.gz
cctools-7.1.11-x86_64-centos7/bin/parrot_run ...

I encountered no issue with this version, the application was able to complete its execution.

Thus, this seems specific to the parrot_run version of CernVM4.
Do you think it is a general issue? In this case, maybe parrot_run should be upgraded

Thanks

2

Many thanks for the detailed investigation! You’re right, we need to update parrot. I’ll look into it and let post here when a new version is available.

3

Meanwhile, I’d like to point out cvmfsexec. It provides unprivileged access to /cvmfs repositories. I didn’t try it yet from inside a Singularity container, but it may work. @dwd do you have experience with cvmfsexec inside a singularity container?

4

I think cvmfsexec (in some of its modes, depending on the kernel features available) should work as long as it has some writable scratch space to work with, but I am unclear on the higher level goal here. The singularity command line already binds /cvmfs, so what is parrot (or cvmfsexec) for? Does /cvmfs not exist outside of singularity, and you’re trying to export it from singularity? Exporting is not going to work. Or is the --bind /cvmfs option just not doing anything? What are the capabilities of the host operating system?

5

Well here is the context: we are interacting with a supercomputer that does not have any external connectivity at all (neither on the worker nodes nor on the edge nodes).
we are building a pipeline that:

  • uses parrot_run to trace applications of interest.
  • Once we get the dependencies of the applications, uses the cvmfs_shrinkwrap tool to create a subset of cvmfs repository containing the required dependencies (because we try not to export the whole repository).
  • tests it within CernVM
  • exports it to the supercomputer.
6

Ok, that’s a very good use case for parrot.

7

The new parrot version 7.2.8 is now published.

Cheers,
Jakob

8

Seems to work fine now.
Thank you very much.