CernVM4 and parrot_run: does parrot_run need an upgrade?


parrot_run installation on CernVM4 (5.4.19) seems to have a weird behaviour when used with certain applications.
I got this issue whith an lhcb-specific application (lb-run) and do not know if this is very specific to this software or a general issue.

How to reproduce it:

  • I instantiate a Singularity container of cernvm4, binding /cvmfs on it:
$ singularity exec --cleanenv --bind /cvmfs /cvmfs/ bash --norc
  • I source LbEnv to get the appropriate environment to execute the command that I want to trace:
$ source /cvmfs/
  • According to the cvmfs documentation, I need to set up the environment to execute parrot_run:
$ export PARROT_CVMFS_REPO=",pubkey=/cvmfs/,pubkey=/cvmfs/"
$ export HTTP_PROXY='<content of CVMFS_HTTP_PROXY available in /etc/cvmfs/config.d/>;DIRECT'
  • Finally I execute parrot_run:
$ parrot_run --name-list namelist --env-list envlist lb-run --siteroot=/cvmfs/ -c x86_64-slc6-gcc48-opt --use="AppConfig v3r395" --use="Gen/DecFiles v30r51" --use="ProdConf" Gauss/v49r18 '$APPCONFIGOPTS/Gauss/' '$APPCONFIGOPTS/Gauss/' '$APPCONFIGOPTS/Gauss/' '$APPCONFIGOPTS/Gauss/' '$DECFILES
ROOT/options/' '$LBPYTHIA8ROOT/options/' '$APPCONFIGOPTS/Gauss/' '$APPCONFIGOPTS/Persistency/' $PWD/

with being:

from ProdConf import ProdConf


Most of the time, very randomly, I get the following issue:

2021/05/18 11:22:54.48 parrot_run[23981] <child:24468> fatal: cannot run application which does not allow tracing (CLONE_UNTRACED)
2021/05/18 11:22:54.48 parrot_run[23981] <child:24468> notice: received signal 15 (Terminated), killing all my children...
2021/05/18 11:22:54.48 parrot_run[23981] <child:24468> notice: sending myself 15 (Terminated), goodbye!

Sometimes, the application is able to run, this seems completely random.

I tried to run the same application with a newer version of parrot_run (7.1.11) that I downloaded as follows:

curl -O
tar -zxf cctools-7.1.11-x86_64-centos7.tar.gz
cctools-7.1.11-x86_64-centos7/bin/parrot_run ...

I encountered no issue with this version, the application was able to complete its execution.

Thus, this seems specific to the parrot_run version of CernVM4.
Do you think it is a general issue? In this case, maybe parrot_run should be upgraded


Many thanks for the detailed investigation! You’re right, we need to update parrot. I’ll look into it and let post here when a new version is available.

Meanwhile, I’d like to point out cvmfsexec. It provides unprivileged access to /cvmfs repositories. I didn’t try it yet from inside a Singularity container, but it may work. @dwd do you have experience with cvmfsexec inside a singularity container?

I think cvmfsexec (in some of its modes, depending on the kernel features available) should work as long as it has some writable scratch space to work with, but I am unclear on the higher level goal here. The singularity command line already binds /cvmfs, so what is parrot (or cvmfsexec) for? Does /cvmfs not exist outside of singularity, and you’re trying to export it from singularity? Exporting is not going to work. Or is the --bind /cvmfs option just not doing anything? What are the capabilities of the host operating system?

Well here is the context: we are interacting with a supercomputer that does not have any external connectivity at all (neither on the worker nodes nor on the edge nodes).
we are building a pipeline that:

  • uses parrot_run to trace applications of interest.
  • Once we get the dependencies of the applications, uses the cvmfs_shrinkwrap tool to create a subset of cvmfs repository containing the required dependencies (because we try not to export the whole repository).
  • tests it within CernVM
  • exports it to the supercomputer.

Ok, that’s a very good use case for parrot.

The new parrot version 7.2.8 is now published.


Seems to work fine now.
Thank you very much.