What does "EIO (01) on local" error mean?

Hello,

in one of our laptops, when turning it on off-line I got this error:

(sie_u22.iac.es) failed to fetch sie_u22.iac.es:/SIE (9fb82f9be5c3311a69497939798162ab3b259164-shake128) (hash: 9fb82f9be5c3311a69497939798162ab3b259164-shake128, error 4 [failed to resolve host address])
(sie_u22.iac.es) EIO (01) on local

Any idea what could be causing this?

For completeness, I include below the context lines for this error in the journal for the “autofs” unit:

$ sudo journalctl -b 0 -u autofs
oct 22 09:07:35 xxx.xxx.iac.es systemd[1]: Starting Automounts filesystems on demand...
oct 22 09:07:45 xxx.xxx.iac.es automount[1223]: setautomntent: lookup(sss): setautomntent: Host is down
oct 22 09:07:55 xxx.xxx.iac.es automount[1223]: setautomntent: lookup(sss): setautomntent: Host is down
oct 22 09:07:57 xxx.xxx.iac.es automount[1223]: problem reading master map, maximum wait exceeded
oct 22 09:07:57 xxx.xxx.iac.es automount[1223]: /usr/sbin/automount: warning: could not read at least one map source after waiting, continuing ...
oct 22 09:07:57 xxx.xxx.iac.es systemd[1]: Started Automounts filesystems on demand.
oct 22 09:08:20 xxx.xxx.iac.es cvmfs2[3200]: (cvmfs-config.cern.ch) (manager 'default') switching proxy from (none) to DIRECT. Reason: set random start proxy from the first proxy group
oct 22 09:08:20 xxx.xxx.iac.es cvmfs2[3200]: (cvmfs-config.cern.ch) (manager 'default') GeoAPI request http://cernvmfs.gridpp.rl.ac.uk/cvmfs/cvmfs-config.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.o\
>
oct 22 09:08:20 xxx.xxx.iac.es cvmfs2[3200]: (cvmfs-config.cern.ch) (manager 'default') GeoAPI request http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/cvmfs-config.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-\
>
oct 22 09:08:20 xxx.xxx.iac.es cvmfs2[3200]: (cvmfs-config.cern.ch) (manager 'default') GeoAPI request http://cvmfs-stratum-one.cern.ch/cvmfs/cvmfs-config.cern.ch/api/v1.0/geo/@proxy@/cernvmfs.gridpp.rl.ac.uk,cvmfs-s1bnl.opensciencegrid.org,cvmfs-s1fnal.\
>
oct 22 09:08:20 xxx.xxx.iac.es cvmfs2[3200]: (cvmfs-config.cern.ch) (manager 'default') failed to retrieve geographic order from stratum 1 servers
oct 22 09:08:20 xxx.xxx.iac.es cvmfs2[3200]: (cvmfs-config.cern.ch) (manager 'default') switching proxy from (none) to DIRECT. Reason: cloned
oct 22 09:08:20 xxx.xxx.iac.es cvmfs2[3200]: (cvmfs-config.cern.ch) switching host from http://cvmfs-stratum-one.cern.ch/cvmfs/cvmfs-config.cern.ch to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/cvmfs-config.cern.ch (failed to resolve host address)
oct 22 09:08:20 xxx.xxx.iac.es cvmfs2[3200]: (cvmfs-config.cern.ch) switching host from http://cernvmfs.gridpp.rl.ac.uk/cvmfs/cvmfs-config.cern.ch to http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/cvmfs-config.cern.ch (failed to resolve host address)
oct 22 09:08:20 xxx.xxx.iac.es cvmfs2[3200]: (cvmfs-config.cern.ch) switching host from http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/cvmfs-config.cern.ch to http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/cvmfs-config.cern.ch (failed to resolve host addres\
>
oct 22 09:08:20 xxx.xxx.iac.es cvmfs2[3200]: (cvmfs-config.cern.ch) failed to download repository manifest (4 - failed to resolve host address)
oct 22 09:08:20 xxx.xxx.iac.es cvmfs2[3200]: (cvmfs-config.cern.ch) CernVM-FS: linking /cvmfs/cvmfs-config.cern.ch to repository cvmfs-config.cern.ch
oct 22 09:08:20 xxx.xxx.iac.es cvmfs2[3237]: (cvmfs-config.cern.ch) warning, could not apply updated catalog revision, entering offline mode
oct 22 09:08:20 xxx.xxx.iac.es cvmfs2[3259]: (sie_u22.iac.es) (manager 'default') switching proxy from (none) to DIRECT. Reason: set random start proxy from the first proxy group
oct 22 09:08:20 xxx.xxx.iac.es cvmfs2[3259]: (sie_u22.iac.es) (manager 'default') switching proxy from (none) to DIRECT. Reason: cloned
oct 22 09:08:20 xxx.xxx.iac.es cvmfs2[3259]: (sie_u22.iac.es) switching host from http://cern1.ll.iac.es/cvmfs/sie_u22.iac.es to http://cern2.ll.iac.es/cvmfs/sie_u22.iac.es (failed to resolve host address)
oct 22 09:08:20 xxx.xxx.iac.es cvmfs2[3259]: (sie_u22.iac.es) failed to download repository manifest (4 - failed to resolve host address)
oct 22 09:08:20 xxx.xxx.iac.es cvmfs2[3259]: (sie_u22.iac.es) CernVM-FS: linking /cvmfs/sie_u22.iac.es to repository sie_u22.iac.es
oct 22 09:08:20 xxx.xxx.iac.es cvmfs2[3294]: (sie_u22.iac.es) warning, could not apply updated catalog revision, entering offline mode
oct 22 09:08:20 xxx.xxx.iac.es cvmfs2[3294]: (sie_u22.iac.es) switching host from http://cern2.ll.iac.es/cvmfs/sie_u22.iac.es to http://cern1.ll.iac.es/cvmfs/sie_u22.iac.es (failed to resolve host address)
oct 22 09:08:20 xxx.xxx.iac.es cvmfs2[3294]: (sie_u22.iac.es) failed to fetch sie_u22.iac.es:/SIE (9fb82f9be5c3311a69497939798162ab3b259164-shake128) (hash: 9fb82f9be5c3311a69497939798162ab3b259164-shake128, error 4 [failed to resolve host address])
oct 22 09:08:20 xxx.xxx.iac.es cvmfs2[3294]: (sie_u22.iac.es) EIO (01) on local │
oct 22 09:11:20 xxx.xxx.iac.es cvmfs2[3237]: (cvmfs-config.cern.ch) switching host from http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/cvmfs-config.cern.ch to http://cvmfs-stratum-one.cern.ch/cvmfs/cvmfs-config.cern.ch (failed to resolve host address)
oct 22 09:11:20 xxx.xxx.iac.es cvmfs2[3294]: (sie_u22.iac.es) switching host from http://cern1.ll.iac.es/cvmfs/sie_u22.iac.es to http://cern2.ll.iac.es/cvmfs/sie_u22.iac.es (failed to resolve host address)
oct 22 09:11:20 xxx.xxx.iac.es cvmfs2[3237]: (cvmfs-config.cern.ch) switching host from http://cvmfs-stratum-one.cern.ch/cvmfs/cvmfs-config.cern.ch to http://cernvmfs.gridpp.rl.ac.uk/cvmfs/cvmfs-config.cern.ch (failed to resolve host address)
oct 22 09:11:20 xxx.xxx.iac.es cvmfs2[3294]: (sie_u22.iac.es) failed to download repository manifest (4 - failed to resolve host address)
oct 22 09:11:20 xxx.xxx.iac.es cvmfs2[3237]: (cvmfs-config.cern.ch) switching host from http://cernvmfs.gridpp.rl.ac.uk/cvmfs/cvmfs-config.cern.ch to http://cvmfs-s1bnl.opensciencegrid.org/cvmfs/cvmfs-config.cern.ch (failed to resolve host address)
oct 22 09:11:20 xxx.xxx.iac.es cvmfs2[3237]: (cvmfs-config.cern.ch) failed to download repository manifest (4 - failed to resolve host address)
oct 22 09:14:20 xxx.xxx.iac.es cvmfs2[3294]: (sie_u22.iac.es) recovered from offline mode
oct 22 09:14:21 xxx.xxx.iac.es cvmfs2[3237]: (cvmfs-config.cern.ch) recovered from offline mode
oct 22 09:17:02 xxx.xxx.iac.es cvmfs2[3294]: (sie_u22.iac.es) CernVM-FS: unmounted /cvmfs/sie_u22.iac.es (sie_u22.iac.es)
oct 22 09:17:02 xxx.xxx.iac.es cvmfs2[4708]: (sie_u22.iac.es) (manager 'default') switching proxy from (none) to DIRECT. Reason: set random start proxy from the first proxy group
oct 22 09:17:02 xxx.xxx.iac.es cvmfs2[4708]: (sie_u22.iac.es) (manager 'default') switching proxy from (none) to DIRECT. Reason: cloned
oct 22 09:17:02 xxx.xxx.iac.es cvmfs2[4708]: (sie_u22.iac.es) CernVM-FS: linking /cvmfs/sie_u22.iac.es to repository sie_u22.iac.es

Hi Angel,

the log message is a bit cryptic, but I think this is completely expected if the machine is offline. What happens is that cvmfs is trying to fetch a new catalog and fails. I should revisit the “offline mode” to make sure that the log better reflects what’s going on.

You don’t see any actual error after the network connection is re-established, right?

Cheers,
Valentin

It means it failed to look up a name in the DNS. There are many such errors in your log for different hosts, so it couldn’t connect to the DNS at all.

Hello,

this is the first time I see this particular error. We have configured this system in our laptops, so many times they start off-line, but never seen this one. Besides, I realized there was a problem with this machine because while off-line, the /cvmfs/sie_u22.iac.es directory was not readable (throwing and input/output error), despite having accessed it several times before while on-line, so it looks like the whole cache feature of CVMFS was not working at all. Once the networks connection was re-established, everything was fine again, but this was a bit troubling, since we rely on our laptops being able to access the CVMFS cache while off-line.

(sie_u22.iac.es) failed to fetch sie_u22.iac.es:/SIE (9fb82f9be5c3311a69497939798162ab3b259164-shake128) (hash: 9fb82f9be5c3311a69497939798162ab3b259164-shake128, error 4 [failed to resolve host address])
(sie_u22.iac.es) EIO (01) on local

Hi,

the concerning error was:

(sie_u22.iac.es) failed to fetch sie_u22.iac.es:/SIE (9fb82f9be5c3311a69497939798162ab3b259164-shake128) (hash: 9fb82f9be5c3311a69497939798162ab3b259164-shake128, error 4 [failed to resolve host address])
(sie_u22.iac.es) EIO (01) on local

which only appears once in the logs and for “local”.

Yes, the laptop was off-line, so all the other error messages about not being able to get in touch with the other servers are expected and don’t bother me, but I never got this particular error for off-line laptops (and in this particular situation the CVMFS cache was not working, so something strange going on, though I have no idea what…)

Many thanks

Hello

those EIO errors have been added in some of the more recent cvmfs version.
with cvmfs_talk -i <repo> internal affairs you get a better description of where those EIO error happened.

It is true that this is not very nice that there is just a number and not more descriptive explanation in the syslog message.

"eio.01",
     "EIO returned to calling process. cvmfs.cc:cvmfs_lookup()"

so the lookup failed because you were in offline mode and the file was not in the local cache.

OK, thank for the clarification.

In any case, this was really a one-off message and have never seen this again. Cheers,