Arch Linux

aldur999

Member

Registered: 2015-09-04

Posts: 2

Journalctl --follow and "Input/output error"

I have a systemd service configured to ncat logs to a remote log managing service.

The shell command is something of the form:

journalctl -o short -f | ncat --send-only REMOTE_HOST REMOTE_PORT

Everything works great, except that, every hour or two, the service fails and I get an "Input/output error." from ncat.
Relevant error logs:

Sep 04 06:01:02 hostname systemd[1]: Started Papertrail.
[...]
Sep 04 08:01:01 hostname sh[1313]: Ncat: Input/output error.
Sep 04 08:01:01 hostname systemd[1]: papertrail.service: Main process exited, code=exited, status=1/FAILURE
Sep 04 08:01:01 hostname systemd[1]: papertrail.service: Unit entered failed state.
Sep 04 08:01:01 hostname systemd[1]: papertrail.service: Failed with result 'exit-code'.

I've tried investigating the issue, setting a large timeout for the idle ncat operations, but it seems that the problem is coming from journalctl.
I've also managed to "solve" the issue by restarting the service every time it fails, but it is a not-very-elegant solution.

Ideas?

The full service code is shown below.

[Unit]
Description=Papertrail
Wants=network-online.target systemd-resolved.service
After=network-online.target systemd-journald.service systemd-resolved.service
Requires=systemd-journald.service

[Service]
ExecStart=/bin/sh -c "journalctl -o short -f | ncat --send-only --ssl CENSORED CENSORED"
TimeoutStartSec=0
Restart=on-failure

[Install]
WantedBy=multi-user.target

Last edited by aldur999 (2015-09-04 07:31:29)

seqizz

Member

Registered: 2011-04-04

Posts: 37

Re: Journalctl --follow and "Input/output error"

two ideas:
- timeout because of no active usage after some time
- rotation triggers something (e.g. like this)

---

Creeds matter very little… The optimist proclaims that we live in the best of all possible worlds; and the pessimist fears this is true. So I elect for neither label. - James Branch Cabell

aldur999

Member

Registered: 2015-09-04

Posts: 2

Re: Journalctl --follow and "Input/output error"

I can definitely exclude the timeout idea, because I also tried setting the related ncat option (-i) to a very long value and the error still occurs.
I'll try disabling (or slowing down) log rotation and I'll report, thanks