Arch Linux

ronjouch

Member

From: Montreal, QC

Registered: 2014-09-18

Posts: 32

Website

[Solved] Python file.readline() crashes Python when ran by inotify

Hi.

I have a python static site generator that generates html from markdown.
To automate site re-generation on source markdown changes, I have inotify/incrond rules that trigger the script when the source md changes.

- Script works perfectly when ran from the command-line...

- ... but when ran by inotify, I see the script being run, but it stops on any readline() or readlines() call

The culprit code (plus debug statements to know what happens when ran "invisibly" by inotify) looks like this:

dbg = open('/tmp/dbgreadline.txt', 'w')
with open(path, "r") as f:
    dbg.write('hello before readline\n')  # <- This debug statement works
    title = f.readline()                  # <- when ran by inotify: boom
    dbg.write('hello after readline\n')   # <- when ran by inotify: I never see this debug statement
dbg.close()

The file exists, has many lines, and everything works as expected (including readline()) when running the script from a terminal. But when ran by incrond, I never get to see the "hello after readline" write, which means the readline() statement seems to crash the Python program. I tried to understand what's going on with a try/except (and attempt to dbg.write() the potential exception), but it didn't even reach the except clause! Python seems to insta-die on the readline() ! (segfault? something else? I don't know, since it's run "silently" by inotify and journalctl doesn't say anything)

Problem started today Sunday, everything was fine on Friday. I tried downgrading all packages to their state of Thursday, without success. My code is versioned and I didn't make any change during the last few days. Finally, I don't remember having done any related configuration change that might be causing this barf. Ideas?

This is on an up-to-date Arch x64 with Python 3.6.2 and Linux 4.12.6.

Thanks for your help.

EDIT: TODO following discussion with colleagues: try in pypy, re-try the try/except with a finally clause, log PYTHONPATH and LD_LIBRARY_PATH, try inotify tracing, test in an Arch VM

EDIT: Turns out my first attempt at printing the error in a try/except was borked, due to a ridiculous syntax error. After fixing I *do* go through the except, which lets me print a glorious UnicodeDecodeError, fixed by specifying `encoding='utf-8'` to my `open()` call . Now, as to *why* this thing worked a few days ago with the same code and markdown content, I don't even.

Last edited by ronjouch (2017-08-15 02:11:13)