You are not logged in.

#1 2013-03-05 19:29:25

firecat53
Member
From: Lake Stevens, WA, USA
Registered: 2007-05-14
Posts: 1,542
Website

Python, FIFO and monsterwm [SOLVED]

This is somewhat cross-posted from here, but I've included some particulars and am asking from a scripting point-of-view regarding my handling of FIFO's in python.

So, monsterwm provides an output to a FIFO that describes the workspace layout, current workspace, urgent hints, etc, that looks something like this (I have 7 workspaces. For each workspace, 7 digits divided by a ':' current monitor, monitor index, dekstop id,#win on that desktop,tiling mode,current flag,urgent flag):

0:1:0:6:1:0:0 0:1:1:1:1:1:0 0:1:2:0:1:0:0 0:1:3:0:1:0:0 0:1:4:0:1:0:0 0:1:5:0:1:0:0 0:1:6:0:1:0:0

To read this in python, I'm trying something like:

fifo = '/tmp/monsterwm.fifo'
def fifo_test():
    while True:
        with open(fifo, 'r') as fd:
            res = fd.readline().strip()
            print(res)

It works, but for some reason it loses information from time to time (the beginning of the string gets cut off), and I don't know why. For example, when switching between desktop 0 and desktop 1, the output should look like:

0:1:0:1:1:0:0 0:1:1:2:1:1:0 0:1:2:0:1:0:0 0:1:3:0:1:0:0 0:1:4:0:1:0:0 0:1:5:0:1:0:0 0:1:6:0:1:0:0
0:1:0:1:1:1:0 0:1:1:2:1:0:0 0:1:2:0:1:0:0 0:1:3:0:1:0:0 0:1:4:0:1:0:0 0:1:5:0:1:0:0 0:1:6:0:1:0:0

However, it frequently looks like:

0:1:0:1:1:0:0 0:1:1:2:1:1:0 0:1:2:0:1:0:0 0:1:3:0:1:0:0 0:1:4:0:1:0:0 0:1:5:0:1:0:0 0:1:6:0:1:0:0
1:0:0 0:1:2:0:1:0:0 0:1:3:0:1:0:0 0:1:4:0:1:0:0 0:1:5:0:1:0:0 0:1:6:0:1:0:0

You can see the second line is cutoff at the beginning and shifted left.

I've tried various iterations of this:
- Moved with 'while True' statement inside the 'with open'
- Tried using os.open and os.read

But no matter what, there continues to be information lost. This is making it difficult to parse the output to use for a statusbar because the most recent line won't necessarily have the correct information because it's cut off! The bash scripts that c00kiem0n5ter provides here seem to work fine with no glitches when rapidly moving between workspaces or moving windows around...meaning no information is lost.

Am I doing something wrong here? No, I don't _have_ to move the scripts to python...but I'd like to see if it'll work! smile

Thanks!
Scott

Last edited by firecat53 (2013-03-06 16:56:26)

Offline

#2 2013-03-05 19:52:05

kaszak696
Member
Registered: 2009-05-26
Posts: 543

Re: Python, FIFO and monsterwm [SOLVED]

I'm not sure readline() from fifo is a good idea, it's best to grab the entire message into memory with read() and then process it. I had a similar problem with rapid access to fifos, sometimes the messages were incomplete or overlapping, so i implemented a simple locking mechanism to make it a bit more realiable. It shouldn't be needed with single writer though.

Last edited by kaszak696 (2013-03-05 20:16:01)


'What can be asserted without evidence can also be dismissed without evidence.' - Christopher Hitchens
'There's no such thing as addiction, there's only things that you enjoy doing more than life.' - Doug Stanhope
GitHub Junkyard

Offline

#3 2013-03-05 21:47:54

firecat53
Member
From: Lake Stevens, WA, USA
Registered: 2007-05-14
Posts: 1,542
Website

Re: Python, FIFO and monsterwm [SOLVED]

It's a little better, but I'm still getting plenty of missed info:

import os

fifo = '/tmp/monsterwm.fifo'
def fifo_test():
    while True:
        fd = os.open(fifo, os.O_RDWR)
        res = os.read(fd, 4096).splitlines()
        print(res[-1])

Also tried:
- (os.O_RDONLY|os.O_NONBLOCK), but that gives me a BlockingIOError (which I didn't think it should, according to my googling)
- Moving the os.open statement before the while True.

Just a normal desktop switch.

b'0:1:0:1:1:0:0 0:1:1:1:1:0:0 0:1:2:0:1:1:0 0:1:3:0:1:0:0 0:1:4:0:1:0:0 0:1:5:0:1:0:0 0:1:6:0:1:0:0 '
b'1:0 0:1:1:1:1:0:0 0:1:2:0:1:0:0 0:1:3:0:1:0:0 0:1:4:0:1:0:0 0:1:5:0:1:0:0 0:1:6:0:1:0:0 '

Thanks for the input! Any other ideas? This is quite puzzling to me...

Scott

Offline

#4 2013-03-05 22:05:18

kaszak696
Member
Registered: 2009-05-26
Posts: 543

Re: Python, FIFO and monsterwm [SOLVED]

I meant the standard read(), like this:

fifo = '/tmp/monsterwm.fifo'
def fifo_test():
    while True:
        with open(fifo) as fd:
            res = fd.read().strip()
            print(res)

Generally you shouldn't use file interface from os module unless you have specific need for them.


'What can be asserted without evidence can also be dismissed without evidence.' - Christopher Hitchens
'There's no such thing as addiction, there's only things that you enjoy doing more than life.' - Doug Stanhope
GitHub Junkyard

Offline

#5 2013-03-05 22:21:48

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Python, FIFO and monsterwm [SOLVED]

I ran into a related problem when writing fipolate, but in the opposite direction (writing to a pipe using a loop). I think my conclusion was that one end of the pipe could be opened and closed multiple times while the other end is being closed. In my case the loop was able to fill up the buffer with repeated copies of the input text before the other end could flush it. in your case it may be that part of the text is being written to the pipe as you are closing it.

You could try keeping it open and reading a fixed amount of data from it when you need it. Overall the code structure would be something like this:

with open(...) as f:
  while True:
    foo = f.read(x)...

If that doesn't work, take a look at the python-pyinotify package. You should be able to use it to get notifications when the file is opened for writing (but not for reading, which is what I originally tried in fipolate).

Fipolate's inotify-driven fifo write loop might provide a good starting point (or not, I don't honestly remember how I did it right now and I'm too lazy to look).


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#6 2013-03-05 22:23:26

firecat53
Member
From: Lake Stevens, WA, USA
Registered: 2007-05-14
Posts: 1,542
Website

Re: Python, FIFO and monsterwm [SOLVED]

Well....read() doesn't work at all. I'm guessing because of this note in the documentation:

If the end of the file has been reached, f.read() will return an empty string ("").

So nothing is ever returned....doesn't a FIFO always end with an EOF?

A lot of the googling on Python FIFO's that I did showed people using os.open and os.read much more often than not.

Should the open() statement fall inside or outside the 'while' loop? It seems to "work" both ways, but I'm not sure which is more correct.

Another note: just doing 'cat /tmp/monsterwm.fifo' and watching the output shows the same 'cutting off' of information that I see in python! So how the heck is the bash script reading that and getting the information all correct?? I'm so confused!!

Scott

Offline

#7 2013-03-05 22:28:29

kaszak696
Member
Registered: 2009-05-26
Posts: 543

Re: Python, FIFO and monsterwm [SOLVED]

firecat53 wrote:

Well....read() doesn't work at all.

Odd. read() should block until some stuff writes into fifo.


'What can be asserted without evidence can also be dismissed without evidence.' - Christopher Hitchens
'There's no such thing as addiction, there's only things that you enjoy doing more than life.' - Doug Stanhope
GitHub Junkyard

Offline

#8 2013-03-06 01:49:46

firecat53
Member
From: Lake Stevens, WA, USA
Registered: 2007-05-14
Posts: 1,542
Website

Re: Python, FIFO and monsterwm [SOLVED]

Xyne wrote:

You could try keeping it open and reading a fixed amount of data from it when you need it. Overall the code structure would be something like this:

with open(...) as f:
  while True:
    foo = f.read(x)...

This is what I did in the first place....it doesn't seem to matter where I place the 'with open'...either before or after the 'while True'. And it doesn't seem to matter if I use read(), read(x), or os.read(f, x). There's still missing information.

I can monitor the actual monsterwm output because the startwm.sh script uses 'tee'...so if I just switch desktops, this is what I see on stdout:

0:1:0:1:1:0:0 0:1:1:0:1:1:0 0:1:2:0:1:0:0 0:1:3:0:1:0:0 0:1:4:0:1:0:0 0:1:5:0:1:0:0 0:1:6:0:1:0:0 
0:1:0:1:1:1:0 0:1:1:0:1:0:0 0:1:2:0:1:0:0 0:1:3:0:1:0:0 0:1:4:0:1:0:0 0:1:5:0:1:0:0 0:1:6:0:1:0:0 

At the same time, if I'm running 'cat /tmp/monsterwm.fifo', or using any of the variants of the python scripts to read the fifo, this is what I [may] see (it changes...sometimes all info there, sometimes not):

0:1:0:1:1:0:0 0:1:1:0:1:1:0 0:1:2:0:1:0:0 0:1:3:0:1:0:0 0:1:4:0:1:0:0 0:1:5:0:1:0:0 0:1:6:0:1:0:0 
:0:1:1:1:0 0:1:1:0:1:0:0 0:1:2:0:1:0:0 0:1:3:0:1:0:0 0:1:4:0:1:0:0 0:1:5:0:1:0:0 0:1:6:0:1:0:0 

Why, why, why is stdout  != fifo output?? Except, it seems, when using c00kiem0n5ter's bash scripts I linked in the first post.

Xyne wrote:

If that doesn't work, take a look at the python-pyinotify package. You should be able to use it to get notifications when the file is opened for writing (but not for reading, which is what I originally tried in fipolate).

But does the fifo get opened each time monsterwm sends new info to it, or does it just stay open? Monsterwm uses printf and fflush to send the output to stdout each time a change is registered, and the startwm script looks something like :

......
while :; do  monsterwm || break; done | tee -a '/tmp/monsterwm.fifo'

Also, would pyinotify be able to react fast enough for a real-time status bar with changing desktop and window information?

To summarize, I think my biggest confusion is why the fifo appears to be missing some information when I look at it with 'cat' or a python script.

Thanks!
Scott

Last edited by firecat53 (2013-03-06 01:51:54)

Offline

#9 2013-03-06 07:21:38

kaszak696
Member
Registered: 2009-05-26
Posts: 543

Re: Python, FIFO and monsterwm [SOLVED]

Hmm, how do you pass the data to fifo? I looked into monsterwm code and it doesn't do any file operations. If all you need is to read it's stdout, then you don't have to use fifos altogether, you can just connect stdout of monsterwm to stdin of your script by a normal bash pipe.

Inotify reacts instantly to any events, that's what it was designed in mind.

I examined the 'while ;: do' line with the help of pytinotify and i noticed that tee never closes the file until the program exits. It might not be a problem, but the way i understand fifos to work is:

  1. read() on fifo, block until something opens fifo for writing.

  2. open() for writing, write() some stuff.

  3. writer close() file, at this moment data arrives to read().

  4. read() end closes the file as well

Basically, closing the fifo is where the magic happens and prevent  mishmashes. Another open()+write() will block until the reading end is ready to feast on some more data. tee probably doesn't do that, and writes to fifo while the script is still processing the last message, so some data ends up in oblivion.

This is all my speculation though, i might be talking out of my arse.

Last edited by kaszak696 (2013-03-06 08:01:32)


'What can be asserted without evidence can also be dismissed without evidence.' - Christopher Hitchens
'There's no such thing as addiction, there's only things that you enjoy doing more than life.' - Doug Stanhope
GitHub Junkyard

Offline

#10 2013-03-06 12:04:36

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Python, FIFO and monsterwm [SOLVED]

Is the fifo meant to be read by multiple applications concurrently? If yes then it must be opened and closed multiple times on both ends and you cannot continuously read from it. If no then you should be able to continuously read from it, but only in one application.

If you are concurrently reading from the fifo inside a loop in two different applications then you are probably triggering a race condition with locking and buffering.

If you control both readers then the simplest solution may be to simply read fixed amount of data from the pipe in each loop and write it to a ramdisk file (e.g. /dev/shm/monsterwm.info). It may even be better to write it to a different file (e.g. /dev/shm/monsterwm.info.new) and move it into place (e.g. with shutils.move) to avoid other read/write race conditions.

You could also try lock files if you control all readers, but I prefer avoiding disk I/O whenever it is easy.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#11 2013-03-06 14:06:54

progandy
Member
Registered: 2012-05-17
Posts: 5,184

Re: Python, FIFO and monsterwm [SOLVED]

Why don't you do everything in one Python script?
Use subprocess.Popen to run monsterwm and pipes instead of a fifo


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

#12 2013-03-06 16:55:50

firecat53
Member
From: Lake Stevens, WA, USA
Registered: 2007-05-14
Posts: 1,542
Website

Re: Python, FIFO and monsterwm [SOLVED]

Thanks so much for the suggestions!! I may not understand the FIFO issue...but just using Popen and PIPE finally worked out.
Something like this:

p = Popen(["monsterwm"], stdout=PIPE)
bar = Popen(["bar"], stdin=PIPE)
res = p.stdout.readline()
while res:
    <bar code>......
    bar.stdin.write("{}{}\n".format(r, tm).encode())
    res = p.stdout.readline()

Now to integrate the rest of the statusbar smile

Thanks!!
Scott

Offline

#13 2013-03-06 17:20:43

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Python, FIFO and monsterwm [SOLVED]

In case you aren't aware of it, you can use "with Popen(...) as p" the same way you can use "with open(...) as f".


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#14 2013-03-06 18:09:19

firecat53
Member
From: Lake Stevens, WA, USA
Registered: 2007-05-14
Posts: 1,542
Website

Re: Python, FIFO and monsterwm [SOLVED]

Ah, ok. That just gets you a cleaner shutdown/cleanup, right? Could there be any advantage to that, in this particular case?

Scott

Offline

#15 2013-03-06 19:45:54

kaszak696
Member
Registered: 2009-05-26
Posts: 543

Re: Python, FIFO and monsterwm [SOLVED]

with statement is exception-proof, it always closes the resource cleanly regardless of any thrown exception.


'What can be asserted without evidence can also be dismissed without evidence.' - Christopher Hitchens
'There's no such thing as addiction, there's only things that you enjoy doing more than life.' - Doug Stanhope
GitHub Junkyard

Offline

Board footer

Powered by FluxBB