Python3: get stdout of a command in real time

swiftycode · 2023-12-21 11:19:28

Hi everyone,

I need to write a Python script that can process/read and print the output (standard output) of a command in real time, line by line. The methods I found online doesn't quite work well, and in most cases the script can't read any standard output unless the invoked command exits/finishes. I have tried using the read line function and disabling line buffering, but it still doesn't work in most cases.

However, I should note that it works in rare circumstances, such as the ping command. I'm guessing that it's related to many programs not flushing the stdout pipe after each line of output.

Can anyone provide a working method? Here's the code I have currently:

import subprocess
import sys

if len(sys.argv)<2:
    print("No command specified")
    exit(1)
print(sys.argv[1:])
def splitarray_to_string(array: list) -> str:
    product=""
    for item in array:
        product+=item+" "
    return product.strip()
process = subprocess.Popen(splitarray_to_string(sys.argv[1:]), shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=0, text=True)

while True:
    line=process.stdout.readline()
    if process.poll() != None:
        break
    if line != '':
        sys.stdout.write(line);sys.stdout.flush()
rc=process.poll()

Thanks!

seth · 2023-12-21 20:35:14

Try to goolge for "python3 fcntl O_NONBLOCK"

swiftycode · 2023-12-22 04:43:51

seth wrote:

Try to goolge for "python3 fcntl O_NONBLOCK"

One method I found online says that I have to set the flag of the stdout pipe to O_NONBLOCK. After inserting

fcntl.fcntl(process.stdout, fcntl.F_SETFL, os.O_NONBLOCK)

into my code before the read loop, I'm getting this error on the read function.

TypeError: underlying read() should have returned a bytes-like object, not 'NoneType'

Handling this error using

except TypeError:
    continue

doesn't fix anything at all (the same effect as before).

seth · 2023-12-22 08:38:27

https://gist.github.com/sebclaeys/1232088
https://medium.com/@denismakogon/python … f92429d18e
https://stackoverflow.com/questions/375 … -in-python
https://ballingt.com/nonblocking-stdin-in-python-3/

swiftycode · 2024-01-23 04:15:39

It would be highly appreciated if anyone can provide a working code implementation that I can directly use; it would save me time from trying to understand and test a bunch of solutions I found online. I would want the program output to behave exactly like terminal output, meaning that the stdout can be read in real time without any waiting or blocking.

Thanks!

P.S. Using asyncio's subprocess functions, which is a solution I found online, doesn't fix the problem for me.

Last edited by swiftycode (2024-01-23 04:24:27)

bulletmark · 2024-01-23 07:48:46

It's not clear which end of the stick you are trying to shake here? That example code runs another program and captures it's output. You say that example code works with ping, but not with other programs. So that must mean your example code works, and your problem is actually those "other" programs. Are they written in Python also, so are you really asking about them? If you are, then just run them with `python -u` instead of `python` (e.g. change the 1st shebang line to add that `-u`). Or change that/those programs to always add flush=True on all the `print` statements, e.g by wrapping `print(msg)` with `def log(msg): print(msg, flush=True)`.

Arch Linux

#1 2023-12-21 11:19:28

Python3: get stdout of a command in real time

#2 2023-12-21 20:35:14

Re: Python3: get stdout of a command in real time

#3 2023-12-22 04:43:51

Re: Python3: get stdout of a command in real time

#4 2023-12-22 08:38:27

Re: Python3: get stdout of a command in real time

#5 2024-01-23 04:15:39

Re: Python3: get stdout of a command in real time

#6 2024-01-23 07:48:46

Re: Python3: get stdout of a command in real time

Board footer