Python3: get stdout of a command in real time

swiftycode · 2023-12-21 11:19:28

Hi everyone,

I need to write a Python script that can process/read and print the output (standard output) of a command in real time, line by line. The methods I found online doesn't quite work well, and in most cases the script can't read any standard output unless the invoked command exits/finishes. I have tried using the read line function and disabling line buffering, but it still doesn't work in most cases.

However, I should note that it works in rare circumstances, such as the ping command. I'm guessing that it's related to many programs not flushing the stdout pipe after each line of output.

Can anyone provide a working method? Here's the code I have currently:

import subprocess
import sys

if len(sys.argv)<2:
    print("No command specified")
    exit(1)
print(sys.argv[1:])
def splitarray_to_string(array: list) -> str:
    product=""
    for item in array:
        product+=item+" "
    return product.strip()
process = subprocess.Popen(splitarray_to_string(sys.argv[1:]), shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=0, text=True)

while True:
    line=process.stdout.readline()
    if process.poll() != None:
        break
    if line != '':
        sys.stdout.write(line);sys.stdout.flush()
rc=process.poll()

Thanks!

seth · 2023-12-21 20:35:14

Try to goolge for "python3 fcntl O_NONBLOCK"

swiftycode · 2023-12-22 04:43:51

seth wrote:

Try to goolge for "python3 fcntl O_NONBLOCK"

One method I found online says that I have to set the flag of the stdout pipe to O_NONBLOCK. After inserting

fcntl.fcntl(process.stdout, fcntl.F_SETFL, os.O_NONBLOCK)

into my code before the read loop, I'm getting this error on the read function.

TypeError: underlying read() should have returned a bytes-like object, not 'NoneType'

Handling this error using

except TypeError:
    continue

doesn't fix anything at all (the same effect as before).

seth · 2023-12-22 08:38:27

https://gist.github.com/sebclaeys/1232088
https://medium.com/@denismakogon/python … f92429d18e
https://stackoverflow.com/questions/375 … -in-python
https://ballingt.com/nonblocking-stdin-in-python-3/

swiftycode · 2024-01-23 04:15:39

It would be highly appreciated if anyone can provide a working code implementation that I can directly use; it would save me time from trying to understand and test a bunch of solutions I found online. I would want the program output to behave exactly like terminal output, meaning that the stdout can be read in real time without any waiting or blocking.

Thanks!

P.S. Using asyncio's subprocess functions, which is a solution I found online, doesn't fix the problem for me.

Last edited by swiftycode (2024-01-23 04:24:27)

bulletmark · 2024-01-23 07:48:46

It's not clear which end of the stick you are trying to shake here? That example code runs another program and captures it's output. You say that example code works with ping, but not with other programs. So that must mean your example code works, and your problem is actually those "other" programs. Are they written in Python also, so are you really asking about them? If you are, then just run them with `python -u` instead of `python` (e.g. change the 1st shebang line to add that `-u`). Or change that/those programs to always add flush=True on all the `print` statements, e.g by wrapping `print(msg)` with `def log(msg): print(msg, flush=True)`.

swiftycode · 2024-05-30 23:37:28

Hi everyone,

After doing some research, I was able to find a working solution using Python's pty.openpty function and specifying the file descriptors in subprocess.Popen. The implementation looks something like this:

stdout_fd, stdout_slave=pty.openpty()
stderr_fd, stderr_slave=pty.openpty()
process=subprocess.Popen(command, stdin=stdout_slave, stdout=stdout_slave, stderr=stderr_slave, bufsize=0, close_fds=True, env=env)
prev_attrs=termios.tcgetattr(sys.stdin)
def get_terminal_size(): return fcntl.ioctl(0, termios.TIOCGWINSZ, struct.pack('HHHH',0,0,0,0))
last_terminal_size=struct.pack('HHHH',0,0,0,0) # placeholder
while process.poll()==None:
    # update terminal attributes from what the program sets
    try: 
        attrs=termios.tcgetattr(stdout_fd)
        # disable canonical and echo mode (enable cbreak) no matter what to avoid duplicate output when copying stdin to stdout_fd
        attrs[3] &= ~(termios.ICANON | termios.ECHO)
        termios.tcsetattr(sys.stdin, termios.TCSADRAIN, attrs)
    except termios.error: pass
    # update terminal size
    try:
        new_term_size=get_terminal_size()
        if new_term_size!=last_terminal_size:
            last_terminal_size=new_term_size
            fcntl.ioctl(stdout_fd, termios.TIOCSWINSZ, new_term_size)
            fcntl.ioctl(stderr_fd, termios.TIOCSWINSZ, new_term_size)
            process.send_signal(signal.SIGWINCH)
    except: pass
    fds=select.select([stdout_fd, sys.stdin, stderr_fd], [], [], 0.01)[0]
    if sys.stdin in fds:
		# --Copy stdin content to stdout_fd--
    if stdout_fd in fds:
		# --Read and handle standard output content--
    if stderr_fd in fds:
		# --Read and handle standard error content--
termios.tcsetattr(sys.stdin, termios.TCSADRAIN, prev_attrs) # restore previous attributes

However, when running some programs (especially shells), I usually get an error message saying "No job control in this shell" or "No access to TTY". Running the fish shell fails with:

 
warning: No TTY for interactive shell (tcgetpgrp failed)
setpgid: Inappropriate ioctl for device

I also encountered other bugs and issues with this implementation. For example, the output of the interactive prompts in "gpg --full-gen-key" doesn't go through the created pty and the code can't process them; Suspending in "vim" doesn't fully restore correct terminal attributes/flags; etc.

How can I change my current implementation so these issues are fixed?

Also, I'm still finding for a solution to make sure that the content from stdout_fd and stderr_fd are processed/outputted in the correct order (Reading them like the code above doesn't guarantee the original order of the program output), and I can still distinguish stdout and stderr for each output. It would be highly appreciated if anyone can provide some recommendations about this.

Last edited by swiftycode (2024-05-30 23:37:59)

Trilby · 2024-05-30 23:43:16

I don't believe I can really help, but even if I could, your track record in this thread for responding constructively to suggestions is not very good. That makes it pretty unappealing for others to try to offer help.

seth · 2024-05-31 08:20:29

However, when running some programs (especially shells), I usually get an error message saying "No job control in this shell" or "No access to TTY". Running the fish shell fails with:

You're trying to run an interactive shell w/o stdout.

So let's go back to

I need to write a Python script that can process/read and print the output (standard output) of a command in real time, line by line.

because "need" seems relative, given it's been almost 6 months and this starts to get all the hallmarks of an https://en.wikipedia.org/wiki/XY_problem

What do you *actually* want to achieve here?
Are you trying to write a terminal multiplexer in python??

ewaller · 2024-05-31 15:10:31

I usually resort to a multiple threaded application for this. Start a thread that can be blocked waiting for output. As it progresses, have it send the data to the main thread using a Queue.

DeaD_EyE · 2024-06-20 11:25:26

import sys
from subprocess import PIPE, STDOUT, Popen


if len(sys.argv) < 2:
    print("No command specified")
    exit(1)


# first argument should be a list with str
# shell=True should not be used
process = Popen(sys.argv[1:], stdout=PIPE, stderr=STDOUT, bufsize=0, text=True)


for line in map(str.rstrip, process.stdout):
    print(line.upper())

# iteration stops, if stdout is closed by the process
# then the process itself terminates, but not instantly

process.wait()
print("Return code:", process.returncode)

Arch Linux

#1 2023-12-21 11:19:28

Python3: get stdout of a command in real time

#2 2023-12-21 20:35:14

Re: Python3: get stdout of a command in real time

#3 2023-12-22 04:43:51

Re: Python3: get stdout of a command in real time

#4 2023-12-22 08:38:27

Re: Python3: get stdout of a command in real time

#5 2024-01-23 04:15:39

Re: Python3: get stdout of a command in real time

#6 2024-01-23 07:48:46

Re: Python3: get stdout of a command in real time

#7 2024-05-30 23:37:28

Re: Python3: get stdout of a command in real time

#8 2024-05-30 23:43:16

Re: Python3: get stdout of a command in real time

#9 2024-05-31 08:20:29

Re: Python3: get stdout of a command in real time

#10 2024-05-31 15:10:31

Re: Python3: get stdout of a command in real time

#11 2024-06-20 11:25:26

Re: Python3: get stdout of a command in real time

Board footer