Arch Linux

masterofamn

Member

Registered: 2012-05-09

Posts: 18

[solved] python 2.x => python 3.5 stdout.readline() problems

Hi all,

I have a problem with migrating a python program to work with python 3.5.1

What I am trying to do is to send (string) input and parse the output of a subprocess. What has been working in 2.x is the following:

engine = subprocess.Popen(
'stockfish',
universal_newlines=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,)

engine.stdin.write('uci\n')
engine.stdin.write('isready\n')

while True :
	text = engine.stdout.readline().strip()
	print(text)
	if text == 'readyok':
	os.system('clear')
	print('All ready...')
	break

So in essence, I send a string 'uci' and 'isready' to the subprocess, and after several lines of output, at some point or the other it will answer with a line "readyok" which I try to catch.

Using the same code in 3.5 does not provide all lines of stdout, but only the first. Still, the "while True" loop does not hang but is actually interrupted almost immedeatly and the program continues, but obviously not working as intended because I miss the output.

Me being a python noob, using it for the first time for a small DIY project, I tried to find out about the differences between 2.x and 3.5. In essence, I understand from the python manual that I am supposed to use communicate() instead of stdout.readline.

When I do this like in

text,err = engine.comunicate()

I get all the text from the subprocess, but the subprocess is terminated after that - which I do not want. 
I also read on the web that the buffering-behaviour was somehow changed (?), but I do not really understand what to make out of it.

Can anyone give me a hint on what I am doing wrong.

Thanks in advance!

Last edited by masterofamn (2016-01-13 19:55:16)

TheSaint

Member

From: my computer

Registered: 2007-08-19

Posts: 1,523

Re: [solved] python 2.x => python 3.5 stdout.readline() problems

I think you need to reduce the buffer in order to have a prompt response.
See these details

import subprocess as SP
import logging
import re
import slex

REOUT = re.compile('??'your searching')
LOGGER = logging.getLogger(__name__)

     def run(arguments)
		MAXBUF = 120
		status = False
		dec = sys.getfilesystemencoding()
		if isinstance(arguments, str):
			CMD = shlex.split(arguments)
		with SP.Popen(CMD,
						bufsize=MAXBUF,
						stdin=SP.DEVNULL,
						stdout=SP.DEVNULL,
						stderr=SP.PIPE) as p:
			while p.poll() is None:
				try:
					p.stderr.flush()
					p.wait(2)
				except SP.TimeoutExpired:
					if p.poll is None:
						break
					msg = char = ''
					while len(msg) <= MAXBUF:
						char = p.stderr.read(1).decode(dec, errors='ignore')
						if char == '\r':
							break
						msg += char
					p.stderr.flush()
					m = re.search(msg)
					if m:
                                               ####### here put your filtering #######################
				except (KeyboardInterrupt, SP.TimeoutExpired):
					LOGGER.debug('Interrupted by the user')
					p.kill()
					status = 255
					break
		status = (status or p.returncode)
		if status:
			LOGGER.debug('Returned %s status during process' %status)
		return status

Here's an example NOT working  to be studied. It works within my application

---

do it good first, it will be faster than do it twice the saint

masterofamn

Member

Registered: 2012-05-09

Posts: 18

Re: [solved] python 2.x => python 3.5 stdout.readline() problems

Thanks for the reply, Saint.

While playing around with the bufsize as you suggested I realized the embarrassing n00b-typo I made. I did not indent the "break" of the if statement properly, so I was breaking the while True loop after the first cycle.

Problem is now solved, sorry for the noise!