[solved] Questions about pipes and awk/sed

jjacky · 2012-04-05 18:24:18

Hi,

So I was trying to do a little script, piping a few commands together, and I found something I can't quite understand.

Basically, I want to pipe the output of a program to awk (or sed) to "filter" it a bit, then pipe it to another program. But I'd like for this to all be "on the fly", that is as soon as the first program sends something to its output, it gets treated by awk (sed) and right away gets send to the other program.

IOW, this:

(echo foo && sleep 2 && echo bar) | cat | less

And that works as expected, in less I see foo, then after a while bar gets added. Good.

Then, I replace cat with awk, as follow:

(echo foo && sleep 2 && echo bar) | awk '1' | less

And I expected the exact same result, but instead less starts, I get to wait, and then all the output appears at once.

This is obviously a problem for me, plus I don't understand why it happens. It works fine piping the echo-s to less, or awk, alone.

In fact,

(echo foo && sleep 2 && echo bar) | cat | cat | awk '1'

works as I want, that is I see "foo", then after a while "bar". So clearly piping multiple process isn't an issue. But then, this:

(echo foo && sleep 2 && echo bar) | cat | cat | awk '1' | cat

pauses without showing anything, and then all the output is processed at once.
It seems like awk (or sed) are able to print their output as soon as it's processed, only when not piped. Is that the case? And if so, why? (And ideally, how can I get rid of it, and work the same when piped?)

Thanks,
-jjacky

Last edited by jjacky (2012-04-05 20:28:13)

portix · 2012-04-05 20:01:27

If the output of awk or sed is sent to another pipe both programs buffer the output. With both programs it is possible to flush the output immediately, with sed using the -u command line option and with awk calling fflush:

(echo foo && sleep 2 && echo bar) | awk '1;fflush()' | less

and

(echo foo && sleep 2 && echo bar) | sed -u  '' | less

jjacky · 2012-04-05 20:27:32

Alright, thanks a lot!

I'm sure I searched for "flush" in awk's man page (and I can't find anything about fflush in there. Oh, because it's a GNU extension, and it's in gawk's man page, which I hadn't looked at...), but I obviously completely overlooked the -u in sed's. Anyhow, thanks.

Arch Linux

#1 2012-04-05 18:24:18

[solved] Questions about pipes and awk/sed

#2 2012-04-05 20:01:27

Re: [solved] Questions about pipes and awk/sed

#3 2012-04-05 20:27:32

Re: [solved] Questions about pipes and awk/sed

Board footer