Problems with piping in bash.

JaQoB · 2008-09-12 11:45:12

Hello,

I having an "interesting" problem.

A program (which generating alot of output to stdout) is started from a bash script.
If I let the program print to the console, everything looks fine, but if i pipe the output to a file (Same result with >/tee/logsave) some lines appears twice in the file (in fact many lines appears twice).

Anyone seen this behaviour before? Anyone have a solution?

Profjim · 2008-09-12 15:37:29

How are you "piping to a file"? Really you can't pipe to a file, you can either pipe to another command (which then gets the piped data as its stdin), or you can "redirect" to a file. Perhaps you're clear on what you meant, but since you're having problems, it's worth verifying that you're entering the right commands. Your example of using "tee" for instance, is wrong.

I'm assuming that what you what to do is to get the output from verbose_command into /path/to/file. Either of these should do it:

verbose_command > /path/to/file
verbose_command | tee /path/to/file

JaQoB · 2008-09-14 15:35:18

Profjim wrote:

How are you "piping to a file"? Really you can't pipe to a file, you can either pipe to another command (which then gets the piped data as its stdin), or you can "redirect" to a file. Perhaps you're clear on what you meant, but since you're having problems, it's worth verifying that you're entering the right commands. Your example of using "tee" for instance, is wrong.
I'm assuming that what you what to do is to get the output from verbose_command into /path/to/file. Either of these should do it:
verbose_command > /path/to/file
verbose_command | tee /path/to/file

I was meaning redirecting to file, but since I have figured out that the same problems appears even when you pipe (verbose_command | tee /path/to/file or even verbose_command | grep something).

Xyne · 2008-09-14 16:35:00

How are you starting the program in the bash script?
Is the bash script using "echo" to display output from the program or is the program printing the output directly to the console?
Exactly what commands are you using to pipe|redirect output?
Have you written the program yourself?
If you answered "yes" to the previous question: If only some lines appear twice, which ones and what's different about how they're printed|echoed?
What's your full name, address, social security number, bank account number, 3 previous addresses, current occupation and place of employment, and shoe size?*

I ask about "echo" in particular because I've found duplicates when playing around with echo & pipes & perl.

*(for legal purposes) Do not answer any part of this question.

Last edited by Xyne (2008-09-14 16:41:02)

Dusty · 2008-09-14 16:43:49

Xyne wrote:

What's your full name, address, social security number, bank account number, 3 previous addresses, current occupation and place of employment, and shoe size?*

lol!

moljac024 · 2008-09-14 16:48:58

Dusty wrote:

Xyne wrote:
What's your full name, address, social security number, bank account number, 3 previous addresses, current occupation and place of employment, and shoe size?*
lol!

The "*" sign means that you must give us your shoe size. Otherwise, we can't help you.

JaQoB · 2008-09-14 21:16:52

Xyne wrote:

How are you starting the program in the bash script?
Is the bash script using "echo" to display output from the program or is the program printing the output directly to the console?
Exactly what commands are you using to pipe|redirect output?
Have you written the program yourself?
If you answered "yes" to the previous question: If only some lines appear twice, which ones and what's different about how they're printed|echoed?
What's your full name, address, social security number, bank account number, 3 previous addresses, current occupation and place of employment, and shoe size?*
I ask about "echo" in particular because I've found duplicates when playing around with echo & pipes & perl.

*(for legal purposes) Do not answer any part of this question.

The program is started from the bashscript, via a makefile.
I haven't written the program myself, but i have access to the source. The program is using an ordinary printf(""), and i cant see any differences between those who get printed once and those who get printed multiple.

Going to digg deeper tomorrow, and getting back here with my findings.

Excuse my english!

Profjim · 2008-09-15 03:38:51

Throw in an extra printf or echo of your own making, and see whether that gives you one output or two.

JaQoB · 2008-09-16 08:42:03

I figured it out now.

Here is how to reproduce the problem:

prompt:~/temp$ cat forktest.c
#include <stdio.h>
#include <errno.h>
#include <unistd.h>
#include <stdlib.h>

int main( int argc, const char* argv[] )
{
int i;
printf("Test1\n");
pid_t child_pid;

for (i=0; i<10; i++ )
{
child_pid = fork();

if (child_pid == 0)
{
printf ("Test2\n");
exit(0);
}
else if (child_pid == -1)
{
exit(1);
}
}
while (wait(0) != -1);
}

prompt:~/temp$ gcc forktest.c -o forktest
prompt:~/temp$ ./forktest
Test1
Test2
Test2
Test2
Test2
Test2
Test2
Test2
Test2
Test2
Test2
prompt:~/temp$ ./forktest > test.txt
prompt:~/temp$ cat test.txt
Test1
Test2
Test1
Test2
Test1
Test2
Test1
Test2
Test1
Test2
Test1
Test2
Test1
Test2
Test1
Test2
Test1
Test2
Test1
Test2
Test1

The problem is related to flushing of buffers. Printing to console flushes after every line break, but redirecting/piping flushes after EOF.
A fflush(stdout) right before the fork() solves the problem.

Profjim · 2008-09-16 13:01:00

Thanks for posting the solution. Glad you figured it out.

Arch Linux

#1 2008-09-12 11:45:12

Problems with piping in bash.

#2 2008-09-12 15:37:29

Re: Problems with piping in bash.

#3 2008-09-14 15:35:18

Re: Problems with piping in bash.

#4 2008-09-14 16:35:00

Re: Problems with piping in bash.

#5 2008-09-14 16:43:49

Re: Problems with piping in bash.

#6 2008-09-14 16:48:58

Re: Problems with piping in bash.

#7 2008-09-14 21:16:52

Re: Problems with piping in bash.

#8 2008-09-15 03:38:51

Re: Problems with piping in bash.

#9 2008-09-16 08:42:03

Re: Problems with piping in bash.

#10 2008-09-16 13:01:00

Re: Problems with piping in bash.

Board footer