cat /dev/ttyUSB0 and awk redirection problem

seal20 · 2012-04-26 02:06:33

Hi awk and bash scripts gurus,

I am listening to a multi-sensor over RS232 plug through a usb adaptater. It apppears on ttyUSB0. The output is plain text, here is a sample line:

[sylvain@sylvain-pc ETSA]$ cat /dev/ttyUSB0 
516,R02749,2.72,1,26-04-2012 09:48:30,6.750,pH,25.3,mV,24.9,C,99.2,%,4,001715,6.65,mg/L,99.9,% sat,139.4,Torr,24.9,C,747.0,mmHg,35.0,ppt,1.4,Torr/%sat,10,1609
516,R02749,2.72,1,26-04-2012 09:48:40,6.750,pH,25.3,mV,24.9,C,99.2,%,4,001715,6.65,mg/L,100.0,% sat,139.5,Torr,24.9,C,747.0,mmHg,35.0,ppt,1.4,Torr/%sat,10,1610

I can redirect the output like this:

[sylvain@sylvain-pc ETSA]$ cat /dev/ttyUSB0 > test.csv

But I want to clean up the output before so I want to use awk. For the purpose of test I will use only a simple command:

[sylvain@sylvain-pc ETSA]$ cat /dev/ttyUSB0 | awk 'BEGIN{FS = ","} ; {print $5 "," $16}'

This is working but if I redirect it, it doesn't I tried to put the redirect at the end or in the awk command (as written in the manual)

[sylvain@sylvain-pc ETSA]$ cat /dev/ttyUSB0 | awk 'BEGIN{FS = ","} ; {print $5 "," $16 > "test.csv"}'
^C
[sylvain@sylvain-pc ETSA]$ cat /dev/ttyUSB0 | awk 'BEGIN{FS = ","} ; {print $5 "," $16}' > test.csv
^C

From the manual the device communicate over rs232 with this paprameter:
Bits per second: 9600
Data bits: 8
Parity: None
Stop bits: 1
Flow control: Hardware

and my stty parameters are:

sylvain@sylvain-pc ETSA]$ stty -F /dev/ttyUSB0 -a
speed 9600 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>; eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W; lnext = ^V; flush = ^O;
min = 1; time = 0;
-parenb -parodd cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon -ixoff -iuclc -ixany -imaxbel -iutf8
opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig icanon iexten echo echoe echok -echonl -noflsh xcase -tostop -echoprt echoctl echoke

In the end I would like to make some script like this to be able to log and visualize the data directly. Now I am using kst, a very good soft for this, but simple is best. At the moment the script is not working... There may be many problems in it but it is a good way to learn and I cannot really start as long as I cannot redirect awk output...

#!/bin/sh
data=$1

# set the ttyUSB for fail proof logging:
stty -F /dev/ttyUSB0 9600  -parenb -parodd cs8 hupcl -cstopb cread clocal -crtscts -ignbrk\
 -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon -ixoff -iuclc -ixany -imaxbel\
 -iutf8 opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0 isig icanon\
 iexten echo echoe echok -echonl -noflsh xcase -tostop -echoprt echoctl echoke

#start logging with $1 passed from command line and format the fields for later use
cat /dev/ttyUSB0 |awk '
BEGIN {
FS = ","    # input field seperator 
OFS = ","   # output field seperator
print "time DO_mg/l"; 
}
NR > 1 {print $5 "," $16}' > $data &


#convert the data file temprary for vizualisation with graph
awk 'BEGIN{FS = ","}; FNR>1{if (length($0) > 1)   print  $2 }'  $data | awk '{print NR " " $1}' | graph -T X

Thanks

Sylvain

PS: if I run the awk command on a file instead of /dev/ttyUSB0 and redirect to another file, it works ! ??

Trilby · 2012-04-26 02:27:44

I'm not sure why you are not getting output when run on the tty, but I'd recommend simplifying at two points. You shove a cat into an awk, then later an awk through a pipe into another awk. One awk command should do for each of these. This would avoid what I call Turducken scripting.

Often simplifying, in itself, will solve some problems. If not, it helps in narrowing down where the issue is.

Last edited by Trilby (2012-04-26 02:28:19)

seal20 · 2012-04-26 02:35:48

Ok simple but still not working:

[sylvain@sylvain-pc ETSA]$ awk 'BEGIN{FS = ","} ; {print $5 "," $16}' /dev/ttyUSB0
26-04-2012 11:23:22,6.74

It is outputing good. But

[sylvain@sylvain-pc ETSA]$ awk 'BEGIN{FS = ","} ; {print $5 "," $16}' /dev/ttyUSB0 > test.csv
^C
[sylvain@sylvain-pc ETSA]$ cat test.csv 
[sylvain@sylvain-pc ETSA]$

For the final script I will try to simplify the "double" awk.. but I need first to be able to redirect the output of awk to a file...

Trilby · 2012-04-26 02:42:33

Why did you have to ^C break the second one? In the first example, does it print that line and terminate, or is it still waiting on more input from /dev/ttyUSB0?

If it's the latter I think we've found the problem: the csv file is not created until the process is complete.

seal20 · 2012-04-26 02:46:52

it is still waiting for more input from /dev/ttyUSB0. So is there a way to write the output from awk "in real time"? like with cat /dev/ttyUSB0 > test.csv

Last edited by seal20 (2012-04-26 02:47:35)

falconindy · 2012-04-26 02:47:29

You're likely running into problems because buffering. You can add a call to fflush() after the print statement and see if that helps.

Related, but may not help -- if your only goal is to extract specific columns, awk is no better at this than cut is. However, it may require changing the underlying stream to use unbuffered stdin.

$ stdbuf -i0 cut -d, -f5,16 /dev/ttyUSB0 > out.csv

Last edited by falconindy (2012-04-26 02:48:45)

seal20 · 2012-04-26 02:55:28

Thanks falconindy your stdbuf command is working!
The fflush is not working but I have to learn how to use it... In the end I want to translate the time in min, for the moment I am just logging every 1min so 1 line = 1 min so I just count the line. That's why I was using awk.

seal20 · 2012-04-26 02:57:45

Ok fflush is working too! with

awk 'BEGIN{FS = ","} ; {print $5 "," $16 ; fflush()}' /dev/ttyUSB0 > test.csv

I will work on the bash script now.

Thanks both of you

Arch Linux

#1 2012-04-26 02:06:33

cat /dev/ttyUSB0 and awk redirection problem

#2 2012-04-26 02:27:44

Re: cat /dev/ttyUSB0 and awk redirection problem

#3 2012-04-26 02:35:48

Re: cat /dev/ttyUSB0 and awk redirection problem

#4 2012-04-26 02:42:33

Re: cat /dev/ttyUSB0 and awk redirection problem

#5 2012-04-26 02:46:52

Re: cat /dev/ttyUSB0 and awk redirection problem

#6 2012-04-26 02:47:29

Re: cat /dev/ttyUSB0 and awk redirection problem

#7 2012-04-26 02:55:28

Re: cat /dev/ttyUSB0 and awk redirection problem

#8 2012-04-26 02:57:45

Re: cat /dev/ttyUSB0 and awk redirection problem

Board footer