Arch Linux

kristopher004

Member

Registered: 2014-06-30

Posts: 33

[Solved] rsync progress in status bar

I'm trying to figure out a way to output rsync progress to my status bar. Ideally I would want current file being transferred, total progress, and current speed. I've gotten close to getting total progress/speed, but have a few problems.

rsync's --info=progress2 outputs current speed and total progress, however lines need to be unbuffered. With the help of google I came up with this:

rsync --info=progress2,stats /path/to/source /path/to/target/ | stdbuf -oL tr '\r' '\n' > /tmp/progress.txt

which gives output like this, updated line by line:

         32.77K   0%    0.00kB/s    0:00:00  
        137.23M  10%  415.38MB/s    0:00:00 (xfr#1, to-chk=1/8)
        444.63M  35%  424.00MB/s    0:00:01  
        894.40M  70%  426.68MB/s    0:00:00  
          1.27G 100%  427.62MB/s    0:00:02 (xfr#2, to-chk=0/8)
          1.27G 100%  427.47MB/s    0:00:02 (xfr#2, to-chk=0/8)

sent 1.27G bytes  received 54 bytes  362.78M bytes/sec
total size is 1.27G  speedup is 1.00

I'm using conky+dzen2 for my status bar, I can display the lines as they are updated with conky:

${tail /tmp/progress.txt 1} | \

This is where I'm stuck. The problem is the blank space before the transfer lines, and my bar does not have enough room for all the information. I would be happy with just grabing column 2 and column 3 as they are being output, however awk seems to only print after rsync process is finished, and I can't figure out the correct deliminator for cut. Any help is appreciated, thank you.

Last edited by kristopher004 (2016-01-03 03:19:49)

MisterAnderson

Member

Registered: 2011-09-04

Posts: 285

Re: [Solved] rsync progress in status bar

EDIT: Wow, I think I went a little overboard on this solution. I think your problem was that awk was buffering it's output but I can't tell where you were originally using it, either in your initial rsync command to write to the file or in your call for conky to read the file. I assume it doesn't matter and the file is for nothing more than tailing the final line from. I've written several solutions in the last two code boxes which should (hopefully) solve the problem. The trick seemed to be piping to awk (or cut or whatever) after your initial tr command, but making sure each section of the pipe had "stdbuf -oL" in it to ensure each line was flushed at the end of the pipe to the file, which can be seen in the first set of solutions. In the last set I replaced the initial tr command with awk itself.

------------------------------

So taking the progess lines in your file and saving them to progress.txt I can get the following output

$ tail -n 1 progress.txt | awk '{ print $2"\t"$3 }'
70%	426.68MB/s
$

That gives you column 2 and 3 with a tab between them and nothing in front. Try combining that with your conky output line.

EDIT: Some more examples of awk to cut down and print only the columns and rows you want.

$ cat progress.txt 
         32.77K   0%    0.00kB/s    0:00:00  
        137.23M  10%  415.38MB/s    0:00:00 (xfr#1, to-chk=1/8)
        444.63M  35%  424.00MB/s    0:00:01  
        894.40M  70%  426.68MB/s    0:00:00  
          1.27G 100%  427.62MB/s    0:00:02 (xfr#2, to-chk=0/8)
          1.27G 100%  427.47MB/s    0:00:02 (xfr#2, to-chk=0/8)

sent 1.27G bytes  received 54 bytes  362.78M bytes/sec
total size is 1.27G  speedup is 1.00
$

$ cat progress.txt | awk '{ out=$1; for(i=2;i<=NF;i++) {out=out" "$i}; print out }'
32.77K 0% 0.00kB/s 0:00:00
137.23M 10% 415.38MB/s 0:00:00 (xfr#1, to-chk=1/8)
444.63M 35% 424.00MB/s 0:00:01
894.40M 70% 426.68MB/s 0:00:00
1.27G 100% 427.62MB/s 0:00:02 (xfr#2, to-chk=0/8)
1.27G 100% 427.47MB/s 0:00:02 (xfr#2, to-chk=0/8)

sent 1.27G bytes received 54 bytes 362.78M bytes/sec
total size is 1.27G speedup is 1.00
$

$ cat progress.txt | awk '/%/ { out=$1; for(i=2;i<=4;i++) {out=out" "$i}; print out }'
32.77K 0% 0.00kB/s 0:00:00
137.23M 10% 415.38MB/s 0:00:00
444.63M 35% 424.00MB/s 0:00:01
894.40M 70% 426.68MB/s 0:00:00
1.27G 100% 427.62MB/s 0:00:02
1.27G 100% 427.47MB/s 0:00:02
$

$ cat progress.txt | awk '/%/ { out=$1; for(i=2;i<=4;i++) {out=out"\t"$i}; print out }'
32.77K	0%	0.00kB/s	0:00:00
137.23M	10%	415.38MB/s	0:00:00
444.63M	35%	424.00MB/s	0:00:01
894.40M	70%	426.68MB/s	0:00:00
1.27G	100%	427.62MB/s	0:00:02
1.27G	100%	427.47MB/s	0:00:02
$

$ tail -n 1 progress.txt | awk '/%/ { out=$1; for(i=2;i<=4;i++) {out=out" "$i}; print out }'
$

EDIT EDIT: It's harder to get rid of those leading spaces with cut because it doesn't treat repeats of a field delimiter (in this case a space) as a single delimiter like awk does but instead treats them separately. You could reduce it to one leading space by squeezing repeated spaces together with

tr -s " "

and then use cut with

cut -d " " -f 3-4

to get the 3rd and 4th fields (the first being what is before the first space I guess).

I think your situation would be best solved by one of the following two versions of your initial command

# using tr to squeeze the spaces
rsync --info=progress2,stats /path/to/source /path/to/target/ | stdbuf -oL tr '\r' '\n' | stdbuf -oL tr -s " " > /tmp/progress.txt
# example output (note the single leading space on this option)
 458,588,160 6% 109.38MB/s 0:01:02

# or only columns 2 and 3
rsync --info=progress2,stats /path/to/source /path/to/target/ | stdbuf -oL tr '\r' '\n' | stdbuf -oL tr -s " " | stdbuf -oL cut -d " " -f 3-4 > /tmp/progress.txt
# example output
3% 114.59MB/s

# or with awk instead of tr
rsync --info=progress2,stats /path/to/source /path/to/target/ | stdbuf -oL tr '\r' '\n' | stdbuf -oL awk '{ out=$1; for(i=2;i<=NF;i++) {out=out" "$i}; print out }' > /tmp/progress.txt
# example output
156,860,416 2% 74.78MB/s 0:01:34

# or with awk and only columns 2 and 3, ignoring lines without a percent symbol
rsync --info=progress2,stats /path/to/source /path/to/target/ | stdbuf -oL tr '\r' '\n' | stdbuf -oL awk '/%/ { out=$2; for(i=3;i<=3;i++) {out=out" "$i}; print out }' > /tmp/progress.txt
# simplified
rsync --info=progress2,stats /path/to/source /path/to/target/ | stdbuf -oL tr '\r' '\n' | stdbuf -oL awk '/%/ { print $2" "$3 }' > /tmp/progress.txt
# example output
2% 100.08MB/s

all of which I tested by running "watch -n .1 progress.txt" during the rsync. I don't have conkey installed to test it with, but hopefully watch recreated that scenario. If I didn't use "stdbuf -oL" on each section of the pipe I wouldn't get output into the file until transfer finished, which I assume was your problem in the first place.

EDIT EDIT EDIT: The magic of awk. You can skip all the "tr" commands and do it with a single "awk" command, using "\r" as your record separater.

# everything in line
rsync --info=progress2,stats /path/to/source /path/to/target/ | stdbuf -oL awk 'BEGIN { RS="\r" } { out=$1; for(i=2;i<=NF;i++) {out=out" "$i}; print out }' > /tmp/progress.txt
# example output
156,860,416 2% 74.78MB/s 0:01:34

# columns 2 and 3
rsync --info=progress2,stats /path/to/source /path/to/target/ | stdbuf -oL awk 'BEGIN { RS="\r" } /%/ { print $1" "$2 }' > /tmp/progress.txt
# example output
2% 100.08MB/s

Last edited by MisterAnderson (2015-12-29 17:39:57)

---

D:

kristopher004

Member

Registered: 2014-06-30

Posts: 33

Re: [Solved] rsync progress in status bar

# columns 2 and 3
rsync --info=progress2,stats /path/to/source /path/to/target/ | stdbuf -oL awk 'BEGIN { RS="\r" } /%/ { print $1" "$2 }' > /tmp/progress.txt
# example output
2% 100.08MB/s

Nice! This is exactly what I was trying for, works perfect.

edit: Your right, my problem was that I wasn't reusing stdbuf, and awk would rebuffer the lines, which is why I couldn't get awk to print line by line. This is all good stuff, thank you.

Last edited by kristopher004 (2015-12-29 19:23:49)

MisterAnderson

Member

Registered: 2011-09-04

Posts: 285

Re: [Solved] rsync progress in status bar

No problem. You should edit your first post and add [SOLVED] at the start.

---

D: