Arch Linux

ua4000

Member

Registered: 2015-10-14

Posts: 559

rsync --info=progress2 is flooding the logfile

I'm using rsync as backup program in some self written shell scripts:

main() {
  ...
  rsync --info=progress2  ...
  ...
}

main 2>&1 | tee -a "${logfile}"

The tee -a is important for me, to have a logfile  for later reference.

With this setup I have a nice total progress (via progress2) during runtime in terminal, but on the other hand this is spamming my logfile with 14k lines:

          ...
          7,32G  96% 6814,27GB/s    0:00:00 (xfr#195785, to-chk=9/234221) 
          7,32G  96% 6814,27GB/s    0:00:00 (xfr#195786, to-chk=2/234221)
          7,32G  96% 6814,29GB/s    0:00:00 (xfr#195792, to-chk=1/234221)
          7,32G  96% 6814,31GB/s    0:00:00 (xfr#195793, to-chk=0/234221)
          7,32G  96% 6814,31GB/s    0:00:00 (xfr#195793, to-chk=0/234221)
          7,32G  96% 6814,31GB/s    0:00:00 (xfr#195793, to-chk=0/234221)

The root cause is clear, in terminal progress2 is updated to the same screen line, in the logfile it goes to a new line.

Any ideas to have a nice overall progress and a pretty logfile ?

dogknowsnx

Guest

Re: rsync --info=progress2 is flooding the logfile

Not sure if there's anything else worth logging in your main function - but have you considered something like

rsync --info=progress2 --log-file=$XDG_RUNTIME_DIR/rsync.log ...
cat $XDG_RUNTIME_DIR/rsync.log >> ${logfile}

instead?

Last edited by dogknowsnx (2022-10-30 15:01:57)

ua4000

Member

Registered: 2015-10-14

Posts: 559

Re: rsync --info=progress2 is flooding the logfile

Thanks, very good hint, this works and the logfile also includes rsync errors - which I feared they could be missing.

But, yes, my main() also has some further calls, e.g "date", "mount", "df -h" before and after... I would like to have them all in the logfile.

Could only the rsync line be excluded from the global stdout/error redirection ?

dogknowsnx

Guest

Re: rsync --info=progress2 is flooding the logfile

You can always use

>/dev/null 2>&1

but than you wouldn't see any output on your terminal, either.

How about splitting your main() into main_pre_rsync() and main_post_rsync()?

Last edited by dogknowsnx (2022-10-30 18:05:36)

cfr

Member

From: Cymru

Registered: 2011-11-27

Posts: 7,178

Re: rsync --info=progress2 is flooding the logfile

If you use rsync's logging capacity, you can specify the output on the terminal and the output to the log file independently using --out-format=<format> and --log-file-format=<format> respectively. See man rsync for details.

Last edited by cfr (2022-10-30 22:52:09)

---

CLI Paste | How To Ask Questions

Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L

ua4000

Member

Registered: 2015-10-14

Posts: 559

Re: rsync --info=progress2 is flooding the logfile

Thanks very much for all tips!
I will do some manpage reading and testing.