You are not logged in.

#1 2022-07-21 12:36:10

d_fajardo
Member
Registered: 2017-07-28
Posts: 1,563

[Solved] Is there a way of speeding up rsync?

I have written a simple rsync backup script. I find the backup very slow:

 #!/bin/su root

# check if drive is mounted
DIR=/run/media/brightchip/plug

if grep -qs $DIR /proc/mounts; then
    echo "Directory $DIR mounted. Continuing..."
    echo "****** Backing up system ******"
    rsync -aAXogv --exclude={"/home/datapool","/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} --delete / $DIR
    echo "****** System backup completed ******"
    exit 0
else  
    echo "^^^^ Error: $DIR is not mounted. Exiting ^^^^"
    exit 7
fi

Is there a way of speeding up rsync?

Last edited by d_fajardo (2022-07-27 11:12:24)

Offline

#2 2022-07-21 12:47:49

Awebb
Member
Registered: 2010-05-06
Posts: 6,275

Re: [Solved] Is there a way of speeding up rsync?

Is the rsync command only slow in thwt script or also if run in shell?

Have you built that -aA... line yourself or is it copypasta. If copypasta, decypher that for yourself first.

Offline

#3 2022-07-21 13:20:06

d_fajardo
Member
Registered: 2017-07-28
Posts: 1,563

Re: [Solved] Is there a way of speeding up rsync?

If copypasta, decypher that for yourself first.

It is copy paste and I most probably changed it too but I see what you mean. The -a flag shouldn't be there.
I noticed most of the slowness happens backing up home dot directories. That's probably where it conflicts.
Will report if it improves.

EDIT: I've only run it from shell.

Last edited by d_fajardo (2022-07-21 13:23:03)

Offline

#4 2022-07-21 14:02:26

seth
Member
Registered: 2012-09-03
Posts: 50,010

Re: [Solved] Is there a way of speeding up rsync?

The -a flag shouldn't be there.

And why not?
Check the rsync manpage on what "-a" equals to.
You've two of the options listed explicitly (those could go) and which of the other ones do you think you can/should spare on a system backup (let alone whether that would speed up the process)

I noticed most of the slowness happens backing up home dot directories

Ie. on many small files… I'd rather suspect -A and -X to be the slowing factor.
You probably want them for the system but can spare them on $HOME and in that case I'd run two cycles - one for the system and one for $HOME

Edit: I'd also check top/iotop to figure what's the limiting factor here (CPU or IO)

Last edited by seth (2022-07-21 14:04:36)

Online

#5 2022-07-21 14:09:04

dogknowsnx
Member
Registered: 2021-04-12
Posts: 648

Re: [Solved] Is there a way of speeding up rsync?

You may also want to exclude directories with large(r) files whose content (the file itself that is) doesn't change (such as audio/video files and pictures, *.pkg.tar.zst files - if you build a lot of aur/local packages), and run a second instance of rsync with '--size-only' on them.

Last edited by dogknowsnx (2022-07-21 14:11:43)


Notifications for Arch Linux package updates
RI - Rest your Eyes and Self

"We are eternal, all this pain is an illusion" - Maynard James Keenan

Offline

#6 2022-07-21 14:17:54

seth
Member
Registered: 2012-09-03
Posts: 50,010

Re: [Solved] Is there a way of speeding up rsync?

rsync should™ default to size + mod time comparism, "-c" is explicit; so I'm not sure skipping only the mod time will help with esp. few large files.

Online

#7 2022-07-21 14:32:39

dogknowsnx
Member
Registered: 2021-04-12
Posts: 648

Re: [Solved] Is there a way of speeding up rsync?

You're right, but in my experience it ('--size-only') is still faster if/when '--verbose' is used.

Last edited by dogknowsnx (2022-07-21 14:34:09)


Notifications for Arch Linux package updates
RI - Rest your Eyes and Self

"We are eternal, all this pain is an illusion" - Maynard James Keenan

Offline

#8 2022-07-21 15:44:08

d_fajardo
Member
Registered: 2017-07-28
Posts: 1,563

Re: [Solved] Is there a way of speeding up rsync?

And why not?
Check the rsync manpage on what "-a" equals to.

Yes I did change my mind after reading the docs further.
I'll have a look at the suggestions.

Offline

#9 2022-07-27 11:12:04

d_fajardo
Member
Registered: 2017-07-28
Posts: 1,563

Re: [Solved] Is there a way of speeding up rsync?

In the end what really made the difference in speed is hardware upgrade. I invested on an NVME drive and enclosure which plugs to a USB C bus and the difference is night and day. Now it takes minutes to backup what used to be hours.
So conclusion is rsync runs fine. It's my choice of bus that is slow. Solved.

Offline

Board footer

Powered by FluxBB