Arch Linux

mamr

Member

Registered: 2008-08-16

Posts: 63

Problem with including file in a rsync transfer

I'm using rsync to backup files. My command looks like this:

rsync -zavhP --exclude-from='/home/user/rsync/rsync.txt' --delete -b --backup-dir '/mnt/backup/deleted/' /home/user/ /mnt/backup/user

The above mentioned rsync.txt looks like this:

- ~/VirtualBox VMs/
+ .config/
+ .vimrc
+ .git
+ .gitignore
+ .zshrc
+ .zsh_history
+ .zshenv
+ .Xresources
+ .offlineimaprc
+ .netrc
+ .mozilla/
+ .local/share/fish/fish_history
- .*

'
Since I'm testing out the fish shell recently, I'm trying to backup the fish_history file. For some reason it doesn't get transferred. I also tried

+ fish_history

but it didn't work either. All the other files from the rsynx.txt get transferred without any problem. What am I doing wrong?

Awebb

Member

Registered: 2010-05-06

Posts: 6,285

Re: Problem with including file in a rsync transfer

What is the exact path of fish_history? Is that even the correct name?

jasonwryan

Anarchist

From: .nz

Registered: 2009-05-09

Posts: 30,424

Website

Re: Problem with including file in a rsync transfer

Not a Sysadmin issue, moving to NC...

---

Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

mamr

Member

Registered: 2008-08-16

Posts: 63

Re: Problem with including file in a rsync transfer

What is the exact path of fish_history? Is that even the correct name?

The exact path is: /home/user/.local/share/fish/fish_history

The file does exist.

The rsync command gets invoked from inside the users home directory, i.e. /home/user/

x33a

Forum Fellow

Registered: 2009-08-15

Posts: 4,587

Re: Problem with including file in a rsync transfer

Try giving the full path to the file.

edacval

Member

From: .LT

Registered: 2008-10-23

Posts: 91

Re: Problem with including file in a rsync transfer

First,  you mixing exclude and include patterns in same file, so you need change command to

rsync -zavP -f 'merge /home/user/rsync/rsync.txt' --delete -b --backup-dir '/mnt/backup/deleted/' /home/user/ /mnt/backup/user

Second, change filtering rules to

- /VirtualBox VMs/
+ /.config/
+ /.vimrc
+ /.git
+ /.gitignore
+ /.zshrc
+ /.zsh_history
+ /.zshenv
+ /.Xresources
+ /.offlineimaprc
+ /.netrc
+ /.mozilla/
## Next escapes final '- .*'  rule, without this will not work
+ /.local/
## uncomment this block, if you need only fish_history file , not whole ~/.local/ dir ##
#+ /.local/share/
#- /.local/*
#+ /.local/share/fish/
#- /.local/share/*
#+ /.local/share/fish/fish_history
#- /.local/share/fish/*
## end block ##
- /.*

mamr

Member

Registered: 2008-08-16

Posts: 63

Re: Problem with including file in a rsync transfer

Thank you! This solution does almost work: rsync now syncs .directory files which seem to be created from the Dolphin file browser. E.g. /home/user/Pictures/.directory does get synced. Why is that?

Awebb

Member

Registered: 2010-05-06

Posts: 6,285

Re: Problem with including file in a rsync transfer

There is no rule, that ignores .* patterns inside sub folders. It appears that rsync suddenly understands what a folder is, the moment a / is present.

mamr

Member

Registered: 2008-08-16

Posts: 63

Re: Problem with including file in a rsync transfer

There is no rule, that ignores .* patterns inside sub folders. It appears that rsync suddenly understands what a folder is, the moment a / is present.

So

+ /.git

does lead to synchronization of every git repo in every subfolder of the folder the rsync command is invoked from but

- /.*

doesn't prevent from additional dotfiles getting synchronized? Do I understand that right? Is there any rule to prevent this problem?

Awebb

Member

Registered: 2010-05-06

Posts: 6,285

Re: Problem with including file in a rsync transfer

I must admit I don't know for sure. I'll experiment a bit and get back at you.

EDIT: Yes, it seems to be exactly that. It looks like rsync is a dick.

EDIT1:

awebb@home rsynctest $ find
.
./.git
./.bla
./rsync.txt
./bla
./bla/.git
./bla/blablub
./bla/blablub/.git
./bla/.bla
awebb@home rsynctest $ rsync -avz -f 'merge rsync.txt' . ../rsynctestmirror/
sending incremental file list
./
rsync.txt
bla/
bla/.bla
bla/.git/
bla/blablub/
bla/blablub/.git/

sent 372 bytes  received 82 bytes  908.00 bytes/sec
total size is 14  speedup is 0.03

with

awebb@home $ cat rsync.txt
+ /.git
- /.*

EDIT2:

How about just .* and not /.* with the slash? I can confirm, that it does not ignore the rsync.txt in my example, that would be .* only. I'd also make sense, because the dot might match just about anything in regular regex (ha!), but not in file-specific rsync. I cannot tell you, why + and - would behave differently, though. I'd say it's just one of the many times rsync tries to enjoy itself at your expense the moment you add or neglect a slash in a path.

I'd wager this is the case here. Play around with rsync paths and different combinations of trailing slashes and existing or non-existing target directories. You'll be amazed at how counter intuitive this is for the average Linux user. It has been suggested, that this is actually BSD style, whatever that means.

Last edited by Awebb (2016-09-15 16:22:53)

mamr

Member

Registered: 2008-08-16

Posts: 63

Re: Problem with including file in a rsync transfer

Thanks for your kind help.

So is it possible to just state the absolute path to the fish_history file in any way? That would be enough for me.

Awebb

Member

Registered: 2010-05-06

Posts: 6,285

Re: Problem with including file in a rsync transfer

Yes, just like the others.

EDIT: To clarify, the path you enter in your text file is relative to the path you have when running the command. It's a text matching pattern. So /path/to/file is the exact string "/path/to/file". If you are inside a folder (cd some_folder), then rsync understands the pattern as anything within that folder. If you look at rsync's output, you'll see what the path looks like internally. Be careful about this little difference:

$ rsync -avz -f 'merge /home/awebb/rsynctest/rsync.txt' /home/awebb/rsynctest/ /tmp/rsyncmirror
sending incremental file list
created directory /tmp/rsyncmirror
./
rsync.txt
.git/
bla/
bla/blablub/

sent 245 bytes  received 93 bytes  676.00 bytes/sec
total size is 13  speedup is 0.04

but:

$ rsync -avz -f 'merge /home/awebb/rsynctest/rsync.txt' /home/awebb/rsynctest /tmp/rsyncmirror
sending incremental file list
created directory /tmp/rsyncmirror
rsynctest/
rsynctest/rsync.txt
rsynctest/bla/
rsynctest/bla/blablub/

Can you spot the difference?

Last edited by Awebb (2016-09-15 23:09:36)

mamr

Member

Registered: 2008-08-16

Posts: 63

Re: Problem with including file in a rsync transfer

Yes, thank you. I had this problem earlier.

I'm still trying to figure out, why it doesn't work.

I tried

+ /.local/share/fish/fish_history
- /.*

but the file doesn't get synchronized.

I tried

+ /.local/
+ /.local/share/fish/fish_history
- /.*

but the file doesn't get synchronized.

I tried

+ /.local/
+ /.local/share/fish/fish_history/
- /.*

but the folder /home/user/.local/share/fish/fish_history/ doesn't get synchronized.

I tried

+ .local/share/fish/fish_history # without the leading '/'
- /.*

but the file doesn't get synchronized.

I tried

+ /.local/
## uncomment this block, if you need only fish_history file , not whole ~/.local/ dir ##
+ /.local/share/
- /.local/*
+ /.local/share/fish/
- /.local/share/*
+ /.local/share/fish/fish_history
- /.local/share/fish/*
## end block ##
- /.*

and the file does get synchronized but I have the above described problem with the .directory files.

Last edited by mamr (2016-09-16 06:38:34)