You are not logged in.
Just to share in case someone has the same need.
I do automatic backups very 3 hours of my home directory. This left my computer slow for some time, and would "reset" disk caching. This patched version of rsync solves that a lot. It comes with tow patches:
- fadvise patch: adds the --drop-cache option, that prevents rsync from using disk cache.
- slowdown patch: from the same author, it does 2 things: sorts file operations on each directory by inode to reduce disk seeks and allows you to specify a microtime to wait between link operations through the parameter --slow-down=USECs
Offline
Off topic but have you considered doing a diff-based backup?
Offline
Have the patches been submitted upstream? If so, have the devs responded?
Last edited by tomk (2012-05-16 20:57:57)
Offline
Off topic but have you considered doing a diff-based backup?
I use rsync with link-dest because I like to search for old stuff without having to restore the old copies. If rdiff-backup can do that I don't know how to achive it, mainly because I didn't know this tool until today.
PS: I didn't know what topic to place it.
Have the patches been submitted upstream? If so, have the devs responded?
Some years ago (i think on 2007), the fadvise patch didn't get accepted because the implementation of POSIX_FADV_DONTNEED in linux was buggy (it would drop from cache files that someelse put), but that was some time ago. On Jan 2011 it got mostly fixed: http://lwn.net/Articles/449420/
On the second patch, the part of sorting files by inode before working on them is how glib already works, but with chunks of 1000 files, as someone reported on lklm. this patch doesn't create chunks, so, on a desktop computer with GBs of ram, if you have directories with milions or bilions of files you would end up with no ram, but I see that as rare situation.
the slowdown part just calls usleep before calling link function.
Last edited by Pajaro (2012-05-16 22:16:16)
Offline
Have the patches been submitted upstream? If so, have the devs responded?
Well, I was wrong. After a deeper read on the article I link, it seems that it didn't get fixed at all. Instead of fixing POSIX_FADV_DONTNEED, what they did was to implement POSIX_FADV_NOREUSE.
The difference between them is that POSIX_FADV_DONTNEED drops the file from cache, while POSIX_FADV_NOREUSE marks the cache page as inactive, instead of dropping.
Anyway, POSIX_FADV_DONTNEED already has a great benefit to me (i just backup my data, so application start remains instant after rsync calls), POSIX_FADV_NOREUSE would be a better approach. But I see a big disavantage with POSIX_FADV_NOREUSE: servers don't have new kernels. I also backup servers, so I can set this patched rsync there without worring about the kernel being recent enough. I could set a dummy --drop-cache argument on the servers, but, as mentioned before, I already see a big difference with this patch.
I run a test (16GB of ram):
1. open all programs I use (eclipse, virtualbox, netbeans, chorme, chromium, firefox, dolphin, filezilla, dolphin)
2. run rsync remote server (around 150GB, most of it gets hardlinked), once without --drop-cache, once with --drop-cache
3. open all programs again
Without --drop-cache application start would stop being instant and "free -m" would show that buffers/cache used all free mem.
With --drop-cache enabled, 1.6GB of ram remain untouched after the test (not used even for buffers or cache, though untouched mem got reduced around 1-2 GB during the process) and application start remains instant. This is what I looked for.
Offline
I use rsync with link-dest because I like to search for old stuff without having to restore the old copies. If rdiff-backup can do that I don't know how to achive it, mainly because I didn't know this tool until today.
There is an important difference: the handling of changed files. rsync --link-dest stores the whole file even when just one bit has been changed. In contrast rdiff-backup stores only that one bit as a chunk. This makes rdiff-backup way more space-efficient especially with big files changing only a little, like logfiles appended or pictures tagged. This also means that older backups done by rdiff-backup are only accessible by rdiff-backup. Luckily there is a fuse filesystem that makes all older sets conveniently browsable: rdiff-backup-fs. Just mount the archive with it, and all files are there like in a file system.
zʇıɹɟʇıɹʞsuɐs AUR || Cycling in Budapest with a helmet camera || Revised log levels proposal: "FYI" "WTF" and "OMG" (John Barnette)
Offline
Luckily there is a fuse filesystem that makes all older sets conveniently browsable: rdiff-backup-fs. Just mount the archive with it, and all files are there like in a file system.
Thanks SanskritFritz, this is great. By now I will stay with rsync because it took me a lot of hours of research, trial and thinking to achieve a configuration that actually satisfies me, but I take note of rdiff-backup and this fuse filesystem for my next step on improving my backup scripts.
Offline