anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

graysky · 2012-04-06 21:34:13

Quite a few people like and use profile-sync-daemon. I have received several requests now asking for a more general version that will allow the flexibility of syncing anything from tmpfs <--> hdd and back again. I just uploaded the first public release of anything-sync-daemon in response to the requests. I am currently working on the wiki page, but I have already written a full manpage that is included with the util. As with anything, BACKUP the data you wish to sync before using it.

What are some example directories you might want to sync to tmpfs?

/srv/http
/foo/bar
/var/lib/monitorix

AUR Package: https://aur.archlinux.org/packages.php?ID=58263
Wiki Page: https://wiki.archlinux.org/index.php/An … ync-daemon
Gitrepo if you just wanna browse the code (it is VERY simplistic): https://github.com/graysky2/anything-sync-daemon

Last edited by graysky (2012-09-22 15:00:04)

karol · 2012-04-06 21:44:44

foo-sync-daemon worked fine, but could the new package be renamed to sync-anything-daemon?

graysky · 2012-04-06 21:50:51

karol wrote:

foo-sync-daemon worked fine, but could the new package be renamed to sync-anything-daemon?

@karol - I like asd because it is similar to psd. Sync browser profiles with profile-sync-daemon; sync anything with anything-sync-daemon I actually think that foo-sync-daemon is kinda cute too.

karol · 2012-04-06 21:52:52

OK :-)

Thanks for another nice tool, I was looking for something just like this.

graysky · 2012-04-06 21:53:57

Please bang away on it and let me know if it breaks in any way. Also, if you have suggestions to make it more robust, I am all ears.

Last edited by graysky (2012-04-06 21:54:23)

nierro · 2012-04-07 08:44:59

@ gravsky, thanks this is wonderful!
Can i use it to sync my chromium cache too? (i only need to put that folder inside asd.conf?)
And lastly, can this replace psd? (i think yes, it can)
EDIT: sorry, another little question: can i use it to sync everything from my .cache folder in tmpfs? Or is it dangerous?
Thanks again for your time

Last edited by nierro (2012-04-07 09:02:39)

graysky · 2012-04-07 09:04:59

nierro wrote:

Can i use it to sync my chromium cache too? (i only need to put that folder inside asd.conf?)

You can but do not cause a race-condition between asd and psd. For chromium, you technically can do this since the cache dir is NOT part of the profile directory. This is NOT general for all browsers. A better solution would be to do this as the man page and wiki article suggest.

nierro wrote:

And lastly, can this replace psd? (i think yes, it can)

Technically yes, but psd knows about different browsers and users whereas asd is 'dumb' in the sense that it blindly syncs what you tell it to sync. I would recommend against it.

nierro · 2012-04-07 09:09:52

That is what i'm doing: launching chromium with cache in /tmp! But i was wondering, what if i put everything from my cache inside tmpfs? is it dangerous?
Thanks for the answers and for the good work!

Edit: i have a little idea, but i don't think you will appreciate it very much... May be you can merge the two projects and putting in the code a little check: if the name inside "what to sync" is a name between the browsers you defined, then use the rules coming from psd, else, put that folder in tmpfs.
Is it a stupid thing?

Last edited by nierro (2012-04-07 09:12:25)

graysky · 2012-04-07 09:12:05

It's not dangerous. Just be aware that chromium is unique in that its cache dir is not under its profile dir. This is NOT true for firefox. What I don't want to happen is for users to install both psd and asd and setup a race condition where psd is managing the profile BUT asd is managing a directory INSIDE the profile all the same time.

...why do you want to have your cache survive a reboot anyway?

nierro · 2012-04-07 09:14:23

graysky wrote:

...why do you want to have your cache survive a reboot anyway?

Sorry gravsky, i'm a perfectly id..t... i woke up an hour ago, so don't know what i was thinking about

graysky · 2012-04-07 09:18:36

Maybe there is a reason, I just can't think of one.

nierro · 2012-04-07 10:56:24

I'm having sort of troubles: it doesn't seem to mount anything into /shm...
I told it to keep sync'ed /var/log and my chromium config (yes i don't like to have 2 daemons running, doing the same thing).
So this is my asd.conf:

#
# /etc/asd.conf
#
# For documentation, see: https://wiki.archlinux.org/index.php/Anything-sync-daemon

# Define where data will reside in tmpfs
# Think hard about this if using utils like bleachbit as it has a nasty habbit
# of nuking files it identifies as junk in /tmp
# 
# A safer location for things is actually /dev/shm
#
# This location must be mounted to tmpfs and MUST have permissions of 777
# Use NO trailing backslash!
TMPFS="/dev/shm"

# Define the target(s) directories in the WHATTOSYNC array 
# Do NOT define a file! These MUST be directories!
#
# Note that the target directories and all subdirs under them will be included
# In other words, this is recursive
#
# Below is an example to wet your appetite
#WHATTOSYNC=('/var/log' '/srv/http' '/home/foo/bar')
WHATTOSYNC=('/var/log' '/home/federico/.config/chromium')

and this is my /etc/mtab:

rootfs / rootfs rw 0 0
/dev/root / ext4 rw,noatime,user_xattr,acl,barrier=1,nodelalloc,data=ordered 0 0
devtmpfs /dev devtmpfs rw,relatime,size=1027028k,nr_inodes=218985,mode=755 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sys /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
run /run tmpfs rw,nosuid,nodev,relatime,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620 0 0
shm /dev/shm tmpfs rw,nosuid,nodev,relatime 0 0
tmpfs /tmp tmpfs rw,nosuid,nodev,noatime,size=614400k 0 0
/var/tmp /var/tmp tmpfs rw,noatime,nodiratime 0 0
gvfs-fuse-daemon /home/federico/.gvfs fuse.gvfs-fuse-daemon rw,nosuid,nodev,relatime,user_id=1000,group_id=100 0 0

So, don't know where is the problem... my /var/log folder is 65Mb, but if i run df -h, then, this is its output:

File system     Dim. Usati Dispon. Uso% Montato su
rootfs          231G   31G    189G  14% /
/dev/root       231G   31G    189G  14% /
devtmpfs       1003M     0   1003M   0% /dev
run            1004M  344K   1003M   1% /run
shm            1004M  152K   1003M   1% /dev/shm
tmpfs           600M  9,4M    591M   2% /tmp
/var/tmp       1004M   84K   1004M   1% /var/tmp

Thanks for the help!

graysky · 2012-04-07 11:56:17

My bad... I need to fix my 'sanity check' code. Gimme 10 min.

nierro · 2012-04-07 12:20:17

thanks!

graysky · 2012-04-07 12:35:36

Took a little longer than 10 min. Updated to v1.2 which should have everything in place. Please let me know if you run into anything else.

nierro · 2012-04-07 12:50:22

Ok now it just works fine!
Thank you very much!
A little question: my chromium cache, mounted in /tmp as wiki says, is always increasing and increasing (until i reboot).
But what if i'm looking at a film in streaming that is larger than the 600Mb i left to my /tmp in tmpfs?

graysky · 2012-04-07 12:51:29

No idea. Do it and report back.
...doesn't chromium have a upper limit to the cache size?

Last edited by graysky (2012-04-07 12:51:57)

nierro · 2012-04-07 12:55:01

I don't know...well, we'll see when i'll look at a film
Thank you again for your time!

spikeh · 2012-04-15 14:58:34

I just upgraded my RAM to 8GB and just as I started to look into using tmpfs to cut some disk IO for my SSD I find this lovely daemon! Thank you so much.

graysky · 2012-04-15 15:25:32

You're welcome, enjoy.

morxa · 2012-04-28 14:49:33

Thanks for asd, it works like a charm!

One thing though: If somebody tries to sync a folder foo and there is already a folder foo-backup (created by the user), the two folders are merged - or even worse the files in foo-backup are deleted. While this is not a very common use case, it's something you should consider... maybe you could simply name the backup folder "foo-asd"?

graysky · 2012-04-28 15:25:21

@morxa - Thanks for the kind words and the suggestion. It's a good idea.

EDIT: just pushed v1.3

Last edited by graysky (2012-04-28 16:06:09)

Gregorz · 2012-04-28 16:25:10

Thanks for your effort. Is there a way how I can test if it works?

graysky · 2012-04-28 16:36:57

Gregorz wrote:

Thanks for your effort. Is there a way how I can test if it works?

Yeah

makepkg -srci # build version 1.3 from the AUR
. /etc/asd.conf
ls -a $WHATTOSYNC*

You should see the links and path(s) to what you're syncing. Note that the backups new end in "*-backup_asd"

Last edited by graysky (2012-04-28 16:38:06)

Gregorz · 2012-04-28 17:36:15

Unfortunately

  -> Lade anything-sync-daemon-1.3.tar.xz herunter...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0
curl: (22) The requested URL returned error: 404

EDIT:

Sorry do not want to sound harsh I had a rough day since my first hdd crashed today. Thanks for your great word. But maybe you have a typo in the pkgbuild?

With best regards,

Greg

EDIT 2 Found the error the new version is not on your server so far.

Last edited by Gregorz (2012-04-28 17:44:35)

Arch Linux

#1 2012-04-06 21:34:13

anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#2 2012-04-06 21:44:44

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#3 2012-04-06 21:50:51

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#4 2012-04-06 21:52:52

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#5 2012-04-06 21:53:57

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#6 2012-04-07 08:44:59

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#7 2012-04-07 09:04:59

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#8 2012-04-07 09:09:52

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#9 2012-04-07 09:12:05

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#10 2012-04-07 09:14:23

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#11 2012-04-07 09:18:36

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#12 2012-04-07 10:56:24

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#13 2012-04-07 11:56:17

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#14 2012-04-07 12:20:17

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#15 2012-04-07 12:35:36

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#16 2012-04-07 12:50:22

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#17 2012-04-07 12:51:29

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#18 2012-04-07 12:55:01

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#19 2012-04-15 14:58:34

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#20 2012-04-15 15:25:32

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#21 2012-04-28 14:49:33

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#22 2012-04-28 15:25:21

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#23 2012-04-28 16:25:10

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#24 2012-04-28 16:36:57

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#25 2012-04-28 17:36:15

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

Board footer