anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

capoeira · 2012-07-05 18:00:01

nice script, thanks

besides /var/log, what other directories would be interesting for a performance improofment?

graysky · 2012-07-06 01:33:55

Dunno

capoeira · 2012-07-06 03:09:35

hehehe

I'll combine this with preload

capoeira · 2012-07-08 13:52:10

how do I check taht this is actualy running?

graysky · 2012-07-08 14:15:42

It is just a call to rsync so nothing is running. There is an entry for corn that triggers it.

capoeira · 2012-07-09 22:27:28

graysky wrote:

It is just a call to rsync so nothing is running. There is an entry for corn that triggers it.

ah thanks, actually I can "see" it's syncing because of longer boot time

victoru · 2012-07-19 00:21:26

graysky wrote:

@unikum - this is a known issue I haven't yet had time to address.
xduugu helped solve this for psd:
https://github.com/graysky2/profile-sync-daemon
Anyone game to do it for asd? I just do not have the time right now.
https://github.com/graysky2/anything-sync-daemon

I've taken a stab at it here: https://github.com/victoru/anything-sync-daemon

I was having the same issues as unikum. I have systemd-sysvcompat installed and successfully started/stopped the asd daemon.

graysky · 2012-07-19 00:33:27

Thanks... Send a pull request if you haven't already and I'll take a look when I have time.

Mox · 2012-07-22 11:19:53

Hi,
asd works like a charm, but I have got a little suggestion: If you change

for i in "${WHATTOSYNC[@]}"; do
		...
done
...

to

for j in "${WHATTOSYNC[@]}"; do
	for i in $j; do
		if [[ "$i" != *-backup_asd ]] ; then

		...
		else
			/bin/true
		fi
	done
done
...

in /usr/bin/anything-sync-daemon, then one could synch all matching folders. This comes handy, to keep all those little config and log files residing in home in ram (I tried synching the whole folder, that didn't work very well) with a line like:

WHATTOSYNC=('/var/log' '/srv/http' '/home/<username>/.*')

.

Last edited by Mox (2012-07-22 18:06:03)

drewofdoom · 2012-07-27 01:36:09

victoru wrote:

graysky wrote:
@unikum - this is a known issue I haven't yet had time to address.
xduugu helped solve this for psd:
https://github.com/graysky2/profile-sync-daemon
Anyone game to do it for asd? I just do not have the time right now.
https://github.com/graysky2/anything-sync-daemon
I've taken a stab at it here: https://github.com/victoru/anything-sync-daemon
I was having the same issues as unikum. I have systemd-sysvcompat installed and successfully started/stopped the asd daemon.

Using your version on my machine. Works well, but there's an interesting issue. Using the prefix asd-$LOGNAME results in the directory structure becoming thus:

/dev/shm/asd-/*

It's not picking up any logname. Is there any particular reason we need to have a logname at all ? Is there any reason we can't leave the prefix simply as asd?

Also, tried with prefix="asd-$USER" and got the same result.

victoru · 2012-07-27 18:46:48

drewofdoom wrote:

Using your version on my machine. Works well, but there's an interesting issue. Using the prefix asd-$LOGNAME results in the directory structure becoming thus:
/dev/shm/asd-/*
It's not picking up any logname. Is there any particular reason we need to have a logname at all ? Is there any reason we can't leave the prefix simply as asd?
Also, tried with prefix="asd-$USER" and got the same result.

It was added to support running multiple asd configs using the same tmpfs.

It seems to have resolved itself on my maching as I have two directories:

/dev/shm/asd-
/dev/shm/asd-<myusername> (asd is using this)

I did change PREFIX_PERMISSIONS to 755 in asd.conf, not sure if that'd be it.

mikkie · 2012-07-29 14:48:17

graysky, just wanted to express my satisfaction with anything-sync-daemon, profile-sync-daemon and linux-ck. Why don't you apply for a Trusted User? I (and probably many others) would so like to see these packages in the official repositories.

Tea2 · 2012-08-02 20:28:01

I was looking into this for savegames, since I've got a terrible computer, saving in Skyrim caused a huge amount of stutter. The basic idea was to use asd to save into /tmp instead.

Unfortunately, though, it doesn't seem to work. The path to the saves is /home/joe/Documents/My Games/Skyrim/Saves; so I added that to asd.conf (I seemed to need to escape the space in My Games; My\ Games). The resulting /tmp/asd/... wasn't writable, and when I made it writable it didn't sync files that I touched in there. Setting it to /dev/shm didn't show anything at all.

It's weird because profile-sync-daemon is working absolutely fine.

Edit: I think I've found the issue. The lines that invoke mv and ln in /usr/bin/anything-sync-daemon don't seem to want to play well with directories that have spaces in them.

[joe@compy386 ~]$ sudo /etc/rc.d/asd restart
:: Syncing tmpfs to physical disc                                        [BUSY] 
mv: cannot stat ‘Games/Skyrim/Saves’ : No such file or directory
ln: target ‘Games/Skyrim/Saves’ is not a directory

I'm not entirely sure how to put $strings in quotations since doing just '$string' would make the script take $string literally. So obviously when I tried that it output "cannot stat ‘$LINK’".

Edit 2: I've got it. Sort of.
I put every instance of $LINK, $STATIC and $VOLATILE in the script in double quotation marks. This at least enabled the sync to take place, but now something strange happens.
/home/joe/Documents/My Games/Skyrim/Saves/Saves is created, containing a copy of everything in the original Saves directory. If I restart the daemon, /home/joe/Documents/My Games/Skyrim/Saves/Saves/Saves is created. If I restart again; /home/joe/Documents/My Games/Skyrim/Saves/Saves/Saves/Saves and so on.

I don't even know where to begin on this one.

Last edited by Tea2 (2012-08-02 21:26:21)

graysky · 2012-08-11 00:20:49

OK gang, total re-write to bring this code in sync with psd. More simplistic and streamlined now. Plz report bugs here.

Smasher816 · 2012-08-19 22:28:39

Recently when opening chromium I get a yellow alert by the wrench saying "Sync Error: Sign in again".
In addition whenever I reboot my computer all the "recently visited" pages,saved passwords, and other cookies are gone.
Only the settings (including about:flags) are preserved.
I can only assume this is an issue with asd, which hopefully can be corrected easily.

WHATTOSYNC=('/var/log' '/home/rowand/.cache' '/home/rowand/.config/chromium')

$systemctl status asd.service
asd.service - Symlinks and syncs user defined dirs to RAM thus reducing HDD/SDD calls and speeding-up the system
	  Loaded: loaded (/usr/lib/systemd/system/asd.service; enabled)
	  Active: active (exited) since Sun, 19 Aug 2012 16:48:13 -0500; 38min ago
	 Process: 1872 ExecStart=/usr/bin/anything-sync-daemon sync (code=exited, status=0/SUCCESS)
	  CGroup: name=systemd:/system/asd.servic

$ls -a $WHATTOSYNC*
/home/rowand/.cache:
.  ..  chromium  dconf  dmenu_run  .flagged  g-ir-scanner

/home/rowand/.config/chromium:
.                             .flagged                       Safe Browsing Csd Whitelist
..                            Local State                    Safe Browsing Download
Certificate Revocation Lists  Safe Browsing Bloom            Safe Browsing Download Whitelist
chromium-backup_asd           Safe Browsing Bloom Filter 2   SingletonCookie
Default                       Safe Browsing Cookies          SingletonLock
First Run                     Safe Browsing Cookies-journal  SingletonSocket

/home/rowand/.config/chromium-backup_asd:
.  ..  chrome_shutdown_ms.txt  chromium-backup_asd  Default  Local State

/var/log:
.   btmp  faillog   lastlog  pacget.log  wtmp        Xorg.0.log.old
..  cups  .flagged  old      pacman.log  Xorg.0.log

$du --max-depth 1 -h /dev/shm | grep asd
312M	/dev/shm/asd-rowand
616K	/dev/shm/asd-root

Last edited by Smasher816 (2012-08-19 22:33:18)

graysky · 2012-08-19 23:49:24

Have to dig into it... Any reason why you don't use psd for browser profiles?

Mr Green · 2012-08-20 05:53:49

@graysky do I need to uncomment VOLATILE variable? or simply leave it as default?

graysky · 2012-08-20 08:49:48

Either is fine. Level it commented just like those defaults in /etc/ssh/sshd_config

Mr Green · 2012-08-20 17:26:55

Tried without got respawn error so added '#' again, reason I ask just got one of those new fangled SSD drives so need to shift some stuff

Thanks

graysky · 2012-08-20 18:00:47

The script defines it if is null in the config. You only need to uncomment if it makes you feel better or if you want another path.

#!/bin/bash
# By graysky <graysky AT archlinux DOT us>
# Inspired by some code originally  written by Colin Verot
...

[[ -z "$VOLATILE" ]] && VOLATILE="/dev/shm"

...

Last edited by graysky (2012-08-20 18:46:52)

Mr Green · 2012-08-20 18:16:31

Might try it with firefox, although there is an about :config option I want to try first.

Smasher816 · 2012-08-22 02:33:10

graysky wrote:

Have to dig into it... Any reason why you don't use psd for browser profiles?

I (like a few others here) figured I would just use one 'daemon'.

Anyways, I found the problem...
My directories were: ~/.config/chromium-backup-asd/chromium-backup-asd/{a,b,c}

I'm not sure how a backup folder got nested, but removing the container fixed it.
As you would expect it was restoring chromium-backup-asd to .config/chromium instead of the actual chromium files.

Edit: After another reboot the same problem has returned.

Last edited by Smasher816 (2012-08-22 23:22:06)

MFserver · 2012-08-29 13:14:02

This is just plain awesome! I've tried a little on my main laptop (2 gigs of RAM), and tried on a 1 gig machine too, but the fun will begin for real when I get my new laptop with 20G RAM (/ in RAM, here we go...). I'll do some extensive testing and report any issues, thanks for this awesome little package!

graysky · 2012-08-29 19:05:50

@MFs - You're welcome. I wouldn't advise that you put / in tmpfs though.

MFserver · 2012-08-29 21:37:13

Indeed, I did get an issue when setting asd to sync /usr/bin (I know you wouldn't normally sync binaries, but I wanted to experimnent a little) and this is what happened:

[root@mfserver /home/mfserver]#
>> rc.d start asd
:: Starting Anything-Sync-Daemon          [BUSY]
bash: rsync: kommandot finns inte
bash: touch: kommandot finns inte         [DONE]

Basically, it moved the files to /usr/bin-backup_asd, created a symlink for /usr/bin, and then (I guess) tried to rsync, which didn't work, because rsync is in /usr/bin, that did not exist. I managed to restore /usr/bin, though. Is this to be handled as a bug, or is it just a consequence of the program's design? Maybe I'll have to follow the various guides avalible for debian-based systems and rebuild the initscripts to get executables in ram.

Arch Linux

#76 2012-07-05 18:00:01

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#77 2012-07-06 01:33:55

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#78 2012-07-06 03:09:35

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#79 2012-07-08 13:52:10

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#80 2012-07-08 14:15:42

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#81 2012-07-09 22:27:28

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#82 2012-07-19 00:21:26

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#83 2012-07-19 00:33:27

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#84 2012-07-22 11:19:53

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#85 2012-07-27 01:36:09

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#86 2012-07-27 18:46:48

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#87 2012-07-29 14:48:17

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#88 2012-08-02 20:28:01

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#89 2012-08-11 00:20:49

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#90 2012-08-19 22:28:39

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#91 2012-08-19 23:49:24

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#92 2012-08-20 05:53:49

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#93 2012-08-20 08:49:48

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#94 2012-08-20 17:26:55

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#95 2012-08-20 18:00:47

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#96 2012-08-20 18:16:31

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#97 2012-08-22 02:33:10

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#98 2012-08-29 13:14:02

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#99 2012-08-29 19:05:50

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#100 2012-08-29 21:37:13

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

Board footer