anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

ndejay · 2012-11-01 01:38:30

I'm sorry, I had tested it earlier today and it failed to delete something. I just did a test right now and it works flawlessly. Sorry for the false alarm!

schnedan · 2012-11-02 19:49:56

Hi, This is the official support for the asd? so, first: is it possible to solve the problem with systemd preventing it to use asd for /var/log?... 2nd: When analysing my home directory, firefox and opera are the applications writing a lot to my poor ssd... trying to add ~/.mozilla to asd ended up with error messages: firefox cant access the profile plus a unuseable profile-backub by asd. How to make it work? (does adding ~/.opera to asd work?)

graysky · 2012-11-02 20:24:34

1) No idea since I am not familiar with the inner workings of systemd.
2) Use profile-sync-daemon for browsers since it was written/optimized for them. Asd is really for other things.

gehidore · 2012-11-09 07:44:20

Just tried to use asd on /var/lib/mysql and it appears to move everything to shm as the symlink is created and the right volume of space is used I can't access the files even as root.

ls -al /var/lib/mysql shows /dev/shm/asd-mysql/
ls -al /dev/shm/ only shows the other items I've successfully setup with asd any thoughts? I've got a 3.6G database I want to load up and run some search / replaces on and I would love to have it loaded to ram for these.

Thanks

-Beaver

graysky · 2012-11-09 20:15:17

@gehidore - Not familiar with mysqld to know what kind of permissions are required for the databases. For example, on my system:

% mount | grep shm
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)

I would also think that it would be key that mysqld starts AFTER the sync is complete. Have you stopped mysqld and restarted it to verify nothing funny there?

EDIT: You did backup your database prior, right?

Last edited by graysky (2012-11-09 20:15:37)

gehidore · 2012-11-10 06:51:32

drwx------ 6 mysql mysql 4096 Nov  7 18:17 mysql

That's the directory before starting asd.

I did just now shutdown mysqld and asd, re-added and restarted asd, once it was finished I then started mysqld up and the directory on shm still doesn't exist.

Luckily this is a new machine so I haven't even imported the database yet due to the size was hoping to use asd to the advantage.

graysky · 2012-11-10 11:11:41

gehidore wrote:

drwx------ 6 mysql mysql 4096 Nov  7 18:17 mysql
That's the directory before starting asd.
I did just now shutdown mysqld and asd, re-added and restarted asd, once it was finished I then started mysqld up and the directory on shm still doesn't exist.
Luckily this is a new machine so I haven't even imported the database yet due to the size was hoping to use asd to the advantage.

@gehidore - You didn't show me anything useful there.

1) Stop asd.
2) Update to version 3.09 of asd (just released it).
3) Post the output of:

$ anything-sync-daemon debug

Last edited by graysky (2012-11-10 12:38:27)

graysky · 2012-11-10 12:49:59

pezz wrote:

Adding /var/log causes systemd-journald to throw errors on the next reboot.
Not sure if anything can be done about this - assuming asd.service is shut down and rsyncs, then moves the directory, while journald is still running.

The recent changes I made to v3.09 of asd seems to allow one to sync /var/log (at least with my testing in a VM). I am interested to see more feedback from people willing to test syncing /var/log using version >=3.09. Please reply here with your experiences. Only after some robust testing can I feel comfortable endorsing the recommendation again.

EDIT: Seems to recover from an ungraceful power down as well (again in the VM).

Last edited by graysky (2012-11-10 13:36:45)

ZekeSulastin · 2012-11-10 13:31:35

people wrote:

systemd and /var/log onoz

Did any of you happen to try setting Storage=volatile (or none) in /etc/systemd/journald.conf so that journald doesn't actually do anything with /var/log/?

gehidore · 2012-11-10 19:46:50

graysky wrote:

@gehidore - You didn'tw me anything useful there.
1) Stop asd.
2) Update to version 3.09 of asd (just released it).
3) Post the output of:
$ anything-sync-daemon debug

Sorry about that, did as you instructed and now it works. Output is such:

::DEBUG MODE::
anything-sync-daemon version: 3.09

settings in /etc/asd.conf will make anything-sync-daemon manage the following dir/user combos when invoked:

target to manage:  /var/lib/mysql
owner/group:       mysql:mysql
backup target:     /var/lib/mysql-backup_asd
link target:       /dev/shm/asd-mysql/var/lib/mysql

graysky · 2012-11-10 20:41:49

gehidore wrote:

target to manage: /var/lib/mysql
owner/group: mysql:mysql
backup target: /var/lib/mysql-backup_asd
link target: /dev/shm/asd-mysql/var/lib/mysql

Ok... And when you start the daemon, does the link target get created? What about the backup target?

gehidore · 2012-11-11 05:33:01

graysky wrote:

Ok... And when you start the daemon, does the link target get created? What about the backup target?

Yes and Yes, everything appears to work.

Thanks!

Ran the db query/replace last night against the onboard ssd took about 72 minutes, I'm curious to see the same query/replace again with the database loaded to shm.

-Beaver

graysky · 2012-11-11 06:17:08

gehidore wrote:

Yes and Yes, everything appears to work.
Thanks!
Ran the db query/replace last night against the onboard ssd took about 72 minutes, I'm curious to see the same query/replace again with the database loaded to shm.

Ok... so it asd works sync'ing mysql? Do report the diff searching on your SSD vs RAM.

gehidore · 2012-11-11 07:09:37

graysky wrote:

Ok... so it asd works sync'ing mysql? Do report the diff searching on your SSD vs RAM.

SSD is sata3 with xfs, ram is PC12800

Less than half the time, 1392.402372 seconds , err 23.2 minutes.

Last edited by gehidore (2012-11-11 07:13:34)

graysky · 2012-11-11 07:20:36

gehidore wrote:

SSD is sata3 with xfs, ram is PC12800
Less than half the time, 1392.402372 seconds , err 23.2 minutes.

Without asd: 72 min
With asd: 23 min

...over 3x faster

gehidore · 2012-11-11 07:26:06

graysky wrote:

Without asd: 72 min
With asd: 23 min
...over 3x faster

How I managed to think 25 was half of 70... it's too late for my brain to function I think.

gehidore · 2012-11-25 06:21:56

I do have one thought that just hit me after transfering a bunch of files back and forth... Why isn't the backup directory hidden? I use tab so much I end up with random tabs while typing emails and such so it gets a little frustrating trying to move items about.

Just a thought.

teateawhy · 2012-12-11 22:19:58

graysky wrote:

pezz wrote:
Adding /var/log causes systemd-journald to throw errors on the next reboot.
Not sure if anything can be done about this - assuming asd.service is shut down and rsyncs, then moves the directory, while journald is still running.
The recent changes I made to v3.09 of asd seems to allow one to sync /var/log (at least with my testing in a VM). I am interested to see more feedback from people willing to test syncing /var/log using version >=3.09. Please reply here with your experiences. Only after some robust testing can I feel comfortable endorsing the recommendation again.
EDIT: Seems to recover from an ungraceful power down as well (again in the VM).

Currently i have two different arch linux installations running this setup:

WHATTOSYNC=('/var/log' ...
VOLATILE="/dev/shm"

systemd/journald.conf

[Journal]
Storage=volatile

EDIT: asd is version 3.10-1

Both are running perfectly, one is a server another a laptop. I can also confirm my laptop survives the ungraceful powerdown, "last | grep crash" finds 4 crashs during the last month. In my opinion the key is configuring journald.conf not to touch /var/log and use /tmp instead, but systemd is indeed much like a black-box.
This seems to be in accordance with the manpage about journald.conf :

man journald.conf wrote:

Storage=
Controls where to store journal data. One of volatile, persistent, auto and none. If volatile journal log data will be
stored only in memory[..] If persistent data will be stored [..] below the /var/log/journal hierarchy [...]

ZekeSulastin wrote:

Did any of you happen to try setting Storage=volatile (or none) in /etc/systemd/journald.conf so that journald doesn't actually do anything with /var/log/?

exactly

Last edited by teateawhy (2012-12-11 22:21:29)

gehidore · 2012-12-11 22:24:17

I had been syncing my work directory + mysqld + http and after the cat unplugged the cord into the ac adapter and the batter died I find that I've lost more than 3 hours of work, luckily I had just pushed my work to github not 30 minutes before but, if it's syncing every hour by default why did 3 hours vanish?

graysky · 2012-12-12 00:47:44

gehidore wrote:

I had been syncing my work directory + mysqld + http and after the cat unplugged the cord into the ac adapter and the batter died I find that I've lost more than 3 hours of work, luckily I had just pushed my work to github not 30 minutes before but, if it's syncing every hour by default why did 3 hours vanish?

Did I have it set to 3 h before? Thought 1 h was the default.

gehidore · 2012-12-12 01:09:38

Mine says 1 hour, not sure why or what unless it was a filesystem error.

gehidore · 2012-12-12 17:42:01

For fun this morning I loaded up a bunch of 'work files' , simple script to modify them every 3 minutes and stop in three hours which my battery said 2 hours 40 minutes remaining. Let it run the battery down. I didn't lose more than the 1 hour of data this time. The big difference is I did recently switch from XFS to ext4 via tar ...

nierro · 2012-12-31 10:18:34

@gravsky: do you think it is useful to keep sync'ed my /home/user/.thumbnails dir? It is 30Mb and it may make my system faster (everything that loads thumbnails will load faster, i guess; e.g. menu will be showed faster, browsing in spacefm too will be faster...)
Am I missing anything or it can be a good idea?
Thanks for your work, and for your hint
Good year!

graysky · 2012-12-31 13:10:45

@nierro - You can ... but I don't see what you would gain by doing so unless you do LOTS of I/O to that directory.

nierro · 2013-01-04 19:33:52

It was a bad idea, i guess
Thanks for pointing that out!

Arch Linux

#176 2012-11-01 01:38:30

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#177 2012-11-02 19:49:56

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#178 2012-11-02 20:24:34

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#179 2012-11-09 07:44:20

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#180 2012-11-09 20:15:17

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#181 2012-11-10 06:51:32

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#182 2012-11-10 11:11:41

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#183 2012-11-10 12:49:59

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#184 2012-11-10 13:31:35

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#185 2012-11-10 19:46:50

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#186 2012-11-10 20:41:49

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#187 2012-11-11 05:33:01

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#188 2012-11-11 06:17:08

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#189 2012-11-11 07:09:37

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#190 2012-11-11 07:20:36

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#191 2012-11-11 07:26:06

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#192 2012-11-25 06:21:56

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#193 2012-12-11 22:19:58

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#194 2012-12-11 22:24:17

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#195 2012-12-12 00:47:44

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#196 2012-12-12 01:09:38

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#197 2012-12-12 17:42:01

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#198 2012-12-31 10:18:34

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#199 2012-12-31 13:10:45

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#200 2013-01-04 19:33:52

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

Board footer