You are not logged in.

#176 2012-11-01 01:38:30

ndejay
Member
Registered: 2012-10-30
Posts: 7

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

I'm sorry, I had tested it earlier today and it failed to delete something. I just did a test right now and it works flawlessly. Sorry for the false alarm!

Offline

#177 2012-11-02 19:49:56

schnedan
Member
Registered: 2012-11-02
Posts: 11

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

Hi, This is the official support for the asd? so, first: is it possible to solve the problem with systemd preventing it to use asd for /var/log?... 2nd: When analysing my home directory, firefox and opera are the applications writing a lot to my poor ssd... trying to add ~/.mozilla to asd ended up with error messages: firefox cant access the profile plus a unuseable profile-backub by asd. How to make it work? (does adding ~/.opera to asd work?)

Offline

#178 2012-11-02 20:24:34

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

1) No idea since I am not familiar with the inner workings of systemd.
2) Use profile-sync-daemon for browsers since it was written/optimized for them.  Asd is really for other things.


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#179 2012-11-09 07:44:20

gehidore
IRC Op
From: Near the pine trees
Registered: 2012-11-09
Posts: 35

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

Just tried to use asd on /var/lib/mysql and it appears to move everything to shm as the symlink is created and the right volume of space is used I can't access the files even as root.

ls -al /var/lib/mysql shows /dev/shm/asd-mysql/
ls -al /dev/shm/ only shows the other items I've successfully setup with asd any thoughts? I've got a 3.6G database I want to load up and run some search / replaces on and I would love to have it loaded to ram for these.

Thanks

-Beaver


Windows get broken, penguins rarely get sucked into jet engines.

Offline

#180 2012-11-09 20:15:17

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

@gehidore - Not familiar with mysqld to know what kind of permissions are required for the databases.  For example, on my system:

% mount | grep shm
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)

I would also think that it would be key that mysqld starts AFTER the sync is complete.  Have you stopped mysqld and restarted it to verify nothing funny there?

EDIT: You did backup your database prior, right?

Last edited by graysky (2012-11-09 20:15:37)


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#181 2012-11-10 06:51:32

gehidore
IRC Op
From: Near the pine trees
Registered: 2012-11-09
Posts: 35

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

drwx------ 6 mysql mysql 4096 Nov  7 18:17 mysql

That's the directory before starting asd.

I did just now shutdown mysqld and asd, re-added and restarted asd, once it was finished I then started mysqld up and the directory on shm still doesn't exist.

Luckily this is a new machine so I haven't even imported the database yet due to the size was hoping to use asd to the advantage.


Windows get broken, penguins rarely get sucked into jet engines.

Offline

#182 2012-11-10 11:11:41

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

gehidore wrote:
drwx------ 6 mysql mysql 4096 Nov  7 18:17 mysql

That's the directory before starting asd.

I did just now shutdown mysqld and asd, re-added and restarted asd, once it was finished I then started mysqld up and the directory on shm still doesn't exist.

Luckily this is a new machine so I haven't even imported the database yet due to the size was hoping to use asd to the advantage.

@gehidore - You didn't show me anything useful there.

1) Stop asd.
2) Update to version 3.09 of asd (just released it).
3) Post the output of:

$ anything-sync-daemon debug

Last edited by graysky (2012-11-10 12:38:27)


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#183 2012-11-10 12:49:59

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

pezz wrote:

Adding /var/log causes systemd-journald to throw errors on the next reboot.

Not sure if anything can be done about this - assuming asd.service is shut down and rsyncs, then moves the directory, while journald is still running.

The recent changes I made to v3.09 of asd seems to allow one to sync /var/log (at least with my testing in a VM).  I am interested to see more feedback from people willing to test syncing /var/log using version >=3.09.  Please reply here with your experiences.  Only after some robust testing can I feel comfortable endorsing the recommendation again.

EDIT: Seems to recover from an ungraceful power down as well (again in the VM).

Last edited by graysky (2012-11-10 13:36:45)


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#184 2012-11-10 13:31:35

ZekeSulastin
Member
Registered: 2010-09-20
Posts: 266

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

people wrote:

systemd and /var/log onoz

Did any of you happen to try setting Storage=volatile (or none) in /etc/systemd/journald.conf so that journald doesn't actually do anything with /var/log/?

Offline

#185 2012-11-10 19:46:50

gehidore
IRC Op
From: Near the pine trees
Registered: 2012-11-09
Posts: 35

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

graysky wrote:

@gehidore - You didn'tw me anything useful there.

1) Stop asd.
2) Update to version 3.09 of asd (just released it).
3) Post the output of:

$ anything-sync-daemon debug

Sorry about that, did as you instructed and now it works. Output is such:

::DEBUG MODE::
anything-sync-daemon version: 3.09

settings in /etc/asd.conf will make anything-sync-daemon manage the following dir/user combos when invoked:

target to manage:  /var/lib/mysql
owner/group:       mysql:mysql
backup target:     /var/lib/mysql-backup_asd
link target:       /dev/shm/asd-mysql/var/lib/mysql

Windows get broken, penguins rarely get sucked into jet engines.

Offline

#186 2012-11-10 20:41:49

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

gehidore wrote:

target to manage:  /var/lib/mysql
owner/group:       mysql:mysql
backup target:     /var/lib/mysql-backup_asd
link target:       /dev/shm/asd-mysql/var/lib/mysql

Ok... And when you start the daemon, does the link target get created?  What about the backup target?


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#187 2012-11-11 05:33:01

gehidore
IRC Op
From: Near the pine trees
Registered: 2012-11-09
Posts: 35

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

graysky wrote:

Ok... And when you start the daemon, does the link target get created?  What about the backup target?

Yes and Yes, everything appears to work.

Thanks!

Ran the db query/replace last night against the onboard ssd took about 72 minutes, I'm curious to see the same query/replace again with the database loaded to shm.

-Beaver


Windows get broken, penguins rarely get sucked into jet engines.

Offline

#188 2012-11-11 06:17:08

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

gehidore wrote:

Yes and Yes, everything appears to work.

Thanks!

Ran the db query/replace last night against the onboard ssd took about 72 minutes, I'm curious to see the same query/replace again with the database loaded to shm.

Ok... so it asd works sync'ing mysql?  Do report the diff searching on your SSD vs RAM.


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#189 2012-11-11 07:09:37

gehidore
IRC Op
From: Near the pine trees
Registered: 2012-11-09
Posts: 35

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

graysky wrote:

Ok... so it asd works sync'ing mysql?  Do report the diff searching on your SSD vs RAM.


SSD is sata3 with xfs, ram is PC12800

Less than half the time, 1392.402372 seconds , err 23.2 minutes.

Last edited by gehidore (2012-11-11 07:13:34)


Windows get broken, penguins rarely get sucked into jet engines.

Offline

#190 2012-11-11 07:20:36

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

gehidore wrote:

SSD is sata3 with xfs, ram is PC12800

Less than half the time, 1392.402372 seconds , err 23.2 minutes.

Without asd: 72 min
With asd: 23 min

...over 3x faster smile


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#191 2012-11-11 07:26:06

gehidore
IRC Op
From: Near the pine trees
Registered: 2012-11-09
Posts: 35

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

graysky wrote:

Without asd: 72 min
With asd: 23 min

...over 3x faster smile

How I managed to think 25 was half of 70... it's too late for my brain to function I think.


Windows get broken, penguins rarely get sucked into jet engines.

Offline

#192 2012-11-25 06:21:56

gehidore
IRC Op
From: Near the pine trees
Registered: 2012-11-09
Posts: 35

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

I do have one thought that just hit me after transfering a bunch of files back and forth... Why isn't the backup directory hidden? I use tab so much I end up with random tabs while typing emails and such so it gets a little frustrating trying to move items about.

Just a thought.


Windows get broken, penguins rarely get sucked into jet engines.

Offline

#193 2012-12-11 22:19:58

teateawhy
Member
From: GER
Registered: 2012-03-05
Posts: 1,138
Website

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

graysky wrote:
pezz wrote:

Adding /var/log causes systemd-journald to throw errors on the next reboot.

Not sure if anything can be done about this - assuming asd.service is shut down and rsyncs, then moves the directory, while journald is still running.

The recent changes I made to v3.09 of asd seems to allow one to sync /var/log (at least with my testing in a VM).  I am interested to see more feedback from people willing to test syncing /var/log using version >=3.09.  Please reply here with your experiences.  Only after some robust testing can I feel comfortable endorsing the recommendation again.

EDIT: Seems to recover from an ungraceful power down as well (again in the VM).

Currently i have two different arch linux installations running this setup:

WHATTOSYNC=('/var/log' ...
VOLATILE="/dev/shm"

systemd/journald.conf

[Journal]
Storage=volatile

EDIT: asd is version 3.10-1

Both are running perfectly, one is a server another a laptop. I can also confirm my laptop survives the ungraceful powerdown, "last | grep crash" finds 4 crashs during the last month. In my opinion the key is configuring journald.conf not to touch /var/log and use /tmp instead, but systemd is indeed much like a black-box.
This seems to be in accordance with the manpage about journald.conf :

man journald.conf  wrote:

Storage=
           Controls where to store journal data. One of volatile, persistent, auto and none. If volatile journal log data will be
           stored only in memory[..] If persistent data will be stored [..] below the /var/log/journal hierarchy [...]

ZekeSulastin wrote:

Did any of you happen to try setting Storage=volatile (or none) in /etc/systemd/journald.conf so that journald doesn't actually do anything with /var/log/?

exactly

Last edited by teateawhy (2012-12-11 22:21:29)

Offline

#194 2012-12-11 22:24:17

gehidore
IRC Op
From: Near the pine trees
Registered: 2012-11-09
Posts: 35

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

I had been syncing my work directory + mysqld + http and after the cat unplugged the cord into the ac adapter and the batter died I find that I've lost more than 3 hours of work, luckily I had just pushed my work to github not 30 minutes before but, if it's syncing every hour by default why did 3 hours vanish?


Windows get broken, penguins rarely get sucked into jet engines.

Offline

#195 2012-12-12 00:47:44

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

gehidore wrote:

I had been syncing my work directory + mysqld + http and after the cat unplugged the cord into the ac adapter and the batter died I find that I've lost more than 3 hours of work, luckily I had just pushed my work to github not 30 minutes before but, if it's syncing every hour by default why did 3 hours vanish?

Did I have it set to 3 h before?  Thought 1 h was the default.


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#196 2012-12-12 01:09:38

gehidore
IRC Op
From: Near the pine trees
Registered: 2012-11-09
Posts: 35

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

Mine says 1 hour, not sure why or what unless it was a filesystem error.


Windows get broken, penguins rarely get sucked into jet engines.

Offline

#197 2012-12-12 17:42:01

gehidore
IRC Op
From: Near the pine trees
Registered: 2012-11-09
Posts: 35

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

For fun this morning I loaded up a bunch of 'work files' , simple script to modify them every 3 minutes and stop in three hours which my battery said 2 hours 40 minutes remaining. Let it run the battery down. I didn't lose more than the 1 hour of data this time. The big difference is I did recently switch from XFS to ext4 via tar ...


Windows get broken, penguins rarely get sucked into jet engines.

Offline

#198 2012-12-31 10:18:34

nierro
Member
From: Milan, Italy
Registered: 2011-09-02
Posts: 849

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

@gravsky: do you think it is useful to keep sync'ed my /home/user/.thumbnails dir? It is 30Mb and it may make my system faster (everything that loads thumbnails will load faster, i guess; e.g. menu will be showed faster, browsing in spacefm too will be faster...)
Am I missing anything or it can be a good idea?
Thanks for your work, and for your hint wink
Good year!

Offline

#199 2012-12-31 13:10:45

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

@nierro - You can ... but I don't see what you would gain by doing so unless you do LOTS of I/O to that directory.


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#200 2013-01-04 19:33:52

nierro
Member
From: Milan, Italy
Registered: 2011-09-02
Posts: 849

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

It was a bad idea, i guess smile
Thanks for pointing that out!

Offline

Board footer

Powered by FluxBB