anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

graysky · 2015-02-04 21:28:06

@gim - Just mount it in /etc/fstab if that's all you want (ie don't care about the data):

tmpfs /home/facade/.cache tmpfs noatime,nodev,nosuid,size=180M 0 0

Last edited by graysky (2015-02-04 21:28:19)

Soukyuu · 2015-02-04 22:33:05

Interesting. Syncing stuff to /tmp also seems to fix the problem I was having with qBittorrent not resuming correctly.

edit: I spoke too soon, it does not. Back to no overlay for me.

Last edited by Soukyuu (2015-02-08 12:21:44)

Minbari · 2015-03-08 19:48:07

Sometimes I recive this error when installing/removing packages:
e.g.

(1/1) loading package files                                                  [-------------------------------------------] 100%
(1/1) checking for file conflicts                                            [-------------------------------------------] 100%
error: failed to commit transaction (conflicting files)
audit: /var/log exists in filesystem
Errors occurred, no packages were upgraded.

asd.conf : WHATTOSYNC=('/var/log')

graysky · 2015-03-08 20:24:43

@minbari - Yes, known issue with the pacman 4.2 update: https://github.com/graysky2/anything-sy … /issues/19

Minbari · 2015-03-08 21:03:40

Looks like I have to stop/remove asd because it's cause me this error:

 kernel: BUG: Dentry ffff8800a2951540{i=3336,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:03 amy-jade kernel: BUG: Dentry ffff8800a29503c0{i=3335,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:03 amy-jade kernel: BUG: Dentry ffff880123c10840{i=3330,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:03 amy-jade kernel: BUG: Dentry ffff8800a29719c0{i=3328,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:03 amy-jade kernel: BUG: Dentry ffff880123c25480{i=3321,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:03 amy-jade kernel: BUG: Dentry ffff8800a290e480{i=32f6,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:03 amy-jade kernel: BUG: Dentry ffff8800a285d240{i=32ea,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:03 amy-jade kernel: BUG: Dentry ffff8800a2858a80{i=32dd,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:03 amy-jade kernel: BUG: Dentry ffff8800a285e840{i=3295,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:03 amy-jade kernel: BUG: Dentry ffff8800a2844780{i=3286,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:03 amy-jade kernel: BUG: Dentry ffff8800b2d7ce40{i=34fc,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:03 amy-jade kernel: BUG: Dentry ffff8800b2d7cd80{i=34fb,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:03 amy-jade kernel: BUG: Dentry ffff8800b2d7ccc0{i=34fa,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:04 amy-jade kernel: BUG: Dentry ffff8800b2d7cc00{i=34f9,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:04 amy-jade kernel: BUG: Dentry ffff8800b2d7cb40{i=34f8,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:04 amy-jade kernel: BUG: Dentry ffff8800b2d7ca80{i=34f7,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:04 amy-jade kernel: BUG: Dentry ffff8800b2d7c480{i=34f6,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:04 amy-jade kernel: BUG: Dentry ffff8800b2d7c600{i=34f5,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:04 amy-jade kernel: BUG: Dentry ffff8800b2d7c6c0{i=34f4,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:04 amy-jade kernel: BUG: Dentry ffff8800b2d7c9c0{i=34f3,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:04 amy-jade kernel: BUG: Dentry ffff8800b2d7c900{i=34f2,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:04 amy-jade kernel: BUG: Dentry ffff8800b2cb9240{i=310f,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:04 amy-jade kernel: BUG: Dentry ffff8800b2cb93c0{i=310e,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:04 amy-jade kernel: BUG: Dentry ffff8800b2cb9540{i=310d,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:04 amy-jade kernel: BUG: Dentry ffff8800b2cb9600{i=310c,n=#ffff8800b2c5c600}  still in use (1) [unmount of tmpfs t
Mar 08 22:47:04 amy-jade kernel: BUG: Dentry ffff8800b2cb96c0{i=0,n=user-1000.journal}  still in use (25) [unmount of tmpfs tmpfs

graysky · 2015-03-08 22:02:26

See the man page for open files... recommend not syncing /var/log

man asd wrote:

Currently, asd cannot handle open files on a sync target so if a hung process has something open there, it can be messy.

picard · 2015-03-11 08:06:21

graysky thank you for asd

ra1 · 2015-03-17 01:40:42

Thank you graysky for both, asd & psd. Your handy scripts save my flash drive

Lockheed · 2015-03-31 09:57:51

I am trying to get asd to handle /var/lib/mysql,

# anything-sync-daemon debug
Anything-sync-daemon v5.69 on Arch Linux

 Systemd service is currently active.
 Systemd resync service is currently active.
 Overlayfs v23 is currently active.

Asd will manage the following per /run/asd.conf settings:

 owner/group id:     mysql/89
 target to manage:   /var/lib/mysql
 sync target:        /var/lib/.mysql-backup_asd
 tmpfs target:       /tmp/asd-mysql/var/lib/mysql
 dir size:           617M
 overlayfs size:     0
 recovery dirs:      none

but mysqld does not like it:

Mar 31 11:43:54 nanoBox mysqld[6017]: 150331 11:43:54 [Warning] Can't create test file /var/lib/mysql/nanoBox.lower-test
Mar 31 11:43:54 nanoBox mysqld[6017]: [96B blob data]
Mar 31 11:43:54 nanoBox mysqld[6017]: 150331 11:43:54 [ERROR] Aborting
Mar 31 11:43:54 nanoBox mysqld[6017]: 150331 11:43:54 [Note] /usr/bin/mysqld: Shutdown complete
Mar 31 11:43:54 nanoBox systemd[1]: mysqld.service: main process exited, code=exited, status=1/FAILURE
Mar 31 11:45:00 nanoBox uwsgi[466]: Tue Mar 31 11:45:00 2015 - [uwsgi-cron] running "/usr/bin/php -f /usr/share/webapps/owncloud/cron.php 1>/dev/null" (pid 6725)
Mar 31 11:45:00 nanoBox uwsgi[466]: PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib/php/modules/geoip.so' - /usr/lib/php/modules/geoip.so: cannot open sh
Mar 31 11:45:01 nanoBox uwsgi[466]: [uwsgi-cron] command "/usr/bin/php -f /usr/share/webapps/owncloud/cron.php 1>/dev/null" running with pid 6725 exited after 1 second(s)
Mar 31 11:45:24 nanoBox systemd[1]: mysqld.service start-post operation timed out. Stopping.
Mar 31 11:45:24 nanoBox systemd[1]: Failed to start MariaDB database server.

However, if I first start mysqld and only after I run asd, it works fine.

I am using overlay, but had the same problem before I enabled it.

graysky · 2015-03-31 12:25:07

Can you try a test for me? Modify your /usr/lib/systemd/system/asd.service like this:

After=winbindd.service mysql.service

Make sure both your asd and mysql services are enabled and reboot. Did that fix it?

Lockheed · 2015-03-31 15:23:56

graysky wrote:

Can you try a test for me? Modify your /usr/lib/systemd/system/asd.service like this:
After=winbindd.service mysql.service
Make sure both your asd and mysql services are enabled and reboot. Did that fix it?

I will test it as soon as I get a chance. However, from the looks of it I am guessing it will solve the issue upon reboot, but won't help if I need to restart mysqld in the course of some maintenance, without reboot.

By the way - can you confirm whether my deduction regarding copying to RAM of data in use was correct?

Last edited by Lockheed (2015-03-31 15:24:54)

graysky · 2015-03-31 15:27:57

Lockheed wrote:

By the way - can you confirm whether my deduction regarding copying to RAM of data in use was correct?

Missed that... not sure what you're talking about.

Lockheed · 2015-03-31 17:23:07

graysky wrote:

Lockheed wrote:
By the way - can you confirm whether my deduction regarding copying to RAM of data in use was correct?
Missed that... not sure what you're talking about.

Looks like I did not save edits to my original post.
I observer that if a VDI image of a running VM is in RAM synced by ASD, I can freely stop and start ASD while the machine is running. Which would suggest that data can be moved to and from RAM even as it is being read/written by other programs (VirtualBox, mysql, etc). This observation was surprising to me and so I wanted to confirm it with you if it is correct.

graysky · 2015-03-31 17:52:19

Never tried it before...

Lockheed · 2015-04-01 21:38:13

@graysky, the reboot problem seems to be solved. However, I noticed another issue. ASD does not shut down properly when I issue "reboot" command as root. It looks like it just stops without waiting to syncronise its content to disk, because I get backup_asd-crashrecovery for a VM folder containing 274MB VM image.

EDIT:
Now that Ubuntu runs on systemd, would you consider adding Anything Sync Daemon to your ppa?

Last edited by Lockheed (2015-04-02 07:23:56)

Lockheed · 2015-04-23 05:50:58

I run into a strange issue that I had not have before. When using asd on /var/log, I cannot start php-fpm:

Apr 23 07:48:25 nanoBox php-fpm[1061]: [23-Apr-2015 07:48:25] ERROR: failed to open error_log (/var/log/php-fpm.log): No such file or directory (2)

However, the file is there:

# ls -l /var/log/php-fpm.log
-rw-r--r-- 1 root root 281 Apr 23 07:44 /var/log/php-fpm.log

asd preview says:

Anything-sync-daemon v5.69 on Arch Linux

 Systemd service is currently active.
 Systemd resync service is currently active.
 Overlayfs v23 is currently active.

Asd will manage the following per /run/asd.conf settings:

 owner/group id:     root/0
 target to manage:   /var/lib/monitorix
 sync target:        /var/lib/.monitorix-backup_asd
 tmpfs target:       /tmp/asd-root/var/lib/monitorix
 dir size:           91M
 overlayfs size:     50M
 recovery dirs:      none

 owner/group id:     root/0
 target to manage:   /var/log
 sync target:        /var/.log-backup_asd
 tmpfs target:       /tmp/asd-root/var/log
 dir size:           44M
 overlayfs size:     5.2M
 recovery dirs:      none

Soukyuu · 2015-06-17 22:16:35

Using 5.73, I noticed a weird behavior after moving my /home to a HDD. One of the folders I'm synching with asd is ~/.cache/chromium (since psd does not load it).
When starting chromium for the first time after boot, it looks as if the cache is loaded from HDD instead of the RAM copy - it gets stuck with "waiting for cache" on my custom start page.
I tried both with overlayfs and without, and still the same behavior. Any ideas?

edit: it kind of seems as if my cache is actually CLEARED on shutdown - the folder contains ~5MB of files atm but asd preview states:

owner/group id:     azure/1000
 target to manage:   /home/azure/.cache/chromium
 sync target:        /home/azure/.cache/.chromium-backup_asd
 tmpfs target:       /tmp/asd-azure/home/azure/.cache/chromium
 dir size:           147M
 recovery dirs:      none

I checked all three directories, and all of them have the same size. My cache used to be ~200MB, I don't know what happened to it.

edit2: definitely something wrong. My cache was growing, but after a few reboots it's back to 147MB. No crash-recovery dirs, but I was shutting it down via Plasma's logout menu anyway...

Last edited by Soukyuu (2015-06-17 22:52:18)

graysky · 2015-06-18 00:20:26

Not sure what to make of that... on a related note, how much RAM do you have? I have my ~/.cache mounted to tmpfs in /etc/fstab:

tmpfs /home/facade/.cache tmpfs noatime,nodev,nosuid,size=180M 0 0

Soukyuu · 2015-06-18 08:13:13

It's as if asd can't flush stuff to disk before the system shuts down, but it seems to only happen sometimes. That would explain my previous problem with qBittorrent randomly losing its progress, as well.
I got 12GB RAM on this machine, so it's plenty. I'd mount my whole .cache into tmpfs, but I actually want to keep it between reboots.

graysky · 2015-06-18 11:15:47

I'm not sure what is going on for you. Do you have any funky setup? Just a root partition on one disk and /home on another disk? Any encryption?

Soukyuu · 2015-06-18 11:22:32

No funky setups, I have an ext4 SSD as / and a separate ext4 HDD as /home/azure without any encryption. The rest are some NTFS partitions left from my windows installation.

All the weird behavior started when I moved my /home to the HDD, but I did "systemctl stop asd" before that, so I think it should have synched stuff to disk before I started moving stuff. Is there any user-specific or maybe a .lock file that I can delete to reset asd completely? Or is it only the /etc/asd.conf?

graysky · 2015-06-18 11:30:24

Just a pid file on /run and nothing in your /home should be there. Overlayfs?

Soukyuu · 2015-06-18 12:46:16

Nothing on run (I see psd though), and yes, overlayfs though my second edit was without it. Just to clarify, when using overlayfs, all data is still loaded to RAM, but only contents of the overlay are flushed to disk, right?

graysky · 2015-06-18 13:56:13

I missed that 2nd edit. Overlayfs is fastest than without it since the changes are what's stored. See the man page of asd in the FAQ.

Soukyuu · 2015-06-18 18:58:44

I asked because it seems that it's somehow trying to load data from the HDD instead - I can hear bursts of IO when starting chromium that only stop when it has completely started up. Since both the profile and the cache should have been already loaded to tmpfs, I don't see how that can happen. Not really familiar with linux tools, but is there one to check which process is currently accessing the HDD/which file?

Arch Linux

#276 2015-02-04 21:28:06

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#277 2015-02-04 22:33:05

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#278 2015-03-08 19:48:07

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#279 2015-03-08 20:24:43

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#280 2015-03-08 21:03:40

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#281 2015-03-08 22:02:26

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#282 2015-03-11 08:06:21

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#283 2015-03-17 01:40:42

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#284 2015-03-31 09:57:51

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#285 2015-03-31 12:25:07

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#286 2015-03-31 15:23:56

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#287 2015-03-31 15:27:57

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#288 2015-03-31 17:23:07

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#289 2015-03-31 17:52:19

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#290 2015-04-01 21:38:13

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#291 2015-04-23 05:50:58

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#292 2015-06-17 22:16:35

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#293 2015-06-18 00:20:26

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#294 2015-06-18 08:13:13

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#295 2015-06-18 11:15:47

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#296 2015-06-18 11:22:32

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#297 2015-06-18 11:30:24

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#298 2015-06-18 12:46:16

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#299 2015-06-18 13:56:13

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#300 2015-06-18 18:58:44

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

Board footer