anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

Soukyuu · 2014-08-08 23:01:40

What could be a reason for this:

Bad entry in your WHATTOSYNC array detected:
/home/userb/.cache
Edit /run/asd.conf correcting the mistake and try again.

The directory exists, it was copied straight from the ls command. The interesting thing is that it DOES get linked to /dev/shm and is also synched, but debug output still says it's a bad entry:

lrwxrwxrwx  1 userb userb    42 Aug  9 00:59 .cache -> /dev/shm/asd-userb/home/userb/.cache

Here's my /etc/asd.conf:

#
# /etc/asd.conf
#

WHATTOSYNC=('/home/usera/.cache' '/home/userb/.cache')
VOLATILE="/dev/shm"

Last edited by Soukyuu (2014-08-09 00:28:16)

graysky · 2014-08-08 23:19:57

If you stop asd from systemd, are all of the three sync targets physical directories? Particularly /home/userb/.cache ?

Soukyuu · 2014-08-08 23:24:57

Stopped asd.service, all directories return to being normal ones. Including

drwxr-xr-x  4 userb userb  4096 Aug  9 00:49 .cache

Last edited by Soukyuu (2014-08-08 23:26:21)

graysky · 2014-08-08 23:29:12

Hmmm... and if you run `asd p` while the systemd service is inactive?

Soukyuu · 2014-08-08 23:32:49

Stopped service, ran 'asd p', same thing

# asd p
Bad entry in your WHATTOSYNC array detected:
/home/userb/.cache
Edit /etc/asd.conf correcting the mistake and try again.

edit: if I run it as "sudo asd p", then it correctly displays that directory. When enabling/starting the service, I used sudo as well. Does that starts/enables the service as my account? Should I be doing this as root instead?
edit2: ooooh I feel so stupid... executing "anything-sync-daemon debug" as usera will throw that error because it doesn't have permission to access userb's folder... sorry, I shouldn't be doing this when I'm tired. Thanks for the great tool!

Last edited by Soukyuu (2014-08-08 23:38:01)

graysky · 2014-08-08 23:49:31

Glad it's not my code Enjoy!

Soukyuu · 2014-08-09 00:43:53

Hmm one more question. The wiki tells me to use crontab to change the sync interval, but I don't have it, nor any cron package installed. According to the cron wiki entry, in that case systemd's timer functionality is used.
There seem to be an asd-resync.timer registered (systemctl list-timers), but it's not located in /etc/systemd/system/ as in that example. Where do I find the timer unit? Does the time interval wiki section for asd need an update?

graysky · 2014-08-09 08:24:10

I updated the wiki... no cron support is needed or recommended on Arch.

berot3 · 2015-01-27 13:17:15

Hi,
I tried to sync dokuwiki, up to now I thought it did, but after some updates I get a blank white site accessing dokuwiki.php...

I try to

WHATTOSYNC=(
        '/srv/http'
        '/var/lib/dokuwiki'
        '/usr/share/webapps/dokuwiki'
)

Without

'/usr/share/webapps/dokuwiki'

everything works just fine.

I also added it to php.ini:

open_basedir = /srv/http/:/home/:/tmp/:/usr/share/pear/:/usr/share/webapps/:/etc/webapps/dokuwiki/:/var/lib/dokuwiki/:/dev/shm/asd-http/usr/dokuwiki/

Ans also made everything chmod +x as well as chown http:http.

Any ideas?
Or would you say that there are performance gains and I can leave it out?

Thanks!

Soukyuu · 2015-01-27 13:46:37

I'm also apparently having a problem with qBittorrent. When I sync the ~/.config/qBittorrent and ~/.local/share/data/qBittorrent/BT_backup directories, restarting qB makes it access the previous version of resume information.
Is it possible that it's somehow able to see "through" the overlay?

graysky · 2015-01-27 14:42:02

@berot3 - Never used that software before (no idea how much IO that target gets) so I don't have an opinion for you.
@Suokyuu - Not sure what you're asking about seeing through the overlay? Does qbitt work as expected without using the overlayfs mode?

probspot · 2015-01-27 15:48:30

Hello,

when trying to install cups (while having anything-sync-daemon running and syncing '/var/log') I encountered the following issue:

Pacman will fail when checking for the existence of /var/log:

error: failed to commit transaction (conflicting files)
cups: /var/log exists in filesystem

The reason for this is that /var/log is technically speaking not a directory but a symlink. Now, pacman's behavior was changed recently to complain about the directory symlinks. See here:

http://allanmcrae.com/2014/12/pacman-4-2-released/ (search for "Directory Symlink Handling"). There is a recommendation there to use bind mounts for cases like those.

Also, in the discussion here: https://github.com/graysky2/anything-sy … n/issues/3, there's also a request for bind mounts.

A per-path configuration flag would be an ideal solution, for example:

WHATTOSYNC=(
('/var/lib/monitorix' 'bind')
('/srv/http' 'symlink')
('/foo/bar' 'bind')
)

The workaround for now is:

1) Stop asd
2) Install misbehaving package
3) Start asd

Last edited by probspot (2015-01-27 15:49:37)

graysky · 2015-01-27 16:00:21

Thanks for pointing this out. For the specific case of dirs where pacman is expected to write out files, bind mounts will need to be used in asd to prevent this behavior since version 4.2 of pacman itself. This is currently an open issue: https://github.com/graysky2/anything-sy … /issues/19

berot3 · 2015-01-27 16:35:44

graysky wrote:

@berot3 - Never used that software before (no idea how much IO that target gets) so I don't have an opinion for you.

Thank you for your help graysky.
I found the error. dokuwiki had broken symlinks when it was moved to dev/shm. I had to replace the following to make it work with asd (basically they all needed 2 extra "../"):

cd /dev/shm/asd-http/usr/share/webapps/dokuwiki
sudo ln -s --force ../../../../../../../var/lib/dokuwiki/data        # original: ../../../../../var/lib/dokuwiki/data
sudo ln -s --force ../../../../../../../etc/webapps/dokuwiki conf    # original: ../../../../../etc/webapps/dokuwiki

cd /dev/shm/asd-http/usr/share/webapps/dokuwiki/lib
sudo ln -s --force ../../../../../../../var/lib/dokuwiki/plugins  # original: ../../../../../var/lib/dokuwiki/plugins
sudo ln -s --force ../../../../../../../var/lib/dokuwiki/tpl      # original: ../../../../../var/lib/dokuwiki/tpl

Would be great if there were a permanent fix because with every dokuwiki update the old symlinks will be restored...

Last edited by berot3 (2015-01-27 16:37:57)

Soukyuu · 2015-01-27 18:34:52

graysky wrote:

@Suokyuu - Not sure what you're asking about seeing through the overlay? Does qbitt work as expected without using the overlayfs mode?

So far I could only test it without asd at all. It seems like with asd, if there is a state_0 that was saved to disk and qB saves resume state_1 to the overlay, after restart it somehow reads state_0 instead of state_1, as if it was reading from disk instead of from the overlay.

I'll report back if it also happens with overlay option turned off, but I don't remember this happening with the older versions of asd. What is the difference between 'bind' and 'symlink'?

graysky · 2015-01-27 19:00:40

@berot3 - Not sure why the symlinks were broken... perhaps once I consider and implement the bind mount alternative, this won't be an issue.
@Soukyuu - I am unfamiliar with that software and the state_0 and state_1 you're referencing. The issue with bind and symlink is that pacman 4.2 will not install to a symlink, but it will to a bind mount. I can't explain the technical differences between the two, but google knows.

berot3 · 2015-01-28 12:33:21

graysky wrote:

@berot3 - Not sure why the symlinks were broken... perhaps once I consider and implement the bind mount alternative, this won't be an issue.

symlinks were broken, because they are relative links which first went up 5 times and then to the target. Now when asd relocates the data to /dev/shm than the new directory is two levels deeper than before. Therefore I had to add two additional "../" at the beginning. Probably dokuwiki devs made this to be portable. If they used absolute links instead then I would have no issues with asd. I hope I could explain it better this time?

could I simply try the bind-method by removing the links that asd made and mount it with bind instead? I could give it a try.

Soukyuu · 2015-01-29 16:10:43

graysky wrote:

@Soukyuu - I am unfamiliar with that software and the state_0 and state_1 you're referencing.

That's just my (failed) attempt to describe the issue.
Basically, the torrent client tracks the torrents loaded and their progress and writes it to disk every few minutes.

So let's say state0 is

torrentA torrentB and torrentC completed

This is what's currently on the disk when I start qB. I then load torrentD, it finishes downloading and qB writes the resume information (state1) as

torrentA torrentB torrentC and torrentD completed

.
This state of the resum file will be on the overlay FS until it's synched to the disk, right?
The issue seems to be that qB somehow loads the resume file at state0 (=on the disk) instead of the up-to-date state1 (=on the overlay). This results in it thinking that torrentD is in the list, but has not completed, resulting in re-downloading of the file.

Disabling the overlayfs solves it, at least I didn't have torrents resetting themselves anymore after I disabled the overlay. I hope this makes things clearer, I really suck at explaining.

graysky · 2015-01-29 19:12:31

It makes sense now, thanks for the explanation. I do not know why the overlay causes this behavior though. My understanding of the technology is that it is transparent. Perhaps in the specific case, there is another mount option required that I am not invoking? Any chance you can ask upstream and see what they say?

Soukyuu · 2015-01-30 00:08:13

I've opened an issue on the github tracker here, let's see what the devs say.

gim · 2015-02-04 19:47:33

Hello! Recently I decided to try ASD, but since I launched this daemon I noticed that it makes systemd-hostnamed.service and systemd-localed.service fail to start at all. Here are some outputs:

$ systemctl --state=failed

  UNIT                      LOAD   ACTIVE SUB    DESCRIPTION
● systemd-hostnamed.service loaded failed failed Hostname Service
● systemd-localed.service   loaded failed failed Locale Service

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

$ systemctl status systemd-hostnamed.service

● systemd-hostnamed.service - Hostname Service
   Loaded: loaded (/usr/lib/systemd/system/systemd-hostnamed.service; static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2015-02-04 18:30:16 EET; 2h 57min ago
     Docs: man:systemd-hostnamed.service(8)
           man:hostname(5)
           man:machine-info(5)
           http://www.freedesktop.org/wiki/Software/systemd/hostnamed
  Process: 679 ExecStart=/usr/lib/systemd/systemd-hostnamed (code=exited, status=226/NAMESPACE)
 Main PID: 679 (code=exited, status=226/NAMESPACE)

Feb 04 18:30:16 V5-573G-Arch systemd[679]: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-hostnamed: Invalid argument
Feb 04 18:30:16 V5-573G-Arch systemd[1]: systemd-hostnamed.service: main process exited, code=exited, status=226/NAMESPACE
Feb 04 18:30:16 V5-573G-Arch systemd[1]: Failed to start Hostname Service.
Feb 04 18:30:16 V5-573G-Arch systemd[1]: Unit systemd-hostnamed.service entered failed state.
Feb 04 18:30:16 V5-573G-Arch systemd[1]: systemd-hostnamed.service failed.

# systemctl start systemd-localed.service

Job for systemd-localed.service failed. See "systemctl status systemd-localed.service" and "journalctl -xe" for details.

Only when I stop asd.service I can start systemd-localed.service successfully. I couldn't find anything related to this in the ArchWiki page. What is happening?

Here are options in my /etc/asd.conf file (VOLATILE I left commented there):

WHATTOSYNC=('/home/gim/.cache' '/home/gim/.adobe')
#VOLATILE="/dev/shm"
USE_OVERLAYFS="yes"
USE_BACKUPS="no"

graysky · 2015-02-04 20:36:01

@gim - Not sure what's going on with that, but others have found that disabling the overlayfs gives the expected behavior. I cannot reproduce the error on my system. I am grateful if you are able to contribute to troubleshooting. See this issue open on github and feel free to post to it there.

Last edited by graysky (2015-02-04 20:36:15)

gim · 2015-02-04 21:15:46

graysky, thanks for the quick reply! I actually found how to fix this on my system and posted in that issue you linked. It's related to the VOLATILE path combined with overlayfs I suppose. If you need some testing, I'll be glad to help you, since you can't reproduce it on your system.

Last edited by gim (2015-02-04 21:18:35)

graysky · 2015-02-04 21:19:10

That is an interesting find... the only danger about using /tmp is documented in the man page:

man asd wrote:

*Note that the default value of "/dev/shm" should work just fine for the VOLATILE setting. Be aware that using software such as bleachbit with asd can be dangerous since bleachbit likes to remove files stored in /tmp. This is why a value of /dev/shm is recommended.

gim · 2015-02-04 21:21:52

graysky, yeah, I noticed that. Though I don't use bleachbit, so it's not a danger to me And I just want to move my cache to the tmpfs, so I don't care about loosing any data.

Last edited by gim (2015-02-04 21:23:20)

Arch Linux

#251 2014-08-08 23:01:40

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#252 2014-08-08 23:19:57

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#253 2014-08-08 23:24:57

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#254 2014-08-08 23:29:12

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#255 2014-08-08 23:32:49

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#256 2014-08-08 23:49:31

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#257 2014-08-09 00:43:53

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#258 2014-08-09 08:24:10

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#259 2015-01-27 13:17:15

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#260 2015-01-27 13:46:37

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#261 2015-01-27 14:42:02

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#262 2015-01-27 15:48:30

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#263 2015-01-27 16:00:21

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#264 2015-01-27 16:35:44

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#265 2015-01-27 18:34:52

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#266 2015-01-27 19:00:40

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#267 2015-01-28 12:33:21

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#268 2015-01-29 16:10:43

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#269 2015-01-29 19:12:31

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#270 2015-01-30 00:08:13

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#271 2015-02-04 19:47:33

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#272 2015-02-04 20:36:01

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#273 2015-02-04 21:15:46

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#274 2015-02-04 21:19:10

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#275 2015-02-04 21:21:52

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

Board footer