anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

graysky · 2012-08-29 22:58:17

Interesting... I don't think it's a bug. Don't try to rm /usr/bin/rm I guess ;P

MFserver · 2012-08-29 23:10:39

Good point!

This little maneuver almost killed my system! I guess the best thing I could do to safeguard this machine is block myself from the root account, because I just might have screwed up when copying files back from /usr/bin-backup_asd... pacman -Syu helped, though.

pezz · 2012-08-31 15:57:17

Adding /var/log causes systemd-journald to throw errors on the next reboot.

Not sure if anything can be done about this - assuming asd.service is shut down and rsyncs, then moves the directory, while journald is still running.

graysky · 2012-08-31 17:44:37

I don't think it's wise to link the entire directory, plus I am not sure how this would affect an init system such as systemd.

pezz · 2012-08-31 22:15:39

Then /var/log shouldn't be recommended in the comments in the asd.conf file.

Works great otherwise though, very nice.

graysky · 2012-08-31 22:37:14

<Headsmack> ... thanks for point that out. Fixed.

Doorbreath · 2012-09-05 16:38:04

I don't understand how it saves ssd writes when it syncs everything to the ssd every hour?

graysky · 2012-09-05 19:48:03

Doorbreath wrote:

I don't understand how it saves ssd writes when it syncs everything to the ssd every hour?

1) There is no I/O to physical disks when you are read/writing to RAM. So I/O intensive tasks are routed to memory rather than to disks.
2) Not everything gets sync'ed back -- just what has changed since the last sync. For example, if only 0.3 MB of your 100+ MB data changed, then only 0.3 MB gets synced over.

Last edited by graysky (2012-09-05 20:44:48)

Doorbreath · 2012-09-06 12:02:57

What if something is writing to the fs at the exact time anything-sync-daemon runs? Problems?

graysky · 2012-09-07 00:36:30

@door - I don't think so... the kernel can schedule I/O traffic just fine.

DarioP · 2012-09-15 15:40:02

I'm running physical simulation on great variety of parameters. Each step of the billions of the whole simulation has some logging I/O to the disk. Together with the ext4 journaling this was resulting in a low loaded CPU and a massively loaded HDD . I was thinking to move to an SSD, then I thought to move folders to the RAM.

I found this program and now my cpu il 100% loaded, my disk is quiet and simulation is running ~3 times faster!! Well done, graysky!!

Here are my perplexities about it:
* Do I really have to stop and restart the daemon if I want to sync now? Moreover $ rc.d sync asd doesn't work for me.
* Do I really have to enter my root password each time I want to do this?
* Do I really have to stop the daemon, modify the conf file as admin, and restart the daemon if I just want to sync the pwd?

It is a great tool but I think it should be made a little bit more flexible and user friendly (not only admin friendly)

graysky · 2012-09-15 20:39:27

1) You can manually trigger a sync by running `sudo /usr/bin/anything-sync-daemon sync`
1b) Right, there are two tokens for the rc.d: start and stop. I just added a third in version 3.5 for just for you: sync.
2) Learn about sudo.
3) No idea what you're doing here. The password is that of your root user and the script doesn't care what that password is other than sudo clearing it to run. I think you're confused. See #2
4) I don't know how it can be any more user friendly than it already is... this is really a very simplistic shell script.

Last edited by graysky (2012-09-15 20:55:45)

DarioP · 2012-09-15 21:13:33

1,2,3) Let me better explain. I have some long scripts which call a variety of tools to complete the task. At the end of these script it would be nice to place a command: sync_now and this should not require me to enter the sudo pass. That's because I may start one script for each core and go for lunch. At the end of each script the sync should be immediate and automatic. I may run the full script as admin.. but I'm sure we can find a better solution!

4) For instance adding commands:
move_to_ram args
list_in_ram_items
move_from_ram args
or something like this to easily access the WHATTOSYNC variable. Surely tomorrow I wont remember where this variable is defined!
Those are things that I may implement with a frontend script which acts on your configuration file, then stop and start the daemon, but again the problem of the annoying sudo password. Moreover I don't want to stop the daemon if some programs are running on the synced files and I just want to add other files to sync.
I think that I should be able to ask the system to move something to and from ram just as user. Don't you think so? Do you think it's possible?

I hope I've been clearer. Your script is perfect to sync in ram logs, caches and other static stuffs, but a daily use of it syncing now this and now that is a bit prohibitive

graysky · 2012-09-15 22:03:25

Just add permission to execute the rc.d daemon without a password. The syntax is explained in /etc/sudoers via comments. You can likely add a line at the end of your last script that simply calls the rc.d file via sudo and call it good.

Mr Green · 2012-09-16 05:22:06

This may possibly help

https://wiki.archlinux.org/index.php/Su … le_Entries

DarioP · 2012-09-16 08:39:00

Yes, thanks to both. I studied the details of sudo much time ago and I had forgotten!

Just a couple of unaswered questions:
1) Sync command starts and stops the daemon deleting and recreating all the links? This would be quite unusable because I may be working on some of them.
2) I would like to be able to link (and unlink) files and folders to the ram without restart the daemon (see 1). Is this possible? Will it be in future?

Thank you

Mr Green · 2012-09-16 09:20:15

2) No expert but it sounds like you need ramdisk set up.

I have tried the other brand of daemon in the past with firefox and ran into some problems with add-ons, does asd support firefox from ram?

Or would it be better to sync cache or thumbnails?

systemd-analyze blame
  7970ms asd.service
  3600ms NetworkManager.service
  2083ms systemd-modules-load.service

Last edited by Mr Green (2012-09-16 09:24:14)

graysky · 2012-09-16 10:05:53

DarioP wrote:

1) Sync command starts and stops the daemon deleting and recreating all the links? This would be quite unusable because I may be working on some of them.
2) I would like to be able to link (and unlink) files and folders to the ram without restart the daemon (see 1). Is this possible? Will it be in future?

You are mistaken. Look at the rc.d file which simply calls the script's 'sync' function. There is nothing about unlinking nor about stopping/starting.

/etc/rc.d/asd

...
  sync)
    stat_busy 'Doing a user requested sync'
    if [[ -f $DAEMON_FILE ]]; then
      /usr/bin/anything-sync-daemon sync
    else
      stat_die
    fi
    stat_done
    ;;
...

/usr/bin/anything-sync-daemon

sync() {
  touch "$DAEMON_FILE"

  # sync to tmpfs and back again
  local DIR USER BACKUP LINK
  for DIR in "${WHATTOSYNC[@]}"; do
    BACKUP="$DIR-backup_asd"  # this is the hdd bound backup in case of power failure
    if [[ -d "$BACKUP" ]]; then
      USER=$(/usr/bin/stat -c %U "$BACKUP")
    else
      USER=$(/usr/bin/stat -c %U "$DIR")
    fi
    LINK="$VOLATILE/asd-$USER$DIR"

    # make tmpfs container
    if [[ -d "$DIR" ]]; then
      [[ -r "$LINK" ]] || su -c "install -dm$PREFIX_PERMISSIONS '$LINK'" $USER

      # backup target and link to tmpfs container
      if [[ "$(readlink "$DIR")" != "$LINK" ]]; then
        mv "$DIR" "$BACKUP"
        ln -s "$LINK" "$DIR"
      fi

      # sync the tmpfs targets to the disc
      if [[ -e $LINK/.flagged ]]; then
        su -c "rsync -a --delete --exclude .flagged '$LINK/' '$BACKUP/'" $USER
      else
        su -c "rsync -a '$BACKUP/' '$LINK/'" $USER
        su -c "touch '$LINK/.flagged'" $USER
      fi
    fi
  done
}

DarioP · 2012-09-16 10:25:05

I was just asking to be sure of the behaviour of this tool.

If I have one file being modified and another finished and to be synced I can call the sync routine without any worries as I understood. Moreover I can add folders in WHATTOSYNC variable and sync them with a sync call without affect the files being modified. If this is true, now I'm definitely happy!

Probably I just got confused by what's written in the wiki:

Using SysVinit (Arch default)
Call the daemon to start, stop, or sync (restart) like any standard Arch daemon:
# rc.d command asd

When I tried sync it was not working so this led me to think that sync needed a manual stop and restart

Last edited by DarioP (2012-09-16 10:31:58)

graysky · 2012-09-16 11:29:04

@DarioP - Ok... I will reword the wiki. Can you verify that the latest version of asd satisfies all of your needs?

thewall · 2012-09-17 22:55:46

Hi everybody.

I used anything-sync-daemon with a SSD drive and, as suggested on this page of the Archwiki, I synced my /var/log directory.
As I read in this thread, that was not a really good idea.
I got errors a couple of days ago about pacman database, but I fixed them. Today my root partition fucked up with errors on superblock: I managed to clone the partition on the HDD and I've been able to boot.
So, I blacklisted asd from /etc/rc.conf, but sometimes (not everytime) at boot I get errors like

Cannot create /var/log/xxx: file or directory doesn't exist

and X can't start cause it respawn to fast.
Luckily I can login from tty2 and solve the problem: to do that I need to start asd, stop it and reboot. Please notice that the daemon was not in /etc/rc.conf (it was not started)!!!

Do you have any idea how can I solve this?

Ps: Thanks a lot for your HUGE work!

Pps: you should also reword the Archwiki page I mentioned before

graysky · 2012-09-17 23:40:15

@thewall - Thanks for the info. The docs and the asd wiki have been edited to include a nice warning. I just edited the SSD page as well totally removing the subsection.

An oversight on my part is why it started without the deamon running; this has been corrected in version 3.6 now in the AUR and on github. Basically, cron would trigger a sync once per hour under the old version. Now, cron is aware of the pid file, and checks to that is present before doing anything.

Please update to v3.6 and let me know if you experience any problems. I'm glad you're savvy enough to keep your ship above the waterline

Last edited by graysky (2012-09-17 23:42:50)

thewall · 2012-09-18 00:01:56

Hey, thanks for your reply.

I updated right now to version 3.6. What should I do? Start and stop the daemon and then reboot?

EDIT: I did it and now I shutdown. Here is 2 AM I'll let you know.

Last edited by thewall (2012-09-18 00:11:07)

graysky · 2012-09-18 00:10:49

No need to reboot at all. Actually, in this case, no need to restart the daemon either. The changes are to the underlying script and crontab for hourly.

Mr Green · 2012-09-18 04:45:01

Been running asd with systemd for a few days now, on some boots I was dropped to console log in instead of silm, last night system would not power down correctly. Loaded 3.6 see what happens...

Arch Linux

#101 2012-08-29 22:58:17

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#102 2012-08-29 23:10:39

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#103 2012-08-31 15:57:17

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#104 2012-08-31 17:44:37

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#105 2012-08-31 22:15:39

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#106 2012-08-31 22:37:14

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#107 2012-09-05 16:38:04

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#108 2012-09-05 19:48:03

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#109 2012-09-06 12:02:57

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#110 2012-09-07 00:36:30

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#111 2012-09-15 15:40:02

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#112 2012-09-15 20:39:27

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#113 2012-09-15 21:13:33

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#114 2012-09-15 22:03:25

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#115 2012-09-16 05:22:06

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#116 2012-09-16 08:39:00

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#117 2012-09-16 09:20:15

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#118 2012-09-16 10:05:53

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#119 2012-09-16 10:25:05

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#120 2012-09-16 11:29:04

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#121 2012-09-17 22:55:46

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#122 2012-09-17 23:40:15

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#123 2012-09-18 00:01:56

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#124 2012-09-18 00:10:49

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

#125 2012-09-18 04:45:01

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

Board footer