You are not logged in.

#101 2012-08-29 22:58:17

graysky
Member
From: The worse toilet in Scotland
Registered: 2008-12-01
Posts: 8,649
Website

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

Interesting... I don't think it's a bug.  Don't try to rm /usr/bin/rm I guess ;P


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#102 2012-08-29 23:10:39

MFserver
Member
From: Sweden
Registered: 2011-10-28
Posts: 43
Website

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

Good point! wink

This little maneuver almost killed my system! I guess the best thing I could do to safeguard this machine is block myself from the root account, because I just might have screwed up when copying files back from /usr/bin-backup_asd... pacman -Syu helped, though.


Sometimes, when I'm trying to get any audio software or hardware working with my system, I wonder why I ditched Windows. But every time I work at a windows computer, I remember it again.

Offline

#103 2012-08-31 15:57:17

pezz
Member
From: Geelong, Australia
Registered: 2010-05-23
Posts: 49

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

Adding /var/log causes systemd-journald to throw errors on the next reboot.

Not sure if anything can be done about this - assuming asd.service is shut down and rsyncs, then moves the directory, while journald is still running.

Offline

#104 2012-08-31 17:44:37

graysky
Member
From: The worse toilet in Scotland
Registered: 2008-12-01
Posts: 8,649
Website

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

I don't think it's wise to link the entire directory, plus I am not sure how this would affect an init system such as systemd.


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#105 2012-08-31 22:15:39

pezz
Member
From: Geelong, Australia
Registered: 2010-05-23
Posts: 49

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

Then /var/log shouldn't be recommended in the comments in the asd.conf file. smile

Works great otherwise though, very nice.

Offline

#106 2012-08-31 22:37:14

graysky
Member
From: The worse toilet in Scotland
Registered: 2008-12-01
Posts: 8,649
Website

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

<Headsmack> ... thanks for point that out.  Fixed.


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#107 2012-09-05 16:38:04

Doorbreath
Member
Registered: 2010-06-08
Posts: 20

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

I don't understand how it saves ssd writes when it syncs everything to the ssd every hour?

Offline

#108 2012-09-05 19:48:03

graysky
Member
From: The worse toilet in Scotland
Registered: 2008-12-01
Posts: 8,649
Website

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

Doorbreath wrote:

I don't understand how it saves ssd writes when it syncs everything to the ssd every hour?

1) There is no I/O to physical disks when you are read/writing to RAM.  So I/O intensive tasks are routed to memory rather than to disks.
2) Not everything gets sync'ed back -- just what has changed since the last sync.  For example, if only 0.3 MB of your 100+ MB data changed, then only 0.3 MB gets synced over.

Last edited by graysky (2012-09-05 20:44:48)


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#109 2012-09-06 12:02:57

Doorbreath
Member
Registered: 2010-06-08
Posts: 20

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

What if something is writing to the fs at the exact time anything-sync-daemon runs? Problems?

Offline

#110 2012-09-07 00:36:30

graysky
Member
From: The worse toilet in Scotland
Registered: 2008-12-01
Posts: 8,649
Website

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

@door - I don't think so... the kernel can schedule I/O traffic just fine.


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#111 2012-09-15 15:40:02

DarioP
Member
From: Geneva
Registered: 2011-01-05
Posts: 110

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

I'm running physical simulation on great variety of parameters. Each step of the billions of the whole simulation has some logging I/O to the disk. Together with the ext4 journaling this was resulting in a low loaded CPU and a massively loaded HDD . I was thinking to move to an SSD, then I thought to move folders to the RAM.

I found this program and now my cpu il 100% loaded, my disk is quiet and simulation is running ~3 times faster!! Well done, graysky!!

Here are my perplexities about it:
* Do I really have to stop and restart the daemon if I want to sync now? Moreover $ rc.d sync asd doesn't work for me.
* Do I really have to enter my root password each time I want to do this?
* Do I really have to stop the daemon, modify the conf file as admin, and restart the daemon if I just want to sync the pwd?

It is a great tool but I think it should be made a little bit more flexible and user friendly (not only admin friendly) wink

Offline

#112 2012-09-15 20:39:27

graysky
Member
From: The worse toilet in Scotland
Registered: 2008-12-01
Posts: 8,649
Website

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

1) You can manually trigger a sync by running `sudo /usr/bin/anything-sync-daemon sync`
1b) Right, there are two tokens for the rc.d: start and stop.  I just added a third in version 3.5 for just for you: sync.
2) Learn about sudo.
3) No idea what you're doing here.  The password is that of your root user and the script doesn't care what that password is other than sudo clearing it to run.  I think you're confused.  See #2 tongue
4) I don't know how it can be any more user friendly than it already is... this is really a very simplistic shell script.

Last edited by graysky (2012-09-15 20:55:45)


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#113 2012-09-15 21:13:33

DarioP
Member
From: Geneva
Registered: 2011-01-05
Posts: 110

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

1,2,3) Let me better explain. I have some long scripts which call a variety of tools to complete the task. At the end of these script it would be nice to place a command: sync_now and this should not require me to enter the sudo pass. That's because I may start one script for each core and go for lunch. At the end of each script the sync should be immediate and automatic. I may run the full script as admin.. but I'm sure we can find a better solution!

4) For instance adding commands:
move_to_ram args
list_in_ram_items
move_from_ram args
or something like this to easily access the WHATTOSYNC variable. Surely tomorrow I wont remember where this variable is defined!
Those are things that I may implement with a frontend script which acts on your configuration file, then stop and start the daemon, but again the problem of the annoying sudo password. Moreover I don't want to stop the daemon if some programs are running on the synced files and I just want to add other files to sync.
I think that I should be able to ask the system to move something to and from ram just as user. Don't you think so? Do you think it's possible?

I hope I've been clearer. Your script is perfect to sync in ram logs, caches and other static stuffs, but a daily use of it syncing now this and now that is a bit prohibitive wink

Offline

#114 2012-09-15 22:03:25

graysky
Member
From: The worse toilet in Scotland
Registered: 2008-12-01
Posts: 8,649
Website

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

Just add permission to execute the rc.d daemon without a password.  The syntax is explained in /etc/sudoers via comments.  You can likely add a line at the end of your last script that simply calls the rc.d file via sudo and call it good.


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#115 2012-09-16 05:22:06

Mr Green
Forum Fellow
From: U.K.
Registered: 2003-12-21
Posts: 5,752

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed


Mr Green

Offline

#116 2012-09-16 08:39:00

DarioP
Member
From: Geneva
Registered: 2011-01-05
Posts: 110

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

Yes, thanks to both. I studied the details of sudo much time ago and I had forgotten! tongue

Just a couple of unaswered questions:
1) Sync command starts and stops the daemon deleting and recreating all the links? This would be quite unusable because I may be working on some of them.
2) I would like to be able to link (and unlink) files and folders to the ram without restart the daemon (see 1). Is this possible? Will it be in future?

Thank you smile

Offline

#117 2012-09-16 09:20:15

Mr Green
Forum Fellow
From: U.K.
Registered: 2003-12-21
Posts: 5,752

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

2) No expert but it sounds like you need ramdisk set up.

I have tried the other brand of daemon in the past with firefox and ran into some problems with add-ons, does asd support firefox from ram?

Or would it be better to sync cache or thumbnails?

systemd-analyze blame
  7970ms asd.service
  3600ms NetworkManager.service
  2083ms systemd-modules-load.service

Last edited by Mr Green (2012-09-16 09:24:14)


Mr Green

Offline

#118 2012-09-16 10:05:53

graysky
Member
From: The worse toilet in Scotland
Registered: 2008-12-01
Posts: 8,649
Website

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

DarioP wrote:

1) Sync command starts and stops the daemon deleting and recreating all the links? This would be quite unusable because I may be working on some of them.
2) I would like to be able to link (and unlink) files and folders to the ram without restart the daemon (see 1). Is this possible? Will it be in future?

You are mistaken.  Look at the rc.d file which simply calls the script's 'sync' function.  There is nothing about unlinking nor about stopping/starting.

/etc/rc.d/asd

...
  sync)
    stat_busy 'Doing a user requested sync'
    if [[ -f $DAEMON_FILE ]]; then
      /usr/bin/anything-sync-daemon sync
    else
      stat_die
    fi
    stat_done
    ;;
...

/usr/bin/anything-sync-daemon

sync() {
  touch "$DAEMON_FILE"

  # sync to tmpfs and back again
  local DIR USER BACKUP LINK
  for DIR in "${WHATTOSYNC[@]}"; do
    BACKUP="$DIR-backup_asd"  # this is the hdd bound backup in case of power failure
    if [[ -d "$BACKUP" ]]; then
      USER=$(/usr/bin/stat -c %U "$BACKUP")
    else
      USER=$(/usr/bin/stat -c %U "$DIR")
    fi
    LINK="$VOLATILE/asd-$USER$DIR"

    # make tmpfs container
    if [[ -d "$DIR" ]]; then
      [[ -r "$LINK" ]] || su -c "install -dm$PREFIX_PERMISSIONS '$LINK'" $USER

      # backup target and link to tmpfs container
      if [[ "$(readlink "$DIR")" != "$LINK" ]]; then
        mv "$DIR" "$BACKUP"
        ln -s "$LINK" "$DIR"
      fi

      # sync the tmpfs targets to the disc
      if [[ -e $LINK/.flagged ]]; then
        su -c "rsync -a --delete --exclude .flagged '$LINK/' '$BACKUP/'" $USER
      else
        su -c "rsync -a '$BACKUP/' '$LINK/'" $USER
        su -c "touch '$LINK/.flagged'" $USER
      fi
    fi
  done
}

CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#119 2012-09-16 10:25:05

DarioP
Member
From: Geneva
Registered: 2011-01-05
Posts: 110

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

I was just asking to be sure of the behaviour of this tool.

If I have one file being modified and another finished and to be synced I can call the sync routine without any worries as I understood. Moreover I can add folders in WHATTOSYNC variable and sync them with a sync call without affect the files being modified. If this is true, now I'm definitely happy! big_smile

Probably I just got confused by what's written in the wiki:

Using SysVinit (Arch default)
Call the daemon to start, stop, or sync (restart) like any standard Arch daemon:

# rc.d command asd

When I tried sync it was not working so this led me to think that sync needed a manual stop and restart wink

Last edited by DarioP (2012-09-16 10:31:58)

Offline

#120 2012-09-16 11:29:04

graysky
Member
From: The worse toilet in Scotland
Registered: 2008-12-01
Posts: 8,649
Website

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

@DarioP - Ok... I will reword the wiki.  Can you verify that the latest version of asd satisfies all of your needs?


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#121 2012-09-17 22:55:46

thewall
Member
Registered: 2012-09-17
Posts: 9

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

Hi everybody.

I used anything-sync-daemon with a SSD drive and, as suggested on this page of the Archwiki, I synced my /var/log directory.
As I read in this thread, that was not a really good idea.
I got errors a couple of days ago about pacman database, but I fixed them. Today my root partition fucked up with errors on superblock: I managed to clone the partition on the HDD and I've been able to boot.
So, I blacklisted asd from /etc/rc.conf, but sometimes (not everytime) at boot I get errors like

Cannot create /var/log/xxx: file or directory doesn't exist

and X can't start cause it respawn to fast.
Luckily I can login from tty2 and solve the problem: to do that I need to start asd, stop it and reboot. Please notice that the daemon was not in /etc/rc.conf (it was not started)!!!

Do you have any idea how can I solve this? smile

Ps: Thanks a lot for your HUGE work!

Pps: you should also reword the Archwiki page I mentioned before wink

Offline

#122 2012-09-17 23:40:15

graysky
Member
From: The worse toilet in Scotland
Registered: 2008-12-01
Posts: 8,649
Website

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

@thewall - Thanks for the info.  The docs and the asd wiki have been edited to include a nice warning.  I just edited the SSD page as well totally removing the subsection.

An oversight on my part is why it started without the deamon running; this has been corrected in version 3.6 now in the AUR and on github.  Basically, cron would trigger a sync once per hour under the old version.  Now, cron is aware of the pid file, and checks to that is present before doing anything.

Please update to v3.6 and let me know if you experience any problems.  I'm glad you're savvy enough to keep your ship above the waterline smile

Last edited by graysky (2012-09-17 23:42:50)


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#123 2012-09-18 00:01:56

thewall
Member
Registered: 2012-09-17
Posts: 9

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

Hey, thanks for your reply. smile

I updated right now to version 3.6. What should I do? Start and stop the daemon and then reboot?

EDIT: I did it and now I shutdown. Here is 2 AM big_smile I'll let you know.

Last edited by thewall (2012-09-18 00:11:07)

Offline

#124 2012-09-18 00:10:49

graysky
Member
From: The worse toilet in Scotland
Registered: 2008-12-01
Posts: 8,649
Website

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

No need to reboot at all.  Actually, in this case, no need to restart the daemon either.  The changes are to the underlying script and crontab for hourly.


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#125 2012-09-18 04:45:01

Mr Green
Forum Fellow
From: U.K.
Registered: 2003-12-21
Posts: 5,752

Re: anything-sync-daemon - keep ANYTHING in tmpfs and sync'ed

Been running asd with systemd for a few days now, on some boots I was dropped to console log in instead of silm, last night system would not power down correctly. Loaded 3.6 see what happens...


Mr Green

Offline

Board footer

Powered by FluxBB