profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

GNA · 2013-01-01 17:34:00

graysky wrote:

Overly simplified example using chromium:
DIR=$HOME/.config/chromium
BACKUP=$DIR-backup

1) Make empty tmpfs container if not there:
 e.g. /tmp/$USER-chromium

2a) Move profile to profile-backup if not there:
 e.g. $DIR --> $BACKUP
2b) Create link to tmpfs
 e.g. /tmp/$USER-chromium --> $DIR

3) Sync the HDD backup to  tmpfs
 e.g, $BACKUP/ /tmp/$USER-chromium/
I believe you are proposing an order of: 1, 3, 2a, 2b. The code would need to adjusted to sync $DIR not $BACKUP for the initial sync. So if rsync is active when the browser is started, the browser would still read $DIR. What I don't know is what happens if files are modified during the simultaneous browser usage/rsync step...?

What I am proposing is to change the order like this:
1, 2a, 3, 2b

So after 2a until reaching 2b, firefox won't find the profile folder, as it was renamed to profile-backup. So it will display something like: "Profile not found". So while the critical part is running, there just wont be any profile from the perspective of firefox. So if you mistakeably try to start firefox too early, it will end with you having to click away an annoying message but not breaking anything. This could be made even better by delaying firefox until this process has finished.

Last edited by GNA (2013-01-01 17:35:19)

dodo3773 · 2013-01-02 07:40:44

Thanks for another great project graysky. I seem to be using more and more things from you. The large benefit of psd for me is that I constantly empty my cache. Now with this it went from a few seconds to less than one. I too am experiencing slow boot times though. My "systemd-analyze blame" puts psd.service at 22209ms. I would rather wait the extra time and continue to use psd though. It's worth it. Just wanted to let you know / verify with my experience what freestyler7 is saying.

GNA · 2013-01-12 10:55:18

Ok, here is my patch:

move

ln -s "$VOLATILE/$user-$browser$suffix" "$DIR"                                          
chown -h $user:$group "$DIR"

after this line:

rsync -aog --delay-updates "$BACKUP/" "$DIR/"

and change that lane to

 rsync -aog --delay-updates "$BACKUP/" "$VOLATILE/$user-$browser$suffix/"

Done.

What does it do? See post #301.

For future reference, does anyone know if it is possible to commit diff files for aur packages?

barraponto · 2013-01-19 13:50:15

Would we benefit from running psd as a user daemon? I saw instructions for MPD at https://wiki.archlinux.org/index.php/MP … th_systemd and thought we could shave some boot time by going that way.

graysky · 2013-01-19 13:51:19

barraponto wrote:

Would we benefit from running psd as a user daemon? I saw instructions for MPD at https://wiki.archlinux.org/index.php/MP … th_systemd and thought we could shave some boot time by going that way.

Bad idea.

1) No benifits.
2) Hardcoded to run as root so it can maintain ownership of sync targets.

fradeve · 2013-01-24 16:26:44

Hi guys,
this is my first month with Arch. I noticed that Firefox was slightly slower on the same machine in comparison with other distros, so I decided to try PSD. This is my conf:

/etc/fstab

tmpfs       /tmp    tmpfs           nodev,nosuid                    0  0

/etc/psd.conf

# List users separated by spaces whose browser(s) profile(s) will get symlinked
# and sync'ed to tmpfs
# Do NOT list a user twice!
#
# Example
# USERS="facade happy"
USERS="fradeve"

# List browsers separated by spaces to include in the sync. Useful if you do not
# wish to have all possible browser profiles sync'ed
#
# Possible values:
#               chromium
#               conkeror.mozdev.org
#               firefox
#               google-chrome
#               heftig-aurora
#               midori
#               opera
#               opera-next
#               qupzilla
#
# Uncomment and select which browsers to manage if you wish
# Otherwise all available/supported browsers will be managed
BROWSERS="firefox midori"

# Define where browser profiles will reside in tmpfs
# This location must be mounted to tmpfs and MUST have permissions of 777
# Use NO trailing backslash!
# If running a standard ARCH install the following default is fine
VOLATILE="/tmp"

profile-sync-daemon parse

Profile-sync-daemon v5.10

Psd will manage the following per /etc/psd.conf settings:

 browser/psname:  firefox/firefox
 owner/group:     fradeve/users
 sync target:     /home/fradeve/.mozilla/firefox/f5cnrqwq.default-1357907171863
 tmpfs dir:       /tmp/fradeve-firefox-f5cnrqwq.default-1357907171863
 profile size:    59M

 browser/psname:  midori/midori
 owner/group:     fradeve/users
 sync target:     /home/fradeve/.config/midori
 tmpfs dir:       /tmp/fradeve-midori
 profile size:    1,4M

So, I enabled PSD with

sudo systemctl enable psd.service

and rebooted.

When loggin in XFCE4, when launching FF, it complains about the missing user profile, and FF refuses to launch.

Any hint?

Thank you

Last edited by fradeve (2013-01-24 18:19:44)

aesiris · 2013-01-24 17:16:35

fradeve wrote:

Hi guys,
this is my first month with Arch. I noticed that Firefox was slightly slower on the same machine in comparison with other distros, so I decided to try PSD. This is my conf:
/etc/fstab
[...]

By default systemd mounts /tmp as a tmpfs, you do not need this line in /etc/fstab; but I don't think this is a problem.

[...]
Every login on XFCE4, when launching FF, it complains about the missing user profile, and FF refuses to launch.
Any hint?
Thank you

Did you run firefox before psd.service was finished?
Maybe you will need to recover your profile from "/home/fradeve/.mozilla/firefox/f5cnrqwq.default-1357907171863-backup"

Please provide also the output of these commands:

systemctl status psd.service
journalctl _SYSTEMD_UNIT=psd.service
ls -l /home/fradeve/.mozilla/firefox/

fradeve · 2013-01-24 18:38:00

Hmmm it seems that things are working now. This is very strange because for 2 days FF gave me almost randomly the above mentioned error. The only thing I've done is

sudo systemctl disable psd.service

reboot.

sudo systemctl enable psd.service

reboot.

Btw:
systemctl status psd.service

psd.service - Profile-sync-daemon
          Loaded: loaded (/usr/lib/systemd/system/psd.service; enabled)
          Active: active (exited) since gio 2013-01-24 20:28:17 CET
         Process: 255 ExecStart=/usr/bin/profile-sync-daemon sync (code=exited, status=0/SUCCESS)

journalctl _SYSTEMD_UNIT=psd.service
empty

ls -l /home/fradeve/.mozilla/firefox/

drwx------ 17 fradeve users 12288 24 gen 19.33 f5cnrqwq.default-1357907171863
-rw-r--r--  1 fradeve users   132 11 gen 13.26 profiles.ini

Since I start FF about a couple of minutes after logging in, I'm pretty sure that psd.service has exited at that time.
Is it possible that, for some reasons, psd.service takes longer to end sometimes after logging in? It seems the only possibile scenario to me.

graysky · 2013-01-24 20:49:50

fradeve wrote:

Since I start FF about a couple of minutes after logging in, I'm pretty sure that psd.service has exited at that time.
Is it possible that, for some reasons, psd.service takes longer to end sometimes after logging in? It seems the only possibile scenario to me.

Yes, it is possible. Since systemd parralellizes everything, many services start at boot and all want disk IO to complete. It could be that you launched the browser before psd finished syncing. Post the output of `systemd-analyze blame` to give a rough idea. My system uses a pretty fast SSD so it is not an issue for me:

% systemd-analyze blame
  3693ms netctl@static.service
   346ms psd.service
   109ms ufw.service
    83ms systemd-udev-trigger.service
    71ms mnt-data.mount
    57ms systemd-logind.service
    56ms lm_sensors.service
    46ms systemd-modules-load.service
    43ms tmp.mount
    43ms systemd-remount-fs.service
    43ms systemd-sysctl.service
    40ms gpm.service
    40ms sys-kernel-debug.mount
    40ms dev-mqueue.mount
    39ms systemd-vconsole-setup.service
    39ms cpupower.service
    36ms dev-hugepages.mount
    36ms systemd-tmpfiles-setup.service
    33ms systemd-udevd.service
    26ms ntpd.service
    26ms polkit.service
    23ms systemd-user-sessions.service
    21ms rtkit-daemon.service
    20ms boot.mount
    14ms udisks2.service
    10ms home.mount
     7ms upower.service
     6ms var.mount
     6ms scratch.mount
     0ms sys-fs-fuse-connections.mount
     0ms home-facade-.cache.mount

fradeve · 2013-01-24 21:13:09

Probably is it.

I think it could be useful -- since it is a pretty interesting command to describe how things works -- adding systemd-analyze blame reference to the PSD wiki page, as a troubleshooting tip, to avoid boring users on the forum

This is my boot chart:

   903ms wicd.service
   540ms psd.service
   341ms gdm.service
   339ms systemd-logind.service
   186ms systemd-vconsole-setup.service
   146ms systemd-udev-trigger.service
   110ms systemd-user-sessions.service
    86ms accounts-daemon.service
    82ms systemd-remount-fs.service
    72ms colord.service
    60ms polkit.service
    55ms udisks2.service
    53ms systemd-udevd.service
    50ms sys-kernel-debug.mount
    40ms dev-mqueue.mount
    39ms systemd-tmpfiles-setup.service
    34ms dev-hugepages.mount
    33ms systemd-sysctl.service
    23ms tmp.mount
    17ms upower.service
    16ms udisks.service
    15ms ntpd.service
     8ms home.mount
     7ms rtkit-daemon.service
     6ms proc-sys-fs-binfmt_misc.mount
     2ms sys-fs-fuse-connections.mount

Thank you guys!

graysky · 2013-01-24 21:18:21

540 ms is pretty quick... I doubt that you are starting firefox too fast unless you are logging in and instantaneously launching the browser (autostart)?

graysky · 2013-01-24 21:35:39

GNA wrote:

Ok, here is my patch:
...

Seems an okay fix at first pass. Others care to test the patch proposed by GNA... preferably those with firefox? Working fine with chromium.

--- /usr/bin/profile-sync-daemon  2012-12-20 08:53:11.000000000 -0500
+++ /usr/bin/profile-sync-daemon  2013-01-24 16:32:46.668852037 -0500
@@ -2,7 +2,7 @@
 # By graysky <graysky AT archlinux DOT us>
 # Inspired by some code originally  written by Colin Verot
 export BLD="\e[01m" RED="\e[01;31m" GRN="\e[01;32m" YLW="\e[01;33m" NRM="\e[00m"
-VERS="5.10"
+VERS="5.11"

 if [[ ! -f /etc/psd.conf ]]; then
  echo "Cannot find /etc/psd.conf so bailing.  Reinstall package to use Profile-sync-daemon." # nothing to do if there is no conf file
@@ -255,8 +255,6 @@
          # backup target and link to tmpfs container
          if [[ $(readlink "$DIR") != "$VOLATILE/$user-$browser$suffix" ]]; then
            mv "$DIR" "$BACKUP"
-           ln -s "$VOLATILE/$user-$browser$suffix" "$DIR"
-           chown -h $user:$group "$DIR"
          fi

          # sync the tmpfs targets to the disc
@@ -265,7 +263,9 @@
          else
            # initial sync
            # keep user from launching browser while rsync is active
-           rsync -aog --delay-updates "$BACKUP/" "$DIR/"
+           rsync -aog --delay-updates "$BACKUP/" "$VOLATILE/$user-$browser$suffix/"
+           ln -s "$VOLATILE/$user-$browser$suffix" "$DIR"
+           chown -h $user:$group "$DIR"
            touch "$DIR/.flagged"
          fi
        fi

fradeve · 2013-01-26 10:28:15

graysky wrote:

540 ms is pretty quick... I doubt that you are starting firefox too fast unless you are logging in and instantaneously launching the browser (autostart)?

Sure, I'm not so fast

Anyway, after a couple of reboots, I noticed the PSD strange behaviour: it seems to start correctly or give error almost randomly; so it can happen that FF works as expected and the PSD daemon was running, or the contrary. I omitted to say that I'm using an encrypted home partition, configured as reported in https://wiki.archlinux.org/index.php/EC … tfs_module, using eCryptfs, PAM, GDM and Xfce4. So, I suppose that the problem is that PSD daemon cannot read browser data until PAM mounts eCryptfs private dir, and the concurrency between the two processes seems to generate random results; sometime it works, sometimes it doesn't.

So, I tried an alternative solution:

sudo systemctl disable psd.service

added the following line in /etc/sudoers

fradeve ALL=(ALL:ALL) NOPASSWD:/usr/bin/systemctl

added the following in $HOME/.config/autostart/

[Desktop Entry]
Encoding=UTF-8
Version=0.9.4
Type=Application
Name=psd.service
Comment=
Exec=sudo /usr/bin/systemctl start psd.service
OnlyShowIn=XFCE;
StartupNotify=false
Terminal=false
Hidden=false

I know, it's awful but -- hey -- it works.

Any comment? Is it too dangerous to give a user sudo permissions without password request on systemctl?

graysky · 2013-01-26 11:16:25

An encrypted $HOME would muck things up and require psd to start after it was in place as you describe. Your solution works and I don't think it's dangerous. How many users have this in their /etc/sudoers

%wheel ALL=(ALL) NOPASSWD: ALL

A more universal solution would be to make psd wait for wherever mount your encrypted $HOME. It's not clear to me what that is and how to do it in the psd.service file... suggestions are welcome.

Last edited by graysky (2013-01-26 11:18:04)

Lockheed · 2013-02-27 12:56:22

I have some questions about the inner workings of this tool:

1. Does it load profile dirs into RAM even if the browser is not running, thus permanently occupying portion of memory?

2. I have external scripts that delete older cache files from opera profile. It is set to run every hour. If that occurs, won't the deleted cache files be restored back form memory upon daemon shutdown?

aesiris · 2013-02-27 17:53:05

Lockheed wrote:

I have some questions about the inner workings of this tool:
1. Does it load profile dirs into RAM even if the browser is not running, thus permanently occupying portion of memory?

Yes.
But if you are on RAM pressure it will probably be swapped out.

Lockheed wrote:

2. I have external scripts that delete older cache files from opera profile. It is set to run every hour. If that occurs, won't the deleted cache files be restored back form memory upon daemon shutdown?

No.
When the service is started your profile directory is a link to a directory in RAM; only a backup is kept on disk and it will be restored in case of ungraceful stop (for example power failure, crash, etc.).
You should just check that psd.service is completely started before running your script.

graysky · 2013-02-27 19:44:55

arseris is right about the first question. Depending on your script, the answer to the 2nd question may also be yes. In other words, psd symlinks your profile dir. If your script is just pointing to the profile dir (the symlink) to delete stuff, it should happily do so blissfully unaware that it is operating on a symlink, rather than on a 'live' directory. There should be no need to stop psd before you run your script is what I'm saying.

Lockheed · 2013-02-27 20:08:29

Ok, guys. That helped alot. Thank you.

FarStar07 · 2013-03-17 16:34:34

Hello graysky,

For your information, there is a change in the profile path of Qupzilla:

QupZilla 1.4.x
------------------
Very important change is that config directory is now in ~/.config/qupzilla by default. It will, however, use the old ~/.qupzilla if it exists (and ~/.config/qupzilla does not).
While it is (and will always be) backwards compatible, I'd recommend you to manually move ~/.qupzilla into ~/.config/qupzilla.

Source

graysky · 2013-03-17 18:25:58

Thanks, FarStar. v5.31 will handle this.

EDIT: Just pushed to the AUR.

Last edited by graysky (2013-03-17 23:35:31)

FarStar07 · 2013-03-18 05:25:58

Thanks

nperry · 2013-03-18 10:25:05

Is is normal for psd and psd-rsync to take such a long time at boot?

─╼ sudo systemd-analyze blame
7297ms psd.service
1131ms psd-resync.service
238ms systemd-fsck@dev-disk-by\x2duuid-431d523f\x2d52ec\x2d44a7\x2dae2d\x2d38c389cb80a5.service
237ms teamviewerd.service
215ms systemd-modules-load.service
214ms var.mount
210ms systemd-vconsole-setup.service
198ms systemd-udevd.service
165ms systemd-sysctl.service
157ms systemd-udev-trigger.service
146ms dhcpcd.service
143ms systemd-remount-fs.service
127ms systemd-fsck@dev-disk-by\x2duuid-7be0c8b4\x2d64f9\x2d405e\x2db0f3\x2d7e2c544bad29.service
110ms home.mount
103ms systemd-random-seed-load.service
102ms systemd-fsck@dev-disk-by\x2duuid-a9de3235\x2d64de\x2d4f3d\x2db4f2\x2d1a9c01b5c272.service
100ms alsa-restore.service
80ms systemd-tmpfiles-setup.service
74ms systemd-logind.service
60ms dev-mqueue.mount
49ms systemd-user-sessions.service
44ms boot.mount
37ms dev-disk-by\x2duuid-c617c491\x2deefd\x2d4b9d\x2daf5d\x2d4b6df908aadd.swap
34ms dev-hugepages.mount
28ms systemd-journal-flush.service
27ms sys-kernel-debug.mount
26ms systemd-tmpfiles-clean.service
24ms tmp.mount
3ms home-neil-.cache.mount

FarStar07 · 2013-03-18 12:10:32

nperry wrote:

Is is normal for psd and psd-rsync to take such a long time at boot?

It depends on the size of the browser user profile(s).
You can clear the browser cache on exit to decrease this boot time.
You can also use graysky's profile-cleaner script to reduce the size of the profile(s).
For me, for Firefox + Opera + Qupzilla => 1482ms psd.service

Last edited by FarStar07 (2013-03-18 12:21:30)

nperry · 2013-03-18 14:04:08

I've just used the profile cleaner and I've now reduced it, but the load time still seems to be a bit high.

─╼ profile-sync-daemon parse
Profile-sync-daemon v5.30 on Arch Linux.
Daemon file /run/psd is present.
Service is currently active.
Psd will manage the following per /etc/psd.conf settings:
browser/psname: chromium/chromium
owner/group: neil/users
sync target: /home/neil/.config/chromium
tmpfs dir: /tmp/neil-chromium
profile size: 90M
browser/psname: firefox/firefox
owner/group: neil/users
sync target: /home/neil/.mozilla/firefox/a62kelqm.default
tmpfs dir: /tmp/neil-firefox-a62kelqm.default
profile size: 32M

I'm not to sure if this is because my /home partition is on a HDD and / is on an SSD.

When you say clearing cache on exit, is there a way to do this for chromium and Firefox, automated?

FarStar07 · 2013-03-18 14:41:07

My Home partition is on an SSD. For me, after cleaning:

$ profile-sync-daemon parse
Profile-sync-daemon v5.31 on Chakra Linux (2013.01.23 - Benz).
Daemon file /run/psd is present.
Service is currently active.
Psd will manage the following per /etc/psd.conf settings:
browser/psname: qupzilla/qupzilla
owner/group: farstar/users
sync target: /home/farstar/.config/qupzilla
tmpfs dir: /tmp/farstar-qupzilla
profile size: 1,2M
browser/psname: firefox/firefox
owner/group: farstar/users
sync target: /home/farstar/.mozilla/firefox/dgdg4wji.default
tmpfs dir: /tmp/farstar-firefox-dgdg4wji.default
profile size: 15M
browser/psname: opera/opera
owner/group: farstar/users
sync target: /home/farstar/.opera
tmpfs dir: /tmp/farstar-opera
profile size: 16M

$ sudo systemd-analyze blame
999ms psd.service

Clearing cache on exit:

- For Firefox: Preferences > Privacy > History
Choose "Use custom settings for history"
Check "clear history when Firefox closes"
In Settings, check "cache".

- For Chrome, I don't know because I don't use it.

Last edited by FarStar07 (2013-03-18 15:09:28)

Arch Linux

#301 2013-01-01 17:34:00

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#302 2013-01-02 07:40:44

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#303 2013-01-12 10:55:18

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#304 2013-01-19 13:50:15

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#305 2013-01-19 13:51:19

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#306 2013-01-24 16:26:44

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#307 2013-01-24 17:16:35

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#308 2013-01-24 18:38:00

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#309 2013-01-24 20:49:50

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#310 2013-01-24 21:13:09

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#311 2013-01-24 21:18:21

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#312 2013-01-24 21:35:39

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#313 2013-01-26 10:28:15

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#314 2013-01-26 11:16:25

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#315 2013-02-27 12:56:22

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#316 2013-02-27 17:53:05

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#317 2013-02-27 19:44:55

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#318 2013-02-27 20:08:29

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#319 2013-03-17 16:34:34

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#320 2013-03-17 18:25:58

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#321 2013-03-18 05:25:58

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#322 2013-03-18 10:25:05

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#323 2013-03-18 12:10:32

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#324 2013-03-18 14:04:08

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#325 2013-03-18 14:41:07

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

Board footer