profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

capoeira · 2012-11-04 11:46:25

Awebb wrote:

I just updated from 3.16-1 to 4.03-1. I lost my chromium profile in the process. It somehow remembered my gmail account I was logged in with, but everything else is "lost". At least it's chromium, so I didn't lose anything but local plugin configs.
This is what I did: I closed the browser… stopped psd… grabbed the tarball… makepkg -s… pacman -U… delete the old psd.conf, rename psd.conf.pacnew and fill in my info… systemctl --system daemon-reload… systemctl start psd… profile dead.

look if you have "chromium-backup" folder in ~/.config
that saved me before
just replace the folder

Awebb · 2012-11-04 11:57:13

Unfortunately, there is no -backup folder. Looks like psd did it's job :-)

also updated the package on my netbook and everything went as expected. I then played a bit with the daemon. It looks like stopping the service while chromium is open has a certain chance of scrambleing the profile, depending on what chromium is doing at that time. I could do some diff'ing, but I have the feeling it is connected to the sync option with the gmail profile, because all the files were still there, chromium only seemed assumed, that it was installed on a new machine, because one of the files it uses to sync with the account was in an inconsistent state. That might explain why all the passwords in the password store are still there, as I unchecked them in the sync option.

Yay, cloud computing is soooo exciting…

Last edited by Awebb (2012-11-04 11:57:43)

graysky · 2012-11-04 13:35:55

Awebb wrote:

... After starting the new version for the first time, there were four folders, /tmp/$USER-{chromium,opera,firefox,google-chrome}, although I only have chromium installed and configured in the psd.conf. After stopping and starting the service again, there was only -chromium, as I am used to.

That would happen if:
1) You have all four browsers installed on your machine AND you have the BROWSERS array in /etc/psd.conf commented.
2) You have all four browsers installed on your machine AND you manually defined all of them in the BROWSERS array in /etc/psd.conf commented.

Awebb wrote:

...I then played a bit with the daemon. It looks like stopping the service while chromium is open has a certain chance of scrambleing the profile, depending on what chromium is doing at that time.

Interesting. I am going to mess around with mine right now -- after it is backed-up of course

EDIT: Without getting too verbose, the version 3 --> version 4 update is a tough one in that I cannot cause pacman to abort the installation if your certain conditions on your system exist. This should only be a problem going to v3.x to v4.x; subsequent version bumps do not care if, for example, browsers are open or closed.

Best practice for the users of version 3.x is to exit your browsers before you attempt to install version 4.x.

Last edited by graysky (2012-11-04 14:11:40)

graysky · 2012-11-04 18:35:54

Your experience, Awebb, caused me to think about how to avoid these situations. I came up with adding a check to the unsync function that sends SIGTERM to the browser if running thus quitting it. This way, browser profile integrity will be preserved if a user stops the deamon while it is running. The downside is that the browser will be quit, but the upside is that user profiles will be intact!

I will work on this later today; plz post your ideas for alternatives to this thread.

I'm thinking pkill -SIGTERM but if that fails, do we need a stronger kill command?

Last edited by graysky (2012-11-04 20:08:20)

Awebb · 2012-11-04 20:36:13

You could start by keeping the browser profile in /tmp instead of killing it instantly, maybe with a -bak suffix. This way the we could undo the damage by closing the browser, restarting the service and renaming the folder in /tmp.

Maybe you could sanitize psd's behavior. There must be a way to check whether a file is open, being written to, whatever. I have incron in mind right now, maybe the underlying mechanism, I don't have much expertise in that field.

About killing the browser:
1. I am not sure I'd be comfortable with this. I sometimes have work done in a browser window, open connections, open downloads, I would not want that to be lost.
2. At the moment, the only reasons for a cleanup-and-exit of psd are a) the user stopping the service or b) updating psd via AUR. Updating via AUR is still quite a manual intervention - I am excluding people relying on yaourt's updater here, because they should know better - so updating psd should obviously be a special moment for a browser user, so closing the browser first does not sound like witchcraft.

graysky · 2012-11-04 20:58:52

@Awebb - Correct, only two conditions that would require this:

1) User stopping the service.
2) Updating v3.x --> v4.x which should be a one-time thing.

To point #1, I can envision several scenarios where data could be lost: all are reboots/shutdowns.

...what if a reboot/shutdown is triggered and the user left the browsers open? For example using `sudo shutdown -h x` which I do pretty frequently.
...what if it is a multi-user box and user1 is logged in with browser open and idle and user2 logs into the box and reboots it? This too has happened to me when the wife leaves her shit open (browser included logged into facebook.com usually) and I sit down, and decide to reboot after rebuilding a kernel
...what if the user leaves a browser open on a laptop and the pm-daemon shuts down the box do due to critical battery levels being reached?

In all the of cases I outlined above, the init system would be running the unsync function. Adding a 'kill' function therein would safe guard the browser profiles in these events so I do think some sort of sanity check/sync back to disk before the daemon goes down is value-added.

I like the idea of placing a backup somewhere, but in the event of reboots/shutdowns, the backup would be lost since it's tmpfs.

Last edited by graysky (2012-11-04 21:01:09)

barraponto · 2012-11-05 03:39:33

Regarding the issue of running the browser before files are cached, what if PSD rsyncd the folders first, symlinked later? That way it'd never run with a crippled profile.

Last edited by barraponto (2012-11-05 03:39:52)

Awebb · 2012-11-05 08:03:40

Multi-user setups usually use a DM. If I understand systemd correctly, shutting down means working backwards through the targets. That means the browser will be dead or in a dying state by the time psd starts shutting down.

EDIT: On a multiuser system, I'd generally make it a habit of closing other user's sessions before rebooting, since psd isn't the only cause of possible inconsistence in such cases.

Last edited by Awebb (2012-11-05 08:05:09)

barraponto · 2012-11-05 17:35:10

I'm running psd 4.10-2 and it's syncing both firefox and chromium, despite setting BROWSERS to firefox on /etc/psd.conf
This is the content of my config file: https://gist.github.com/03114a3ccb5744c8caa1

graysky · 2012-11-05 19:57:17

barraponto wrote:

I'm running psd 4.10-2 and it's syncing both firefox and chromium, despite setting BROWSERS to firefox on /etc/psd.conf
This is the content of my config file: https://gist.github.com/03114a3ccb5744c8caa1

@barraponto - Please post the output of:

% profile-sync-daemon debug

Last edited by graysky (2012-11-05 19:57:50)

soupcan · 2012-11-08 01:08:49

I'm having a similar problem to samuvuo. I have no output from echo $SUDO_USER - I've reinstalled sudo and checked my configuration, I can't see anything wrong. My user is not present in the root lies as in samuvo's case. Added my user to sudoers via this line: USER_NAME ALL=(ALL) ALL. psd is starting on boot but it seems that my profile is not immediately synced. I have to login to chromium and then close and open it a few times before my settings are synced.
output of profile-sync-debug:

% profile-sync-daemon debug                                                                                 ~
::DEBUG MODE::
profile-sync-daemon version: 4.11

settings in /etc/psd.conf will make profile-sync-daemon manage the following browers/user combos when invoked:

browser to manage: chromium
owner/group:       killian:power
homedir:           /home/killian
backup target:     /home/killian/.config/chromium-backup

browser to manage: firefox
owner/group:       killian:power
homedir:           /home/killian
backup target:     /home/killian/.mozilla/firefox-backup

Hoping I can figure out what's going on, I'm going to test a few things. Hopefully I can figure out what's wrong with my sudo config. I'll provide any other outputs/info if needed.
EDIT: Things seem to be working right after a reboot now... I'll keep checking.

Last edited by soupcan (2012-11-08 01:28:15)

graysky · 2012-11-08 09:52:47

From your output, psd sees that you have a Firefox profile and a chromium profile. Psd does not use sudo at all so the status of that on your system does not matter. Are you starting it via the included service file?

bassu · 2012-11-08 17:25:04

Graysky: You are fuckin AWESOME!!! Love you!!!
First, the ck repo and now the psd -- you must be god-sent

Last edited by bassu (2012-11-08 17:25:32)

soupcan · 2012-11-08 23:59:06

graysky wrote:

From your output, psd sees that you have a Firefox profile and a chromium profile. Psd does not use sudo at all so the status of that on your system does not matter. Are you starting it via the included service file?

Yes, starting it on boot with systemctl enable psd.service.

Awebb · 2012-11-11 13:43:06

I have now found why chromium thinks the profile is gone. After the last reboot, there was a /tmp/myuser-chromium/chromium-backup with all the data, but no data in /tmp/myuser-chromium. Closing the browser, deleting all other files and then moving "chromium-backup/* .." solved it. There must be something going wrong sometimes either on syncing to the hdd before the reboot or on copying the data to /tmp on boot. I will inspect more.

graysky · 2012-11-11 14:08:11

@Awebb - Odd... this is exactly what the check function does

EDIT: I think I made the crash recovery more robust. Please try v4.13.
https://aur.archlinux.org/packages/profile-sync-daemon

Last edited by graysky (2012-11-11 14:32:52)

pingpong · 2012-11-13 05:16:44

Hum, are there tweaks that can make psd.service boot up faster? Here is my systemd-analyze blame output:
While 2.1s is not bad but it's still almost 7 times slower than the 2nd slowest service which is wicd.

Psd is very nice so I plan to keep it. And any suggestions to make it load faster are welcome, thanks in advance.

  2133ms psd.service
   323ms wicd.service
   233ms systemd-logind.service
   173ms systemd-udev-trigger.service
   166ms systemd-user-sessions.service
   155ms systemd-vconsole-setup.service
   155ms systemd-modules-load.service
   154ms sys-kernel-debug.mount
   147ms dev-mqueue.mount
   127ms dev-hugepages.mount
   124ms systemd-sysctl.service
    94ms systemd-remount-fs.service
    44ms tmp.mount
    39ms colord.service
    37ms systemd-udevd.service
    20ms systemd-tmpfiles-setup.service
    13ms home.mount
    10ms boot.mount

graysky · 2012-11-13 05:22:44

@pingpong - You can do several things:

1) Reduce your browser profile's footprint.
2) Reduce the number of browser profiles you want psd to manage (see /etc/psd.conf).
3) Upgrade to an SSD (if you haven't already).

#1 can be accomplished via several methods including removing/limiting cache, history, etc. You can also use profile-cleaner for browser profile maintenance.

Last edited by graysky (2012-11-13 05:22:56)

pingpong · 2012-11-13 05:29:52

@graysky

Thanks for the quick reply! For 2) I have only one browser (firefox) and one profile being synced, so not much I can do there. And the system is on an SSD indeed.
I will look into 1) and report back if I can achieve something significantly faster.

graysky · 2012-11-13 12:43:36

@pingpong - How large is the firefox directory currently?

du -Lhs $HOME/.mozilla/firefox

Run profile-cleaner and repeat that command to see the difference.

As a point of comparison for you, my chromium profile is 98M and syncing takes <700 ms.

% systemd-analyze blame | grep psd
   695ms psd.service

% du -Lhs $XDG_CONFIG_HOME/chromium
98M	/home/facade/.config/chromium

Last edited by graysky (2014-11-03 08:40:33)

pingpong · 2012-11-13 14:56:39

@graysky

Thanks for the tips. Well here is what my output of

du -Lhs $HOME/.mozilla/firefox

~ [i]> du -Lhs $HOME/.mozilla/firefox/
du: cannot access ‘/home/pong/.mozilla/firefox/5ydjr43n.default/lock’
90M	/home/pong/.mozilla/firefox/

So around 90M and that's after running profile-cleaner.
Unfortunately, in my case, after all these psd.service still takes 2163ms to start.

By the way, I don't understand why there is a link

lrwxrwxrwx  1 pong users       15 Nov 13 06:44 lock -> 127.0.0.1:+2090

in my profile though, could it be the problem? Firefox regenerate the link after it's been removed. Any idea?

graysky · 2012-11-13 15:33:49

I believe that you have to close firefox before you run the du scan. Chromium has a similar lock when running.

Did you run the cleaner?
How faster is your SSD (SATA2 or SATA3)?
Mine is a SATA3 for reference.

Last edited by graysky (2012-11-13 16:18:11)

pingpong · 2012-11-13 15:37:14

@graysky

Oh, I just realize that having that lock link is normal when firefox is running. It goes away when I quit it.

However, in the output systemd-analyed plot, it shows that psd.service has a red bar (i.e. activating) for the whole 2s. While others services has their bars part red and part pink.
I don't know if this helps in diagnosis you start 3 times faster than I.

graysky · 2012-11-13 16:17:58

pingpong wrote:

...in the output systemd-analyed plot, it shows that psd.service has a red bar (i.e. activating) for the whole 2s. While others services has their bars part red and part pink. I don't know if this helps in diagnosis you start 3 times faster than I.

No idea... I assume that 'activating' in this context means that you're waiting for the rsync to finish. I am thinking that you simply have a slower machine/SSD than I do.

My CPU is an i7-3770k @ 4.5 GHz.
My chipset is a Z77.
My SSD is OCZ-VERTEX 4 on the Z77 SATA3 controller. See its benchmark results on the wiki page: https://wiki.archlinux.org/index.php/SS … Controller

Several questions for you:

1)

graysky wrote:

How fast is your SSD (SATA2 or SATA3)? Mine is a SATA3 for reference.

2) How brand/model is your SSD?
3) What do your benchmark numbers look like in endpoints shown on the wiki page?

Last edited by graysky (2012-11-13 16:26:29)

pingpong · 2012-11-13 16:54:01

It may well be the case. You have an awesome system

Mine is a Toshiba R835-P56. The cpu is i5 2450 @2.3 GHz.
Has to look to find out the chipset. And it's on Vertex 3. (SATA 3)

I like it, it's more than enough for what I need it for.

Hum... it there is way to let me have the login prompt before psd finish syncing the profile?

Without it my boot time is just < 3s.

Arch Linux

#201 2012-11-04 11:46:25

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#202 2012-11-04 11:57:13

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#203 2012-11-04 13:35:55

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#204 2012-11-04 18:35:54

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#205 2012-11-04 20:36:13

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#206 2012-11-04 20:58:52

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#207 2012-11-05 03:39:33

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#208 2012-11-05 08:03:40

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#209 2012-11-05 17:35:10

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#210 2012-11-05 19:57:17

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#211 2012-11-08 01:08:49

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#212 2012-11-08 09:52:47

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#213 2012-11-08 17:25:04

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#214 2012-11-08 23:59:06

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#215 2012-11-11 13:43:06

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#216 2012-11-11 14:08:11

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#217 2012-11-13 05:16:44

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#218 2012-11-13 05:22:44

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#219 2012-11-13 05:29:52

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#220 2012-11-13 12:43:36

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#221 2012-11-13 14:56:39

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#222 2012-11-13 15:33:49

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#223 2012-11-13 15:37:14

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#224 2012-11-13 16:17:58

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#225 2012-11-13 16:54:01

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

Board footer