profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

graysky · 2012-11-28 22:17:25

OK... I just released an update to profile-cleaner. Not only that, but I re-wrote it to use GNU parallel rather than the old for loops. It is FAST now with multiple profiles.

See results over in the profile-cleaner thread.

Last edited by graysky (2012-11-28 23:19:41)

pingpong · 2012-11-28 23:06:48

That's great. I updated them. Many thanks!

alain · 2012-11-30 17:29:41

@graysky

I've installed profile-sync-daemon and is seems to work normaly with my profiles. Many thanks!!

However, there is a small bugs: you assume that profiles folders have different names, which may be not the case (in /tmp a profile is created under $user-$browser$suffix).

But i had 2 profiles with the same name (but differents path):
Path=/Data/User1/FirefoxProfile
Path=/Data/User2/FirefoxProfile

In /tmp, I saw:
myLogin-firefox-FirefoxProfile

So I have corrected this by appending the username to "FirefoxProfile", so:
Path=/Data/User1/FirefoxProfileUser1
Path=/Data/User2/FirefoxProfileUser2

and the result in /tmp is correct:
myLogin-firefox-FirefoxProfileUser1
myLogin-firefox-FirefoxProfileUser2

So I suggest that you manage differently $suffix (but I didn't understand how you initialize it )

Best regards.
Alain

Last edited by alain (2012-11-30 17:32:58)

graysky · 2012-11-30 22:27:03

Interesting... it inits $suffix on line 216 which just sets it to remove the longest part of the full path if the broswers are either firefox or aurora. If any other browser, suffix is a null:

[[ "$browser" = "firefox" ]] || [[ "$browser" = "heftig-aurora" ]] && suffix="-${item##*/}" || suffix=

The suffix="-${item##*/}" just removes everything except the list dir you have defined in the Path= line your profiles.ini in this case.

Anyway, to your point. I am unsure how to go about a situation when two members of the array are identical... suggestions are welcome and I will think on it.

alain · 2012-12-01 16:38:06

Ok, thanks for the explanation.
The simplest way to modify the actual code is maybe to add an index to the item. So if two items are identical, their folders will be differents.
For example:

Path=/Data/User1/FirefoxProfile
Path=/Data/User2/FirefoxProfile
Path=/Data/User2/FirefoxProfile2

will be transformed into:

myLogin-firefox-FirefoxProfile-1
myLogin-firefox-FirefoxProfile-2
myLogin-firefox-FirefoxProfile2-3

But solutions more elegants should exist

Last edited by alain (2012-12-01 16:38:24)

graysky · 2012-12-01 23:59:14

alain wrote:

...The simplest way to modify the actual code is maybe to add an index to the item.

I can do that but what happens if the user removes a profile or edits profiles.ini while the daemon is 'running?' It would result in the mis-match the next time a sync or unsync occurs... I think we need a more robust solution, or I can list this as a known issue in the man page or in the readme.install and leave it as-is.

Last edited by graysky (2012-12-02 00:00:07)

alain · 2012-12-02 09:02:14

graysky wrote:

I can do that but what happens if the user removes a profile or edits profiles.ini while the daemon is 'running?' It would result in the mis-match the next time a sync or unsync occurs...

I thought that you read profiles.ini only at startup (it may be a solution too)....
A big warning should be make into the ma page while a more robust solution is not found, because if two profiles have the same name, users will lost one of them (after sync).

graysky · 2012-12-02 16:39:50

OK. I added a check function that simply refuses to start if the situation you described is encountered. Also added this caveat to the man page. It works on my test system. Please restore your profiles.ini and filesystem to your original condition and try version 5.07-1. Let me know if it also refuses to start for you which it should.

Thank you by the way for helping to make psd a more robust product

EDIT: Try running psd in 'parse' mode which will show you exactly what it plans to sync and manage for you based on your /etc/psd.conf

Last edited by graysky (2012-12-02 21:06:00)

pingpong · 2012-12-03 03:28:20

I ran into a problem

profile-cleaner v1.86

Error: no profile directory for chromium found.

Any idea? Thanks in advance.

Awebb · 2012-12-03 07:28:03

Disable psd before cleaning the profile.

EDIT:

Or… file a bug report to the cleaner project, because it does not follow symlinks.

Last edited by Awebb (2012-12-03 07:28:44)

graysky · 2012-12-03 08:29:50

pingpong wrote:

I ran into a problem
profile-cleaner v1.86

Error: no profile directory for chromium found.
Any idea? Thanks in advance.

It does follow links and has been designed to work with psd. Post the output of:

$ psd parse

Last edited by graysky (2012-12-03 08:30:01)

freestyler7 · 2012-12-03 12:20:44

Hello. So, i did an upgrade yesterday:
upgraded profile-cleaner (1.66-1 -> 1.86-1)
upgraded profile-sync-daemon (4.13-1 -> 5.08-2)

But did the mistake of running profile-cleaner && installing the new psd version, all without stopping the psd service first.
I powered on my pc an hour ago and the boot up seemed way too fast.
I thought "must be some crazy optimizations in the new version or something"
Seeing /tmp folder at "0% used" seemed a bad sign, and trying to open firefox confirmed that. "your firefox profile cannot be loaded. It may be missing or inaccessible."

[free@archlinux ~]$ ls .mozilla/
extensions firefox firefox-backup

And also "profile-sync-daemon parse" output is empty.

The profile isn't there. I'll get the backup but i was wondering if it was my fault or a bug in the new version?
I also changed firefox option to 200mb fixed cache value yesterday instead of letting it handle it dynamically. ( i had almost 300 mb profile which took 30-40 secs according to systemd blame)

Somewhat irrelevant: I noticed a fstab.pacnew that was created few days ago that it's empty. Does that mean i don't need the /tmp mount entry in my fstab anymore or i misunderstood?

Thanks for the programs, makes me want less to buy an SSD (and that's good, because no money atm )

Last edited by freestyler7 (2012-12-03 13:41:22)

graysky · 2012-12-03 13:10:49

My fault. The psd.install I wrote for version 5 got corrupted and I installed from a backup that did not include the version check kit should have. I need to edit this. What should have happened is pacman should have stopped psd version <5.02 for you.

Do this to recover. Stop psd.
1) rm -f /run/psd # if it is there
2) mv ~/.mozilla/firefox ~/.mozilla/firefox-oops
3) mv ~/.mozilla/firefox-backup ~/.mozilla/firefox

Start psd and all should be well. Thanks for bringing this to my attention.

EDIT: you may remove that line from your fstab if you are using systemd which does it for you.

Last edited by graysky (2012-12-03 13:12:18)

alain · 2012-12-03 13:37:40

graysky wrote:

OK. I added a check function that simply refuses to start if the situation you described is encountered. Also added this caveat to the man page. It works on my test system. Please restore your profiles.ini and filesystem to your original condition and try version 5.07-1. Let me know if it also refuses to start for you which it should.

Hi graysky,

I will test the new version probably this week-end and will give you a feed back

graysky wrote:

Thank you by the way for helping to make psd a more robust product

You're welcome , and thanks to you for sharing your work, it's greatly appreciated

freestyler7 · 2012-12-03 13:41:45

Thanks, all working fine.

pingpong · 2012-12-03 17:12:32

@garysky

The output of

psd parse

is

Profile-sync-daemon v5.08

Psd will manage the following per /etc/psd.conf settings:

 browser/psname:  firefox/firefox
 owner/group:     pong/users
 sync target:     /home/pong/.mozilla/firefox/5ydjr43n.default
 tmpfs dir:       /tmp/pong-firefox-5ydjr43n.default
 profile size:    26M

So do I need to get chromium to psd.conf? Just separate it by a space from firefox? like

BROWSERS="firefox"

?

In fact, I seldom use chromiu, so I don't necessary want psd to manage it everytime when I start the machine. I just want to use profile-cleaner to clean its database once in a while just like before. Get it be done? Again thanks in advance

graysky · 2012-12-03 19:19:48

pingpong wrote:

So do I need to get chromium to psd.conf? Just separate it by a space from firefox?

Yes. Please refer to the man page, the wiki, or the comments in /etc/psd.conf itself.

pingpong wrote:

In fact, I seldom use chromiu, so I don't necessary want psd to manage it everytime when I start the machine. I just want to use profile-cleaner to clean its database once in a while just like before. Get it be done? Again thanks in advance

I am guessing that something isn't right with the location of your chromium profile; profile-cleaner expects it to be in $XDG_CONFIG_HOME/chromium and will error out if it can't find it. Post the output of:

% ls -l $XDG_CONFIG_HOME/chromium

pingpong · 2012-12-03 23:12:29

Well the command just list my home directory.
In fact

echo $XDG_CONFIG_HOME

return an empty line. (So that variable is not defined).

And problem solved after I setting the $XDG_CONFIG_HOME variable to ~/.config

Thanks for the explaination.

Last edited by pingpong (2012-12-03 23:18:03)

graysky · 2012-12-04 00:11:57

@pingpong - That is crazy. Maybe I need to change that variable in the script...

EDIT: Done. Thanks for the bug report.

Last edited by graysky (2012-12-04 00:42:20)

fschiff · 2012-12-04 03:11:07

Should have used
XDG_CONFIG_HOME=${XDG_CONFIG_HOME:-$HOME/.config}
(unless I mis-rememered that)

some (few) might have changed XDG_CONFIG_HOME to $HOME/config (no period) or something instead of not having it defined.

Feydaykyn · 2012-12-04 09:05:07

Hello Graysky,

first, thanks for the psd program, it's great !

I have an issue with psd taking ages (30s+) to start, it may only be a symptom of something else, but that's a start...
Psd is configured to work only with Firefox, with two profiles cleaned with profile-cleaner (thanks for this one too, by the way), of 125M and 128M.
Here the systemd-analyse blame |grep psd :
34979ms psd.service

If you have any idea, I'll be glad to hear it.

Thanks you !

Last edited by Feydaykyn (2012-12-04 11:37:06)

graysky · 2012-12-04 10:06:47

@fschiff - Good suggestion. Implemented in v1.88 (https://aur.archlinux.org/packages/profile-cleaner).
@Feydaykyn - I am guessing that have a spinning HDD (not SSD) and/or an older CPU and/or slow HDD and/or because systemd is aggressively starting it along with your desktop in parallel.

Try this experiment:
1) Close your browsers and stop psd.
2) Manually start psd from a shell once your system is idle (as in not loading other startup items).
3) Post the output of `systemd-analyze blame | grep psd`

Feydaykyn · 2012-12-04 11:36:02

Hi Graysky,

I tried the experiment and it took 11549ms to start. So I went to check the hdparm values and it had been kept to battery level while I am on AC, so I fixed that and now I get a "flashy" 4976ms (I've got a spinning HDD as you guessed).

Thanks for your time.

graysky · 2012-12-04 12:22:48

@Feydaykyn - OK. Glad you got to the bottom of it. Short of purchasing an SSD, I think you're stuck with it as such. If it makes you feel better, my workstation is driven by a pretty fast SSD and psd takes 2x as long to load at startup as it does when I start it manually. I believe this is due to the aggressive nature of systemd's parallelization which is a good thing for overall startup time.

Startup:

% systemd-analyze blame
  3499ms netcfg@ethernet-static.service
   695ms psd.service
   319ms ufw.service
   163ms arch32.service
   116ms systemd-tmpfiles-setup.service
   113ms systemd-modules-load.service
   103ms systemd-udev-trigger.service
    98ms lm_sensors.service
    93ms var.mount
    67ms cpupower.service
    56ms systemd-vconsole-setup.service
    50ms systemd-logind.service
    49ms ntpd.service
    46ms systemd-sysctl.service
    45ms mnt-data.mount
    43ms dev-hugepages.mount
    43ms systemd-remount-fs.service
    41ms tmp.mount
    34ms gpm.service
    33ms sys-kernel-debug.mount
    30ms polkit.service
    30ms dev-mqueue.mount
    30ms rtkit-daemon.service
    21ms systemd-udevd.service
    19ms systemd-user-sessions.service
    17ms home.mount
    15ms udisks2.service
     8ms upower.service
     3ms home-facade-.cache.mount
     2ms boot.mount
     0ms scratch.mount
     0ms sys-fs-fuse-connections.mount

Manual:

% systemd-analyze blame | grep psd
   349ms psd.service

This is syncing a single chromium profile of 81M:

% psd p
Profile-sync-daemon v5.08

Psd will manage the following per /etc/psd.conf settings:

 browser/psname:  chromium/chromium
 owner/group:     facade/users
 sync target:     /home/facade/.config/chromium
 tmpfs dir:       /tmp/facade-chromium
 profile size:    81M

Last edited by graysky (2012-12-04 12:26:14)

yminus · 2012-12-05 23:25:12

Whenever I reboot my system I am loosing part of my profile.

I'm syncing 1 firefox profile, psd is running as system service, my system is Gentoo Linux.

Version: 5.06

My profile is about 240MB in total:

  121,6MiB [##########] /Cache
   44,6MiB [###       ]  urlclassifier3.sqlite
   23,2MiB [#         ] /extensions
   10,0MiB [          ]  places.sqlite
    8,1MiB [          ] /bookmarkbackups
    5,7MiB [          ] /zotero
    5,6MiB [          ] /adblockplus
    4,5MiB [          ] /startupCache
    3,8MiB [          ]  xmarks-baseline-[...].json
    3,8MiB [          ] /safebrowsing
    1,4MiB [          ]  urlclassifier.pset
  708,0KiB [          ]  places.sqlite-wal
[...]

Syncing time seems to be ok:

time profile-sync-daemon sync

real    0m6.393s
user    0m2.249s
sys     0m2.318s

time profile-sync-daemon unsync

real    0m2.505s
user    0m0.113s
sys     0m0.325s

How can I isolate the problem?

Arch Linux

#251 2012-11-28 22:17:25

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#252 2012-11-28 23:06:48

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#253 2012-11-30 17:29:41

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#254 2012-11-30 22:27:03

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#255 2012-12-01 16:38:06

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#256 2012-12-01 23:59:14

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#257 2012-12-02 09:02:14

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#258 2012-12-02 16:39:50

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#259 2012-12-03 03:28:20

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#260 2012-12-03 07:28:03

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#261 2012-12-03 08:29:50

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#262 2012-12-03 12:20:44

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#263 2012-12-03 13:10:49

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#264 2012-12-03 13:37:40

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#265 2012-12-03 13:41:45

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#266 2012-12-03 17:12:32

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#267 2012-12-03 19:19:48

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#268 2012-12-03 23:12:29

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#269 2012-12-04 00:11:57

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#270 2012-12-04 03:11:07

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#271 2012-12-04 09:05:07

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#272 2012-12-04 10:06:47

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#273 2012-12-04 11:36:02

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#274 2012-12-04 12:22:48

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#275 2012-12-05 23:25:12

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

Board footer