profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

ccom · 2012-01-26 11:53:41

I am not sure it's a good idea to mount .cache as tmpfs. Gnome stores its wallpaper there, and there is possibly other software using it as non-volatile cache.

graysky · 2012-01-26 14:36:25

ccom wrote:

Google Chrome also uses the directory ~/.cache/google-chrome. Is it possible/desirable to include it into psd?

https://wiki.archlinux.org/index.php/Pr … nd_Caveats

swanson · 2012-02-01 08:10:52

I have a strange problem of all my addons, like Adblock plus, needs reinstalling once a day. At least. I just have 3 addons ; Adblock, Chatzilla, Better Privacy, so it shouldn't be too heavy. This is the structure of /tmp/myuser-mozilla;

drwxrwxrwt 14 root    root    4,0K  1 feb 09.04 ..
drwxr-xr-x  5 .. .. 4,0K 10 feb  2011 extensions
drwxr-xr-x  5.. ..  4,0K  7 feb  2011 firefox
drwxr-xr-x  4 .. ..  4,0K  7 feb  2011 firefox-3.5.abandoned
-rw-r--r--  1 .. ..     0  1 feb 07.21 .flagged

(my user left out)

EDIT; there are 2 user dirs in the dir firefox, one old and one new. Migth it be the old one causing this?

Last edited by swanson (2012-02-01 08:18:34)

cdiamond · 2012-02-01 15:58:50

Hi graysky,
how to ensure everything is working?
psd in DAEMONS in rc.conf. I see /tmp/cdiamond-chromium folder created with chromium proifile.
Also I set /tmp/cache as cache dir for this browser. But when I am surfing HDD led still blinking.

graysky · 2012-02-01 21:31:08

@swanson - just for sanity. Do this:

1) Post your /etc/psd.conf
2) Stop psd (sudo rc.d stop psd)
3) Clean up ~/.mozilla
3b) Move out that "firefox-3.5.abandoned" as it will speed up your syncs and help to isolate the problem.

Here is my ~/.mozilla

$ ls -l ~/.mozilla/
total 12
-rw-r--r-- 1 facade users  335 Aug 13 15:40 appreg
drwxr-xr-x 5 facade users 4096 Dec  2 16:13 extensions
drwxr-xr-x 4 facade users 4096 Jan  2 08:59 firefox

4) Once cleaned up, start psd and report back.

@cdiamond - If you have /tmp/$USER-$BROWSER which you do, can you verify psd linked your ~/.config/chromium to the tmp space?

$ ls -lh ~/.config/chromium
lrwxrwxrwx 1 root root 21 Feb  1 16:00 /home/$USER/.config/chromium -> /tmp/$USER-chromium

If that is the case, install iotop and run it as root:

# iotop -Poa

Now begin using chromium and watch as iotop logs what programs are generating I/O.

swanson · 2012-02-02 07:24:14

@graysky; thx for your patience.

kalpik · 2012-02-06 02:14:59

This is really great! Thanks! Just one issue.. I've moved the chrome cache dir to .config/google-chrome/cache and it's successfully stored on /tmp by psd, but I don't think it retains the cache after a reboot. Anything I'm doing wrong?

graysky · 2012-02-06 03:53:21

What you describe should be just fine... Can you confirm that the cache dir is indeed empty on a reboot? Use something like ncdu to tell.

kalpik · 2012-02-06 04:41:23

I will check this once I'm home. At work right now.. But chrome seems to load all images etc again, that lead me to believe the cache isn't retained. Anyway, I'll confirm as soon as I can.

Thanks!

kalpik · 2012-02-06 16:26:21

Ok, got home and checked. psd *does* retain all the cache, but as soon as I start Chrome, it seems to delete everything and start fresh

EDIT: This only happens when I restart psd.. If psd is running, I can close/open Chrome as many times and it retains cache.

Last edited by kalpik (2012-02-06 16:27:54)

graysky · 2012-02-06 16:40:39

@kal - hmm... can't understand the behavior. Have a look at the code in /usr/bin/profile-sync-daemon which is very trivial.

Stop = unlink and rename back to ~/.config/google-chrome
Start = link and rename ~/.config/google-chrome

I actually use chromium and not chrome, and I start it intentionally moving the cache out of the $HOME entirely: chromium --enable-seccomp-sandbox --scroll-pixels=230 --memory-model=low --disk-cache-dir=/tmp/cache --user-agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/16.0.912.77 Safari/535.7"

...are you starting psd from /etc/rc.conf or manually?

kalpik · 2012-02-06 16:58:33

Yeah, that's what even I'm not understanding.. The way you're doing it, anyway it won't retain the cache..

I'm starting psd from rc.conf.

graysky · 2012-02-06 17:22:32

Very interesting... I just installed google chrome from the AUR in an attempt to reproduce your problem... I can't

So, psd is started, and I start google-chrome like this:

google-chrome --disk-cache-dir=/home/facade/.config/google-chrome/cache

I see the cache directory where I defined it, but since psd is active, there is a softlink in my homedir above --> /tmp/facade-google-chrome and in that dir is the cache dir which is full of files. If I stop psd, it behaves as expected: full cache dir under $HOME's dirtree. Starting it again behaves as expected: back to /tmp/facade-google-chrome with a full cache dir there... dunno what's up with it on your system... how did you move the cache dir as you stated?

kalpik wrote:

I've moved the chrome cache dir to .config/google-chrome/cache and it's successfully stored on /tmp by psd, but I don't think it retains the cache after a reboot.

Last edited by graysky (2012-02-06 17:23:00)

kalpik · 2012-02-06 17:25:15

Yes, I started chrome with the home dir argument *while* psd was running.. So the cache was created while the folder was symlinked to /tmp

I also get the full cache directory whether or not psd is running, but as soon as I start chrome after starting psd, chrome just "ignores" the cache, and starts afresh :s

graysky · 2012-02-06 17:37:53

That's very odd. I don't know what to say about it. Did you make a symlink to your cache dir or start the app with the --disk-cache-dir=/home/facade/.config/google-chrome/cache switch? How are you determining that the cache dir is being ignored?

kalpik · 2012-02-06 17:41:48

I started with --disk-cache-dir=/home/kalpik/.config/google-chrome/cache while psd was running so the folder was created in /tmp

Before I start chrome, I see about 50-60 different files/folder inside cache, and as soon as I start chrome, the cache decreases to some 6-8 files.. Also, chrome loads all images etc again..

kalpik · 2012-02-06 17:50:55

Ok, one more funny thing.. If I manually copy the cache directory somewhere, restart psd and copy back the cache directory (overriding the existing one), then Chrome does not ignore it! Could it be something to do with timestamps? Some sort of cache poisoning protection in Chrome?

graysky · 2012-02-06 18:55:49

@kalpik - No idea... good news from my view is that it doesn't seem to be related to my code

kalpik · 2012-02-07 01:53:22

graysky wrote:

@kalpik - No idea... good news from my view is that it doesn't seem to be related to my code

No problem I'll investigate a little more, and let you know if I manage to solve it

Niclas · 2012-02-18 15:10:17

I've got some strange problem with chromium.
I installed psd yesterday and it worked flawlessly. Today I discovered that the symlink wasn't there so I tried to start the daemon manually:

$ sudo /etc/rc.d/psd start
:: Starting Profile-Sync-Daemon [BUSY]
rsync: failed to set times on "/home/niclas/.config/chromium/niclas-chromium": Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1052) [sender=3.0.9]
[DONE]

Do I have to change some permissions in my local configuration of chromium to get this working?

[EDIT]
aww nevermind, I had chromium running while starting psd

But if I now try to stop psd (with chromium closed) I get this:

$ sudo /etc/rc.d/psd stop
:: Stopping Profile-Sync-Daemon [BUSY]
rsync: chgrp "/home/niclas/.config/chromium-backup/niclas-chromium" failed: Operation not permitted (1)
rsync: chgrp "/home/niclas/.config/chromium-backup/chromium-backup/niclas-chromium" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1052) [sender=3.0.9]

Last edited by Niclas (2012-02-18 15:14:15)

thetrivialstuff · 2012-02-19 02:56:34

I'm toying with a similar idea to this, that takes this concept a bit further. In my home directory, I have the two directories "._tmp" and "._tmpsync".

"._tmp" has empty directories for stuff I never want/need to persist across reboots -- cookies directories, .cache-type things, that .macromedia folder, font caches, etc. It also has "templates" for configs that don't change often enough to need rsyncing back after use. For example, I use Opera as my main browser, but occasionally pop into Firefox for something. I have Firefox set up the way I like it, but I don't care at all about keeping any new settings, history, bookmarks, etc. across reboots. So my "pristine" Firefox profile is in here, and this copy never gets written to.

"._tmpsync" behaves the way profile-sync-daemon treats browser profiles -- it gets copied to /tmp on login, synced periodically back to disk as I'm using it. That is, hourly, and also every time I log off, shut down, or do anything "dangerous" like put the netbook to sleep (I have been known to forget that it's only sleeping and do a battery swap :P). (-- Does profile-sync-daemon do this? If not, it would be a neat feature request, to tie it into e.g. acpid's on-sleep triggers.)

In "._tmpsync" I have not just my browser profile, but also my mail client's profile (which sees a lot of writes to the IMAP cache when it's running), and my .kde directory (things like konqueror will often rewrite their whole config file even if you haven't changed any settings, because I think it stores some recently-used-URL lists in there) and some other things.

Adding new things to either directory is easy -- just drag them from my homedir to the container in /tmp/, symlink from its original location, and it gets synched back on the next periodic rsync and ever after in the future. To add new things to the one-way "._tmp" directory, copy them there, then also into the /tmp/ container, and symlink -- then ever after, it gets refreshed to a pristine copy on login.

Note that you also don't need to add a whole directory, and you can mix & match parts between ._tmp and ._tmpsync -- for instance, for those of you who do store mail locally, you could leave ~/.thunderbird as a normal directory in ~/, but then go into it and move the cache-like parts to ._tmpsync. Or, if you archive your mail by year, you could put only the current inbox into .tmpsync (you can do this at the per-file level) and leave everything else in place.

Showing all of this off, here's my current Opera setup:

$ ls -l .opera | cut -d " " -f 9-
.opera -> /tmp/._felix_tmpsync/.opera

$ find .opera/ -type l | xargs ls -l | cut -d " " -f 9-
.opera/cache -> /tmp/._felix/opera/cache
.opera/cookies4.dat -> /tmp/._felix/opera/cookies4.dat
.opera/icons -> /tmp/._felix/opera/icons
.opera/opcache -> /tmp/._felix/opera/opcache
.opera/pstorage -> /tmp/._felix/opera/pstorage
.opera/temporary_downloads -> /tmp/._felix/opera/temporary_downloads
.opera/widgets -> /tmp/._felix/opera/widgets

(OK, I lied a bit about the names above -- when they're in /tmp/, I name the containers with my username because I'm doing the same thing for multiple users.)

cache, cookies4.dat,* icons, opcache, pstorage, and temporary_downloads are all empty files/directories in ._tmp/. The 'widgets' directory seems to like to pick up cruft, even when I'm not using those widgets -- I have only one widget that I actually use -- so it starts life empty of all but that one widget and its config file. If opera wants to write cruft to it while it's running, fine; it won't be there next boot :)

* it seems that opera doesn't like cookies4 being a symlink, and overwrites it with a real file when you quit. I know opera has a "dump new cookies on exit" setting, which I use, but strangely a few cookies do seem to persist in this file (Google seems particular good at making lasting cookies). I might have to just ignore it, or exclude it from rsync, but that kind of kills the generality a bit if rsync has to know about particular files/directories...

~Felix.

thetrivialstuff · 2012-02-19 03:08:47

Ooh, ooh -- replying to myself, but I just thought of an addendum: I'm using btrfs now, so what I'm gonna do is combine the periodic resynching with btrfs snapshot-taking, since I wanted to set up a script to do that regularly anyway.

So, if we schedule a btrfs snapshot the instant before each periodic sync, then that means that if something messes up on the rsync and part of ~/._tmpsync gets nuked we have an automatic ability to roll back and rescue it :)

~Felix.

Last edited by thetrivialstuff (2012-02-19 03:09:12)

qnd · 2012-02-27 04:11:06

Very useful and well done, I like how transparent and easy to use it is. Having a good wiki page is great too.

However, I had the same issue as neuromancer85 on page one (not using systemd though, just added as a normal daemon) with some addons not loading. All preferences for Firefox and addons were intact however.

I didn't do extensive testing, but it seemed to happen whenever I launched my browser quickly after booting. (If it's of any importance I have the OS on an SSD with /home and /var residing on HDD partitions.) The daemon was set to start in the background (following the example in the wiki article), so I set it to foreground (which adds a couple of seconds to boot), and I haven't had the issue since.

So I'm thinking launching the browser before the daemon has fully finished starting could be the cause of troubles like these, and it might be worth considering to set it to start in the foreground, if only just in case.

graysky · 2012-02-27 10:19:33

I also suspect it is because you backgrounded the daemon and the sync didn't complete as you launched it. I'll add a note on the wiki about this. Thank you for your post.

qnd · 2012-03-05 22:38:05

Actually, foregrounding the daemon adds a rather hefty 30 seconds to boot. Is there something that could be done about it, or will it have to be an unfortunate side effect of enjoying the benefits of this daemon, problem-free?

Arch Linux

#51 2012-01-26 11:53:41

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#52 2012-01-26 14:36:25

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#53 2012-02-01 08:10:52

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#54 2012-02-01 15:58:50

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#55 2012-02-01 21:31:08

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#56 2012-02-02 07:24:14

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#57 2012-02-06 02:14:59

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#58 2012-02-06 03:53:21

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#59 2012-02-06 04:41:23

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#60 2012-02-06 16:26:21

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#61 2012-02-06 16:40:39

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#62 2012-02-06 16:58:33

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#63 2012-02-06 17:22:32

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#64 2012-02-06 17:25:15

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#65 2012-02-06 17:37:53

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#66 2012-02-06 17:41:48

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#67 2012-02-06 17:50:55

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#68 2012-02-06 18:55:49

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#69 2012-02-07 01:53:22

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#70 2012-02-18 15:10:17

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#71 2012-02-19 02:56:34

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#72 2012-02-19 03:08:47

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#73 2012-02-27 04:11:06

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#74 2012-02-27 10:19:33

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#75 2012-03-05 22:38:05

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

Board footer