profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

graysky · 2023-01-25 10:20:41

Thanks for reporting and for the fix. Added.

ron9 · 2023-02-18 17:51:21

Hello

I found profile-sync-daemon for MS edge on AUR. But it didn't work.
/usr/share/psd/browsers show both microsoft-edge-beta and -dev. But nothing in psd.conf.
Sync with psd.service didn't change anything.
It would be nice if you could help me get this to work!
MS edge is based on chromium.
It is MS and I don't like that but I like the browser.

Ron9

graysky · 2023-02-18 19:41:06

If edge is based on chrome just use a drop-in: https://github.com/graysky2/profile-syn … psd.1#L156

You should not need an AUR package/psd is in [community]

ron9 · 2023-02-18 22:26:10

Sorry. This was a AUR package just to get Edge in psd.conf in your profile-sync-daemon. I have used PSD with Falcon for a long time.
I just don't now how to get Edge in browser list in psd.conf...

Ron9

graysky · 2023-02-19 11:06:21

Just edit it once you have the drop-in

ron9 · 2023-02-19 22:40:00

Did the copy and edit chromium - edge, restart psd = ok
Thank you working!

Ron9

graysky · 2023-03-19 18:04:45

I recently dropped the sudo requirement and am using fuse-overlayfs to the overlayfs mount. I'd like some testers to bang on it. You can download the dev version from this AUR package: https://aur.archlinux.org/packages/prof … daemon-git

As always, make a backup of your profiles first!

gblargg · 2023-03-22 17:37:40

BTW rsync by default doesn't seem to align the delta updates to files with the block size. Add --block-size=1024 to help ensure this (it defaults to a fraction of the file, e.g. an arbitrary value like 700 bytes). You can run with -vvv and look for "match at" lines. It still can become unaligned when things are inserted or removed from the middle of a file and matches are smaller but I don't know hot to help in that case.

Example to see the blocks (using a smaller non-power-of-2 block size for clarity):

dd if=/dev/urandom of=in bs=1 count=1000
cp in out
# change two sections of file
dd if=/dev/urandom of=in bs=1 count=10 seek=450 conv=notrunc
dd if=/dev/urandom of=in bs=1 count=10 seek=850 conv=notrunc
# ignore-times needed when running lots of tests quickly and they get same timestamp
rsync -vvv --ignore-times --no-whole-file --block-size=100 in out

built hash table
hash search b=100 len=1000
match at 0 last_match=0 j=0 len=100 n=0
match at 100 last_match=100 j=1 len=100 n=0
match at 200 last_match=200 j=2 len=100 n=0
match at 300 last_match=300 j=3 len=100 n=0
match at 500 last_match=400 j=5 len=100 n=100
match at 600 last_match=600 j=6 len=100 n=0
match at 700 last_match=700 j=7 len=100 n=0
match at 900 last_match=800 j=9 len=100 n=100

Last edited by gblargg (2023-03-22 18:07:15)

graysky · 2023-03-24 15:58:07

What would specifying a value for block-size do functionally? What is wrong without doing it? Not much on the option in the man page.

man rsync wrote:

--block-size=SIZE, -B
This forces the block size used in rsync's delta-transfer algorithm to a fixed value. It is normally selected based on the size of each file being updated. See the technical report for details.
Beginning in 3.2.3 the SIZE can be specified with a suffix as detailed in the --max-size option. Older versions only accepted a byte count.

gblargg · 2023-03-25 19:00:49

graysky wrote:

What would specifying a value for block-size do functionally? What is wrong without doing it? Not much on the option in the man page.

To be sure we're on the same page, I've establshed some context first.

The tool is partly to reduce wear on drives (writes to SSDs) as browsers are notorious for this. When updating the copy on the SSD, you'd like to write only the changed portions of a file. Changes come into basic types: insertion/deletion, and mere change in place. Insertion/deletion naturally shift everything after in the file, requiring a rewrite of everything after that point, so there's little that can be done to reduce writes in this case. But change in place only requires rewriting the SSD pages containing the changes.

Rsync has an in-place delta update mode that breaks the destination file into a series of blocks of equal size, then skips rewriting any that need no change. By default the block size is a fraction of the file, down to 700 bytes at minimum, and up to 131K. Rsync's selection of block size is optimized to reduce network traffic when updating over a slow link.

The default block size is almost always either not an even division of the SSD page size, or larger than an SSD page size. This means that even when a single byte in a block was changed, it can overlap two or more SSD pages, causing unnecessary writes (write amplification). By using a block size that is a clean division of a page size (e.g. 512, 1024, 2048, 4096), this ensures that this amplification won't occur; no blocks in the destination file will overlap SSD page boundaries.

One caveat, I haven't looked into detail how you handle multiple snapshots. If they use hardlinks (to allow sharing of data between snapshots) I don't think rsync could do delta updates in place, since those would modify the same file in other snapshots if they linked to it.

This page by the rsync author has a nice description of the algorithm: https://rsync.samba.org/tech_report/

koseci1949 · 2023-04-01 12:14:55

Hello,

I've installed the package this morning, that's the output of `journalctl --user-unit psd.service`:

Apr 01 11:19:08 thinkpad systemd[982]: Starting Profile-sync-daemon...
Apr 01 11:19:08 thinkpad profile-sync-daemon[989]: psd startup check successful
Apr 01 11:19:08 thinkpad systemd[982]: Finished Profile-sync-daemon.

And that's `psd preview`:

Profile-sync-daemon v6.48

 systemd service: active
 resync-timer:    active
 sync on sleep:   disabled
 use overlayfs:   disabled

Psd will manage the following per /home/anon/.config/psd/.psd.conf:

 browser/psname:  firefox/firefox
 owner/group id:  anon/1000
 sync target:     /home/anon/.mozilla/firefox/d1ef0kop.default
 tmpfs dir:       /run/user/1000/psd/anon-firefox-d1ef0kop.default
 profile size:    13M
 recovery dirs:   none

 browser/psname:  firefox/firefox
 owner/group id:  anon/1000
 sync target:     /home/anon/.mozilla/firefox/4aogg4vd.default-release
 tmpfs dir:       /run/user/1000/psd/anon-firefox-4aogg4vd.default-release
 profile size:    196M
 recovery dirs:   none

My browser is Firefox, how can I verify that profile is on RAM?

FarStar07 · 2023-04-18 11:39:38

Hello graysky,

Is it possible to support LibreWolf (a Firefox fork)? => https://librewolf.net
The profile is located in ~/.librewolf/

Thank you very much and have a nice day.

Last edited by FarStar07 (2023-04-18 12:02:57)

graysky · 2023-04-18 12:22:24

Try a dropin... see one of the many closed issues on the project github.

FarStar07 · 2023-04-18 14:02:33

Thank you graysky. It works perfectly!

thoth · 2023-08-14 20:45:12

@graysky I seem to be losing logins to many sites after rebooting, and I have to re - login again, whereas most of the time the login sessions are retained when I am not using profile-sync-daemon, is this by design? Can these be retained somehow while still gaining the reduced ssd writes and speed?

graysky · 2023-08-26 12:43:37

No, something changed with recent chromium release. I too am not able to have passwords get saved.

Utini · 2023-08-31 10:51:00

Thanks for this!

I came upon this by checking performance loss with LUKS2 on modern & fast NVME's.
Now I am trying to reduce the workload on my NVME and used PSD + "browser.cache.disk.parent_directory" to move Firefox into RAM.
Additionally ZRAM (in RAM) instead of ZSWAP.
Because why not make use of those 32GB of RAM

SidMan · 2023-12-19 19:07:02

Is there a way to only sync on shutdown ? Disable hourly sync ?

agapito · 2023-12-19 23:55:19

SidMan wrote:

Is there a way to only sync on shutdown ? Disable hourly sync ?

https://wiki.archlinux.org/title/Profil … _intervals

Last edited by agapito (2023-12-19 23:55:48)

SidMan · 2023-12-20 04:30:04

It shows how to set interval, not how to disable it completely.

agapito · 2023-12-20 09:00:55

SidMan wrote:

It shows how to set interval, not how to disable it completely.

If you set 1000000 minutes, it is like deactivating it.

glitsj16 · 2023-12-20 11:16:29

@SidMan To completely disable the resync timer you could copy /usr/lib/systemd/user/psd.service to /etc/systemd/system/user/psd.service and remove (or comment) the `Wants=psd-resync.service` line. Probably best to quit all browsers, stop psd.service, disable psd & psd-resync.service and re-enable psd after making that change.

graysky · 2024-01-20 20:25:41

graysky wrote:

No, something changed with recent chromium release. I too am not able to have passwords get saved.

This was not related to chromium but to a corrupted database, see: https://wiki.archlinux.org/title/Chromi … d_database

Arch Linux

#901 2023-01-25 10:20:41

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#902 2023-02-18 17:51:21

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#903 2023-02-18 19:41:06

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#904 2023-02-18 22:26:10

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#905 2023-02-19 11:06:21

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#906 2023-02-19 22:40:00

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#907 2023-03-19 18:04:45

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#908 2023-03-22 17:37:40

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#909 2023-03-24 15:58:07

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#910 2023-03-25 19:00:49

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#911 2023-04-01 12:14:55

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#912 2023-04-18 11:39:38

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#913 2023-04-18 12:22:24

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#914 2023-04-18 14:02:33

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#915 2023-08-14 20:45:12

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#916 2023-08-26 12:43:37

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#917 2023-08-31 10:51:00

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#918 2023-12-19 19:07:02

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#919 2023-12-19 23:55:19

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#920 2023-12-20 04:30:04

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#921 2023-12-20 09:00:55

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#922 2023-12-20 11:16:29

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

#923 2024-01-20 20:25:41

Re: profile-sync-daemon - keep browser profiles in tmpfs and sync'ed

Board footer