profile-cleaner reduces browser profile size by cleaning/reindexing

graysky · 2012-11-28 23:19:07

OK... I just released an update to profile-cleaner. Main change is that I re-wrote it to use GNU parallel rather than the old for loops. It is FAST now with multiple profiles.

Here is a contrived example syncing 6 firefox profiles comparing the classical version of Profile-cleaner which used for loops to process profiles to the new version:
* There are a total of 312 M in the 6 "dirty" profiles.
* Test machine is an Intel 3770K @ 4.50 GHz.
* After cleaning there is a total of 213 M in the 6 "clean" profiles.
new 4.07s user 0.42s system 318% cpu 1.411 total
old 8.15s user 0.43s system 95% cpu 8.942 total
Your mileage may vary.

graysky · 2012-11-30 22:17:33

Version 1.84 is now colorized and shows total size recovered. Thanks to skydrome for the pretty color code.

Sample output and results using an "uncleaned" firefox profile:

maggie · 2012-12-01 23:45:10

The new colors are a nice touch!

jchristi · 2012-12-06 19:03:37

Script does not seem to work for me:

[jchristi@dhcp-88-12 ~]$ pc f
profile-cleaner v1.88

 Cleaning profile for firefox

grep: /tmp/accounting.firefox: No such file or directory
 Profile(s) for firefox reduced by  Mbytes.
[jchristi@dhcp-88-12 ~]$

My firefox profile is in ~/.mozilla/firefox and thunderbird is in ~/.thunderbird.

I'm running Fedora 17 not Arch. I imagine that is part or all of the problem. Any ideas on how to get it working for me? Thanks.

Last edited by jchristi (2012-12-06 19:05:28)

WorMzy · 2012-12-06 19:13:57

Hi, Graysky, could you add a '2>/dev/null' to the 'find /tmp ...' statement on line 10. e.g.

find /tmp -name 'work2do.*' -o -name 'accounting.*' -o -name "totalwork2do" 2>/dev/null | SHELL=/bin/bash parallel rm -f

I'm currently getting:

$ profile-cleaner f
find: `/tmp/tmux-1001': Permission denied
profile-cleaner v1.88

 Cleaning profile for firefox

and although it still works, the error is bugging me.

Cheers.

graysky · 2012-12-06 22:08:26

Done.

Bump to v1.89-1
Changelog: Less verbose output on errors that are meaningless.
Commit: https://github.com/graysky2/profile-cle … 2b965eb0f3

WorMzy · 2012-12-06 22:24:12

Not quite.

You've put the redirect after the pipe, so find still writes to STDERR.

$ profile-cleaner f
find: `/tmp/tmux-1001': Permission denied
profile-cleaner v1.89

 Cleaning profile for firefox

Cheers for the quick response though.

burninate · 2012-12-06 22:41:35

Nice script! Only, it doesn't seem to do anything with my profiles (and I have never touched them before):

~/build/profile-cleaner ⇨ LC_ALL=en_US.UTF-8 profile-cleaner c
find: `/tmp/go-mtpfs121813086': Permission denied
profile-cleaner v1.89

 Cleaning profile for chromium
 Cleaning Archived History                     done  -0 Mbytes
 Cleaning History Index 2012-11                done  -0 Mbytes
 Cleaning History Index 2012-09                done  -0 Mbytes
 Cleaning History-journal                      done  -0 Mbytes
 Cleaning History                              done  -0 Mbytes
 Cleaning Archived History-journal             done  -0 Mbytes
 Cleaning History Index 2012-12                done  -0 Mbytes
 Cleaning History Provider Cache               done  -0 Mbytes
 Cleaning History Index 2012-10                done  -0 Mbytes
 Cleaning Safe Browsing Download Whitelist     done  -0 Mbytes
 Cleaning Safe Browsing Bloom Filter 2         done  -0 Mbytes
 Cleaning Safe Browsing Csd Whitelist          done  -0 Mbytes
 Cleaning Safe Browsing Download               done  -0 Mbytes
 Cleaning Safe Browsing Bloom                  done  -0 Mbytes
 Cleaning Safe Browsing Cookies                done  -0 Mbytes

 Profile for chromium reduced by 0 Mbytes.

~/build/profile-cleaner ⇨ LC_ALL=en_US.UTF-8 profile-cleaner f
find: `/tmp/go-mtpfs121813086': Permission denied
profile-cleaner v1.89

 Cleaning profile for firefox
 Cleaning search.sqlite                        done  -0 Mbytes
 Cleaning index.sqlite                         done  -0 Mbytes
 Cleaning formhistory.sqlite                   done  -0 Mbytes
 Cleaning extensions.sqlite                    done  -0 Mbytes
 Cleaning chromeappsstore.sqlite               done  -0 Mbytes
 Cleaning addons.sqlite                        done  -0 Mbytes
 Cleaning signons.sqlite                       done  -0 Mbytes
 Cleaning content-prefs.sqlite                 done  -0 Mbytes
 Cleaning places.sqlite                        done  -0 Mbytes
 Cleaning permissions.sqlite                   done  -0 Mbytes
 Cleaning webappsstore.sqlite                  done  -0 Mbytes
 Cleaning downloads.sqlite                     done  -0 Mbytes
 Cleaning cookies.sqlite                       done  -0 Mbytes
 Cleaning urlclassifier3.sqlite                done  -0 Mbytes 
Profile(s) for firefox reduced by 0 Mbytes.

Also, that go-mtpfs file is related to my phone being mounted.

Last edited by burninate (2012-12-06 22:45:33)

graysky · 2012-12-06 22:46:05

Fek... haste makes waste. Take two:

Bump to v1.90-1
Changelog: Repositioned the 2>/dev/null to be more meaningful.
Commit: https://github.com/graysky2/profile-cle … 03761cd329

Send me a pull request next time

graysky · 2012-12-06 22:47:07

@burninate - Those warning should not appear in v1.90... I now not sure why you get nothing upon cleaning though...?

WorMzy · 2012-12-06 22:53:16

graysky wrote:

Fek... haste makes waste. Take two:
Bump to v1.90-1
Changelog: Repositioned the 2>/dev/null to be more meaningful.
Commit: https://github.com/graysky2/profile-cle … 03761cd329
Send me a pull request next time

I was considering it, but for the sake of 12 characters though, it hardly seemed worth it.

Updated package works fine, cheers.

burninate · 2012-12-07 12:36:45

@graysky
It upgraded profile-cleaner to v1.90 and chromium and firefox as well. Didn't change the behaviour. Could this have something to do with permissions or the like?

anonymous_user · 2012-12-07 13:58:28

Take a note of the sizes of Firefox's sqlite databases and see if this command has any effect:

$ cd ~/.mozilla/firefox/*.default
$ for i in *.sqlite; do sqlite3 $i vacuum; sqlite3 $i reindex; done

Last edited by anonymous_user (2012-12-07 14:00:39)

jchristi · 2012-12-07 14:45:50

Thanks anonymous_user, that's what I ended up doing and that works fine.

graysky · 2012-12-07 19:31:40

burninate wrote:

It upgraded profile-cleaner to v1.90 and chromium and firefox as well. Didn't change the behaviour. Could this have something to do with permissions or the like?

I don't think so since the script runs as your user. From the output you posted, it is working just fine. Did you see any change in profile size when you ran the for loop that anon suggested? I am guessing not.

Supplantr · 2012-12-08 04:58:24

I encountered a "(standard_in) 1: parse error" error. Noncritical, and possibly a fluke, but here it is nonetheless.

skydrome · 2012-12-08 08:45:03

the chromium part has not been right for a long time, use https://aur.archlinux.org/packages/browser-vacuum for a working example

graysky · 2012-12-08 11:39:05

@skydrome - I looked at your clean function which is a bit more complex than mine; I suspect that you are a more savvy bash scripter than I Can you suggest what is the source of the 'not been right' in mine using my style/grammar?

do_clean() {
  echo -en "${GRN} Cleaning${NRM} ${1##*/}"
  bsize=$(ls -l "$1" | gawk {'print $5'})
  sqlite3 "$1" vacuum
  sqlite3 "$1" reindex
  asize=$(ls -l "$1" | gawk {'print $5'})
  dsize=$(echo "scale=2; ($bsize-$asize)/1048576" | bc)
  echo -e "$(tput cr)$(tput cuf 46) ${GRN}done${NRM}  -${YLW}${dsize}${NRM} Mbytes"
  echo "$name,$dsize" >> /tmp/accounting.$name
}

BTW, thank you for the cool color code!

skydrome · 2012-12-08 19:21:29

i suspect the root of the problem is

find "$profilepath" -maxdepth 2 -type f -name 'Histor*' -o -name 'Archiv*' -o -name 'Safe*' >> /tmp/work2do.$name

what i do is simply

find . -maxdepth 1 -type f -print0
if [[ $(file "$db" | grep SQLite) ]]; then

no temp files, no external tools, just 1 at a time, simple
i think you've just made it far more complicated than it needs to be.

Last edited by skydrome (2012-12-08 19:26:23)

graysky · 2012-12-08 21:11:00

Supplantr wrote:

I encountered a "(standard_in) 1: parse error" error. Noncritical, and possibly a fluke, but here it is nonetheless.

Thanks... problem was the 'for' in my script which splits on spaces. Period. Regardless of quoting. Now I use 'read' which does not.

Plz try version 1.91-1
https://aur.archlinux.org/packages/profile-cleaner

...wtf does chromium use spaces in its profile anyway >:/

Last edited by graysky (2012-12-08 21:11:28)

Thorsten Reinbold · 2012-12-09 13:22:27

I'm using Ver. 1.91-1. I get this, with Google Chrome:

[nordlicht@Elias:~ $ profile-cleaner gc
profile-cleaner v1.91

 Cleaning profile for google-chrome
grep: /tmp/accounting.google-chrome: No such file or directory
 Profile for google-chrome reduced by  Mbytes.

Last edited by Thorsten Reinbold (2012-12-09 13:24:35)

graysky · 2012-12-09 13:47:09

Thorsten Reinbold wrote:

I'm using Ver. 1.91-1. I get this, with Google Chrome:

Opps, my bad. Fixed in v1.92-1. Plz download and verify that it works on your system (works on mine):

% pc gc
profile-cleaner v1.92

 Cleaning profile for google-chrome
 Cleaning Cookies                              done  -0 Mbytes
 Cleaning Archived History                     done  -0 Mbytes
 Cleaning Shortcuts                            done  -0 Mbytes
 Cleaning Login Data                           done  -0 Mbytes
 Cleaning Top Sites                            done  -0 Mbytes
 Cleaning History                              done  -0 Mbytes
 Cleaning Web Data                             done  -0 Mbytes
 Cleaning Favicons                             done  -0 Mbytes
 Cleaning Network Action Predictor             done  -0 Mbytes
 Profile for google-chrome reduced by 0 Mbytes.

Thorsten Reinbold · 2012-12-09 13:54:21

Works again! Thank you!

asdf-chan · 2012-12-09 20:09:10

The last update broke compatibility with hetfig-aurora.

λ cirno ~ → LANG=en_US.UTF-8 pc h
profile-cleaner v1.94

/usr/bin/pc: line 72: $profiles_extracted: ambiguous redirect
/usr/bin/pc: line 50: $profiles_extracted: ambiguous redirect
 Cleaning profile for heftig aurora
cat: /tmp/work2do.heftig: No such file or directory
cat: aurora: No such file or directory

grep: aurora: No such file or directory
grep: /tmp/accounting.heftig: No such file or directory
grep: aurora: No such file or directory
 Profile(s) for heftig aurora reduced by  Mbytes.

graysky · 2012-12-09 20:40:25

Bump to v1.95-1
Changelog: Fixed bug introduced by new code for aurora.
Commit: https://github.com/graysky2/profile-cle … a0ae120082

Plz try and report back; works for me.

Arch Linux

#26 2012-11-28 23:19:07

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#27 2012-11-30 22:17:33

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#28 2012-12-01 23:45:10

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#29 2012-12-06 19:03:37

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#30 2012-12-06 19:13:57

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#31 2012-12-06 22:08:26

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#32 2012-12-06 22:24:12

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#33 2012-12-06 22:41:35

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#34 2012-12-06 22:46:05

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#35 2012-12-06 22:47:07

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#36 2012-12-06 22:53:16

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#37 2012-12-07 12:36:45

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#38 2012-12-07 13:58:28

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#39 2012-12-07 14:45:50

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#40 2012-12-07 19:31:40

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#41 2012-12-08 04:58:24

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#42 2012-12-08 08:45:03

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#43 2012-12-08 11:39:05

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#44 2012-12-08 19:21:29

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#45 2012-12-08 21:11:00

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#46 2012-12-09 13:22:27

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#47 2012-12-09 13:47:09

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#48 2012-12-09 13:54:21

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#49 2012-12-09 20:09:10

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

#50 2012-12-09 20:40:25

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

Board footer