You are not logged in.

#26 2012-11-28 23:19:07

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,645
Website

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

OK... I just released an update to profile-cleaner.  Main change is that I re-wrote it to use GNU parallel rather than the old for loops.  It is FAST now with multiple profiles.

Here is a contrived example syncing 6 firefox profiles comparing the classical version of Profile-cleaner which used for loops to process profiles to the new version:

* There are a total of 312 M in the 6 "dirty" profiles.
* Test machine is an Intel 3770K @ 4.50 GHz.
* After cleaning there is a total of 213 M in the 6 "clean" profiles.

new 4.07s user 0.42s system 318% cpu 1.411 total
old 8.15s user 0.43s system 95% cpu 8.942 total

Your mileage may vary.


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#27 2012-11-30 22:17:33

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,645
Website

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

Version 1.84 is now colorized and shows total size recovered.  Thanks to skydrome for the pretty color code.

Sample output and results using an "uncleaned" firefox profile:
screen.jpg


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#28 2012-12-01 23:45:10

maggie
Member
Registered: 2011-02-12
Posts: 255

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

The new colors are a nice touch!

Offline

#29 2012-12-06 19:03:37

jchristi
Member
Registered: 2012-12-06
Posts: 2

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

Script does not seem to work for me:

[jchristi@dhcp-88-12 ~]$ pc f
profile-cleaner v1.88

 Cleaning profile for firefox

grep: /tmp/accounting.firefox: No such file or directory
 Profile(s) for firefox reduced by  Mbytes.
[jchristi@dhcp-88-12 ~]$

My firefox profile is in ~/.mozilla/firefox and thunderbird is in ~/.thunderbird.

I'm running Fedora 17 not Arch. I imagine that is part or all of the problem. Any ideas on how to get it working for me? Thanks.

Last edited by jchristi (2012-12-06 19:05:28)

Offline

#30 2012-12-06 19:13:57

WorMzy
Administrator
From: Scotland
Registered: 2010-06-16
Posts: 12,417
Website

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

Hi, Graysky, could you add a '2>/dev/null' to the 'find /tmp ...' statement on line 10. e.g.

find /tmp -name 'work2do.*' -o -name 'accounting.*' -o -name "totalwork2do" 2>/dev/null | SHELL=/bin/bash parallel rm -f

I'm currently getting:

$ profile-cleaner f
find: `/tmp/tmux-1001': Permission denied
profile-cleaner v1.88

 Cleaning profile for firefox

and although it still works, the error is bugging me.

Cheers.


Sakura:-
Mobo: MSI MAG X570S TORPEDO MAX // Processor: AMD Ryzen 9 5950X @4.9GHz // GFX: AMD Radeon RX 5700 XT // RAM: 32GB (4x 8GB) Corsair DDR4 (@ 3000MHz) // Storage: 1x 3TB HDD, 6x 1TB SSD, 2x 120GB SSD, 1x 275GB M2 SSD

Making lemonade from lemons since 2015.

Offline

#31 2012-12-06 22:08:26

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,645
Website

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

Done.

Bump to v1.89-1

Changelog: Less verbose output on errors that are meaningless.
Commit: https://github.com/graysky2/profile-cle … 2b965eb0f3


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#32 2012-12-06 22:24:12

WorMzy
Administrator
From: Scotland
Registered: 2010-06-16
Posts: 12,417
Website

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

Not quite. tongue

You've put the redirect after the pipe, so find still writes to STDERR.

$ profile-cleaner f
find: `/tmp/tmux-1001': Permission denied
profile-cleaner v1.89

 Cleaning profile for firefox

Cheers for the quick response though.


Sakura:-
Mobo: MSI MAG X570S TORPEDO MAX // Processor: AMD Ryzen 9 5950X @4.9GHz // GFX: AMD Radeon RX 5700 XT // RAM: 32GB (4x 8GB) Corsair DDR4 (@ 3000MHz) // Storage: 1x 3TB HDD, 6x 1TB SSD, 2x 120GB SSD, 1x 275GB M2 SSD

Making lemonade from lemons since 2015.

Offline

#33 2012-12-06 22:41:35

burninate
Member
Registered: 2012-07-28
Posts: 22

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

Nice script! Only, it doesn't seem to do anything with my profiles (and I have never touched them before):

~/build/profile-cleaner ⇨ LC_ALL=en_US.UTF-8 profile-cleaner c
find: `/tmp/go-mtpfs121813086': Permission denied
profile-cleaner v1.89

 Cleaning profile for chromium
 Cleaning Archived History                     done  -0 Mbytes
 Cleaning History Index 2012-11                done  -0 Mbytes
 Cleaning History Index 2012-09                done  -0 Mbytes
 Cleaning History-journal                      done  -0 Mbytes
 Cleaning History                              done  -0 Mbytes
 Cleaning Archived History-journal             done  -0 Mbytes
 Cleaning History Index 2012-12                done  -0 Mbytes
 Cleaning History Provider Cache               done  -0 Mbytes
 Cleaning History Index 2012-10                done  -0 Mbytes
 Cleaning Safe Browsing Download Whitelist     done  -0 Mbytes
 Cleaning Safe Browsing Bloom Filter 2         done  -0 Mbytes
 Cleaning Safe Browsing Csd Whitelist          done  -0 Mbytes
 Cleaning Safe Browsing Download               done  -0 Mbytes
 Cleaning Safe Browsing Bloom                  done  -0 Mbytes
 Cleaning Safe Browsing Cookies                done  -0 Mbytes

 Profile for chromium reduced by 0 Mbytes.
~/build/profile-cleaner ⇨ LC_ALL=en_US.UTF-8 profile-cleaner f
find: `/tmp/go-mtpfs121813086': Permission denied
profile-cleaner v1.89

 Cleaning profile for firefox
 Cleaning search.sqlite                        done  -0 Mbytes
 Cleaning index.sqlite                         done  -0 Mbytes
 Cleaning formhistory.sqlite                   done  -0 Mbytes
 Cleaning extensions.sqlite                    done  -0 Mbytes
 Cleaning chromeappsstore.sqlite               done  -0 Mbytes
 Cleaning addons.sqlite                        done  -0 Mbytes
 Cleaning signons.sqlite                       done  -0 Mbytes
 Cleaning content-prefs.sqlite                 done  -0 Mbytes
 Cleaning places.sqlite                        done  -0 Mbytes
 Cleaning permissions.sqlite                   done  -0 Mbytes
 Cleaning webappsstore.sqlite                  done  -0 Mbytes
 Cleaning downloads.sqlite                     done  -0 Mbytes
 Cleaning cookies.sqlite                       done  -0 Mbytes
 Cleaning urlclassifier3.sqlite                done  -0 Mbytes 
Profile(s) for firefox reduced by 0 Mbytes.

Also, that go-mtpfs file is related to my phone being mounted.

Last edited by burninate (2012-12-06 22:45:33)

Offline

#34 2012-12-06 22:46:05

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,645
Website

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

Fek... haste makes waste.  Take two:

Bump to v1.90-1

Changelog: Repositioned the 2>/dev/null to be more meaningful.
Commit: https://github.com/graysky2/profile-cle … 03761cd329

Send me a pull request next time tongue


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#35 2012-12-06 22:47:07

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,645
Website

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

@burninate - Those warning should not appear in v1.90... I now not sure why you get nothing upon cleaning though...?


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#36 2012-12-06 22:53:16

WorMzy
Administrator
From: Scotland
Registered: 2010-06-16
Posts: 12,417
Website

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

graysky wrote:

Fek... haste makes waste.  Take two:

Bump to v1.90-1

Changelog: Repositioned the 2>/dev/null to be more meaningful.
Commit: https://github.com/graysky2/profile-cle … 03761cd329

Send me a pull request next time tongue

I was considering it, but for the sake of 12 characters though, it hardly seemed worth it. tongue

Updated package works fine, cheers.


Sakura:-
Mobo: MSI MAG X570S TORPEDO MAX // Processor: AMD Ryzen 9 5950X @4.9GHz // GFX: AMD Radeon RX 5700 XT // RAM: 32GB (4x 8GB) Corsair DDR4 (@ 3000MHz) // Storage: 1x 3TB HDD, 6x 1TB SSD, 2x 120GB SSD, 1x 275GB M2 SSD

Making lemonade from lemons since 2015.

Offline

#37 2012-12-07 12:36:45

burninate
Member
Registered: 2012-07-28
Posts: 22

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

@graysky
It upgraded profile-cleaner to v1.90 and chromium and firefox as well. Didn't change the behaviour. Could this have something to do with permissions or the like?

Offline

#38 2012-12-07 13:58:28

anonymous_user
Member
Registered: 2009-08-28
Posts: 3,059

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

Take a note of the sizes of Firefox's sqlite databases and see if this command has any effect:

$ cd ~/.mozilla/firefox/*.default
$ for i in *.sqlite; do sqlite3 $i vacuum; sqlite3 $i reindex; done

Last edited by anonymous_user (2012-12-07 14:00:39)

Offline

#39 2012-12-07 14:45:50

jchristi
Member
Registered: 2012-12-06
Posts: 2

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

Thanks anonymous_user, that's what I ended up doing and that works fine.

Offline

#40 2012-12-07 19:31:40

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,645
Website

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

burninate wrote:

It upgraded profile-cleaner to v1.90 and chromium and firefox as well. Didn't change the behaviour. Could this have something to do with permissions or the like?

I don't think so since the script runs as your user.  From the output you posted, it is working just fine.  Did you see any change in profile size when you ran the for loop that anon suggested?  I am guessing not.


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#41 2012-12-08 04:58:24

Supplantr
Member
From: a state of sunshine
Registered: 2011-12-12
Posts: 149
Website

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

I encountered a "(standard_in) 1: parse error" error. Noncritical, and possibly a fluke, but here it is nonetheless.
vZ2x6NQ


I use linux and I dont understand nothing in this post.

Offline

#42 2012-12-08 08:45:03

skydrome
Member
From: The Soviet States of Legos
Registered: 2012-09-03
Posts: 8
Website

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

the chromium part has not been right for a long time, use https://aur.archlinux.org/packages/browser-vacuum for a working example

Offline

#43 2012-12-08 11:39:05

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,645
Website

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

@skydrome - I looked at your clean function which is a bit more complex than mine; I suspect that you are a more savvy bash scripter than I smile  Can you suggest what is the source of the 'not been right' in mine using my style/grammar?

do_clean() {
  echo -en "${GRN} Cleaning${NRM} ${1##*/}"
  bsize=$(ls -l "$1" | gawk {'print $5'})
  sqlite3 "$1" vacuum
  sqlite3 "$1" reindex
  asize=$(ls -l "$1" | gawk {'print $5'})
  dsize=$(echo "scale=2; ($bsize-$asize)/1048576" | bc)
  echo -e "$(tput cr)$(tput cuf 46) ${GRN}done${NRM}  -${YLW}${dsize}${NRM} Mbytes"
  echo "$name,$dsize" >> /tmp/accounting.$name
}

BTW, thank you for the cool color code!


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#44 2012-12-08 19:21:29

skydrome
Member
From: The Soviet States of Legos
Registered: 2012-09-03
Posts: 8
Website

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

i suspect the root of the problem is

find "$profilepath" -maxdepth 2 -type f -name 'Histor*' -o -name 'Archiv*' -o -name 'Safe*' >> /tmp/work2do.$name

what i do is simply

find . -maxdepth 1 -type f -print0
if [[ $(file "$db" | grep SQLite) ]]; then

no temp files, no external tools, just 1 at a time, simple
i think you've just made it far more complicated than it needs to be.

Last edited by skydrome (2012-12-08 19:26:23)

Offline

#45 2012-12-08 21:11:00

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,645
Website

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

Supplantr wrote:

I encountered a "(standard_in) 1: parse error" error. Noncritical, and possibly a fluke, but here it is nonetheless.

Thanks... problem was the 'for' in my script which splits on spaces. Period. Regardless of quoting.  Now I use 'read' which does not.

Plz try version 1.91-1
https://aur.archlinux.org/packages/profile-cleaner

...wtf does chromium use spaces in its profile anyway >:/

Last edited by graysky (2012-12-08 21:11:28)


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#46 2012-12-09 13:22:27

Thorsten Reinbold
Member
From: Germany
Registered: 2011-12-06
Posts: 353

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

I'm using Ver. 1.91-1. I get this, with Google Chrome:

[nordlicht@Elias:~ $ profile-cleaner gc
profile-cleaner v1.91

 Cleaning profile for google-chrome
grep: /tmp/accounting.google-chrome: No such file or directory
 Profile for google-chrome reduced by  Mbytes.

Last edited by Thorsten Reinbold (2012-12-09 13:24:35)

Offline

#47 2012-12-09 13:47:09

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,645
Website

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

Thorsten Reinbold wrote:

I'm using Ver. 1.91-1. I get this, with Google Chrome:

Opps, my bad. Fixed in v1.92-1. Plz download and verify that it works on your system (works on mine):

% pc gc
profile-cleaner v1.92

 Cleaning profile for google-chrome
 Cleaning Cookies                              done  -0 Mbytes
 Cleaning Archived History                     done  -0 Mbytes
 Cleaning Shortcuts                            done  -0 Mbytes
 Cleaning Login Data                           done  -0 Mbytes
 Cleaning Top Sites                            done  -0 Mbytes
 Cleaning History                              done  -0 Mbytes
 Cleaning Web Data                             done  -0 Mbytes
 Cleaning Favicons                             done  -0 Mbytes
 Cleaning Network Action Predictor             done  -0 Mbytes
 Profile for google-chrome reduced by 0 Mbytes.

CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#48 2012-12-09 13:54:21

Thorsten Reinbold
Member
From: Germany
Registered: 2011-12-06
Posts: 353

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

Works again! Thank you! smile

Offline

#49 2012-12-09 20:09:10

asdf-chan
Member
Registered: 2011-10-23
Posts: 31

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

The last update broke compatibility with hetfig-aurora.

λ cirno ~ → LANG=en_US.UTF-8 pc h
profile-cleaner v1.94

/usr/bin/pc: line 72: $profiles_extracted: ambiguous redirect
/usr/bin/pc: line 50: $profiles_extracted: ambiguous redirect
 Cleaning profile for heftig aurora
cat: /tmp/work2do.heftig: No such file or directory
cat: aurora: No such file or directory

grep: aurora: No such file or directory
grep: /tmp/accounting.heftig: No such file or directory
grep: aurora: No such file or directory
 Profile(s) for heftig aurora reduced by  Mbytes.

Offline

#50 2012-12-09 20:40:25

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,645
Website

Re: profile-cleaner reduces browser profile size by cleaning/reindexing

Bump to v1.95-1

Changelog: Fixed bug introduced by new code for aurora.
Commit: https://github.com/graysky2/profile-cle … a0ae120082

Plz try and report back; works for me.


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

Board footer

Powered by FluxBB