defragmentation of /var/lib/pacman/ (reiserfs)

dp · 2005-04-21 18:17:22

on my laptop i run arch from 0.4 and since then i have over 1200 packages installed. i used reiserfs and realized now, that it accumulated a certain amount of fragmentation over time and is now accessing /var/lib/pacman/ very slowly (that's why pacman sometimes takes very long do to anything) ... coupled to this, the hdd on this laptop is not the fastest ...

now i just for fun tried something i didn't believe really that it would help but it helped a lot (pacman is now much faster) - what i did:

[root@Asteraceae lib]# pwd
/var/lib

[root@Asteraceae lib]# cp -r pacman/ pacman_cpy/
[root@Asteraceae lib]# rm -r pacman
[root@Asteraceae lib]# mv pacman_cpy/ pacman/

this defragmented the big collection of small files stored across my hdd to one place and now pacman is much faster

phrakture · 2005-04-21 18:39:46

wow... interesting... but it makes sense...

iBertus · 2005-04-22 00:27:57

never thought about doing that before. i'll give it a try when i have a change later tonight.

apeiro · 2005-04-22 22:19:07

Wow, that's a good (and simple!) idea, Damir.

It might be a good thing to include as a pacman option itself.

pacman --dbmaint or some such.

T-Dawg · 2005-04-23 00:23:25

monthly cron job?

celeron2002 · 2005-04-23 01:02:44

this works very good, but this is a reiserf's problem or it is something normal in any filesystem?

T-Dawg · 2005-04-23 01:10:17

All file systems have a difficult time with lots of small files. Ext3 fairs a little better than Reiserfs from what I've heard..
I'm curious to see how Reiser4 deals with them.

phrakture · 2005-04-23 01:25:43

woah, this was actually a drastic difference...

I tried searching for something... then did the little mv/cp thing... and searched again... it's like 10x faster to search now...

WTG dp!

dp · 2005-04-23 08:38:45

nice to see that this helps also others ...

i measured before and after conditions on another machine runnign arch from 0.4 (>2years) and the speed is significantly better.

i measured (by hand) from the message "Top-level DB Path: /var/lib/pacman" in verbose pacman, to results of e.g. a -Ss search (verbosity enabled: -v)

before copying db-files to new place on hdd:
12min 33sec

after copying:
0min 5sec

that's what i mean by significant ;-)

btw: what does WTG mean?

dp · 2005-04-23 08:41:01

Penguin wrote:

monthly cron job?

once a year or so would be enough frequent

EDIT: if doing this copying automagically, i suggest you to have the lock-file of pacman in temp, so that pacman knows that it cannot use the db for a short while (while this cp/rm/mv is working)

dp · 2005-04-23 08:52:22

apeiro wrote:

Wow, that's a good (and simple!) idea, Damir.
It might be a good thing to include as a pacman option itself.
pacman --dbmaint or some such.

thank you. this really helped me with speed of pacman but i feel it more a workaround (a very effective one!) than the solution. however, as it really solves the problem, i can live with this workaround ;-)

yet another idea towards the solution i can think of now is to use a tarball to store everything and using a ramdrive while working with these files (extract everything from tarball to ramdrive and mount it @ /var/lib/pacman (at boot) and copy everything back to tarball when something is updated (e.g. pacman is run)). this would keep it simple but minimize the accumulation of a defragmentation to a minimum (on the hdd, the whole db will be a single file that is only used as a mirror to the ramdrive)

including such a workaround in pacman i don't mind but other people will say that this is not a good solution (one of my colleagues at the uni already laughted at me using such "primitive" methods against defragmentation, but i told him that it worked great and is very effective and i don't mind if it is "primitive" or not as long as it does the job). also there will be others to claim that this is unneccessary as everybody can do a cp -r and then rm and then mv by hand.

Florin · 2005-04-23 10:22:12

WTG= Way To Go

dp · 2005-04-23 10:44:15

Florin wrote:

WTG= Way To Go

thx

T-Dawg · 2005-04-23 11:05:12

DP or anyone, how would I insert an if statement in this little script to check if the lock was already on before executing the rest.

#!/bin/bash
2>/tmp/pacman.lck
cd /var/lib
cp -r pacman/ pacman_temp/ && rm -r pacman
mv pacman_temp/ pacman
rm -f /tmp/pacman.lck
# end of file

dp · 2005-04-23 11:40:13

this is more or less, how the if statement works:

if [ -f "/path/to/file" ]   
then
 do something because file exists (e.g. return 1)
else   
 do something else, because file does not exist
fi

T-Dawg · 2005-04-23 13:18:21

thx DP, I got it

lanrat · 2005-04-23 14:25:46

Interesting. I just realized that my last hardware upgrade was not the reason of speeding up of all operations on pacman database :-) (among other things I've copied all files from old hd to the new one). Good to know. Now this solution lands in my scripts directory. Tnx.

shadowhand · 2005-04-24 21:37:50

Hmm, strange. I just did this with Reiser4 and it made a difference, but it was very slight. I guess even R4 doesn't do small files extremely well.

dp · 2005-04-24 22:35:03

shadowhand wrote:

Hmm, strange. I just did this with Reiser4 and it made a difference, but it was very slight. I guess even R4 doesn't do small files extremely well.

this IS interesting ... i always meant that reiser4 is able to handle this better

of course, perfect non-fragmented data you cannot have over a long time using very small pieces changed/added every short-time-span ... fragmentation accumulates. a perfect fs needs to have a very good algortithm to store data in a more organised way (keeping an index what dir keeps what kind of data that is on what ocasions accessed and modified)

after some thinking about this, i come to conclude that /var/lib/pacman is an ideal test-case for intelligent fs' to handle/benchmark fragmentation over time ;-)

shadowhand · 2005-04-25 06:50:52

Although, I do have to interject that I transferred my current install to Reiser4 from ReiserFS, so it's possible that's where the fragmentation came from? Maybe?

Also, I've never had 8 minutes searches, they've always been within seconds. I'm only noticing a 1-2 second speed up.

IceRAM · 2005-04-25 12:49:17

An interesting link on Linux filesystems fragmentation:
http://www.informatik.uni-frankfurt.de/ … /reiserfs/

I've googled around, but I can't find any tool to check fragmentation for ReiserFS partition. Does anyone know any?

dp · 2005-04-25 17:26:55

IceRAM wrote:

An interesting link on Linux filesystems fragmentation:
http://www.informatik.uni-frankfurt.de/ … /reiserfs/
I've googled around, but I can't find any tool to check fragmentation for ReiserFS partition. Does anyone know any?

the only thing i can find about this is on the site you posted :
http://www.informatik.uni-frankfurt.de/ … etest.html

cactus · 2005-04-26 03:59:01

I agree. good workaround, but I would hate to see something like that make its way into the code for pacman. It is highly dependent upon the underlying filesystem too.

Anyway, good find dp.

xirus · 2005-07-11 17:54:02

wow nice dp!!

I finally understand why some others didn't have this incredibly slow pacman - and after 2.5 years, it WAS slow
Now almost instantaniously, thx!
Penguin too, for the script

phrakture · 2005-07-11 18:53:54

xirus wrote:

wow nice dp!!
I finally understand why some others didn't have this incredibly slow pacman - and after 2.5 years, it WAS slow
Now almost instantaniously, thx!
Penguin too, for the script

the script is now included with pacman as "pacman-optimize"

Arch Linux

#1 2005-04-21 18:17:22

defragmentation of /var/lib/pacman/ (reiserfs)

#2 2005-04-21 18:39:46

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#3 2005-04-22 00:27:57

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#4 2005-04-22 22:19:07

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#5 2005-04-23 00:23:25

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#6 2005-04-23 01:02:44

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#7 2005-04-23 01:10:17

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#8 2005-04-23 01:25:43

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#9 2005-04-23 08:38:45

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#10 2005-04-23 08:41:01

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#11 2005-04-23 08:52:22

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#12 2005-04-23 10:22:12

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#13 2005-04-23 10:44:15

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#14 2005-04-23 11:05:12

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#15 2005-04-23 11:40:13

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#16 2005-04-23 13:18:21

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#17 2005-04-23 14:25:46

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#18 2005-04-24 21:37:50

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#19 2005-04-24 22:35:03

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#20 2005-04-25 06:50:52

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#21 2005-04-25 12:49:17

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#22 2005-04-25 17:26:55

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#23 2005-04-26 03:59:01

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#24 2005-07-11 17:54:02

Re: defragmentation of /var/lib/pacman/ (reiserfs)

#25 2005-07-11 18:53:54

Re: defragmentation of /var/lib/pacman/ (reiserfs)

Board footer