Pacman Optimization

ody · 2006-03-31 20:45:08

DO NOT USE THIS SCRIPT, please see thread: http://bbs.archlinux.org/viewtopic.php?t=20385 instead.

Hey

I just slapped together an idea I had today to keep pacman running quickly without the need to pacman-optimize; your mileage may vary. The aim is to keep the pacman db in a contiguous as possible place on your hd to make reading it quicker.

So anyway the script below will do what I have done for you but make sure you increase the count= on the dd line to come into line with what you think you will need for your pacman db, run "du -sh /var/lib/pacman" to get an idea of how much space you will need (I would say set it to atleast 150) when creating the loopback file with dd. Now the loopback file itself may be created fragmented so the best time to do this would be during a fresh install, but I have noticed a massive increase in speed and my root is years old.

**warning, I take no responsibility for you doing this**

#!/bin/sh
# pacman-cage (c) Andrew Rose (rose.andrew@gmail.com)
dd if=/dev/zero of=/var/lib/pacman.db bs=1M count=150
yes | mkfs.ext3 /var/lib/pacman.db
mkdir /mnt/tmp-pacman
mount -o loop /var/lib/pacman.db /mnt/tmp-pacman
cp -R /var/lib/pacman/. /mnt/tmp-pacman
umount /mnt/tmp-pacman
rmdir /mnt/tmp-pacman 
mv /var/lib/pacman /var/lib/pacman-bk
mkdir /var/lib/pacman
echo "/var/lib/pacman.db /var/lib/pacman ext3 loop,defaults 0 0" >> /etc/fstab
mount -a
pacman -Sy

Please let me know of any improvments you can think of, or problems I have not thought of.

cheers,
ody

Cerebral · 2006-03-31 21:15:25

iphitus was playing around with this idea earlier.. he said he was planning on posting what he did (maybe wikifying it) sometime... I suppose he either hasn't gotten the time to do that yet, or I just missed it.

In any case, I think he said it helped out quite a bit - I might give this a try sometime soon.

ody · 2006-03-31 21:30:58

I did do a quick search for pacman dd / loop back before I went ahead with this so I didn't step on any toes, sorry if this is the case.

lilsirecho · 2006-04-01 03:02:39

Not fully familiar with pacman.db.

I find current, extra, community and local in /var/lib/pacman.

I think your program should reflect ...local...if it is the installed pacman packages db that you are looping to hold them in an exclusive accessible format.

Perhaps I do not understand the use of .db in your script?

Would like to try the script if I can understand it more fully. My local db is 1440 items which may indeed be spread about wildly.

I made a new install about 3 weeks or so back.

Would be glad to try it if I do understand it correctly.

Cerebral · 2006-04-01 03:36:59

lilsirecho: What his script does is make a single file, pacman.db, format it as an ext3 partition, and copy /var/lib/pacman to there. Then it removes /var/lib/pacman from the filesystem, and mounts the pacman.db file as an ext3 system in its normal location. The files don't become fragmented, because it's all one little filesystem.

lilsirecho · 2006-04-01 04:52:44

I surmised that the database was enclosed in a loopback setup to reduce the search time should the system be fragmented.

I am not certain if I understand the use of the .... .db ext....since it is not a normal identifier in addressing directory or file.

Also, in the event that pacman is upgraded, or even the loopback device, is there a possible breakdown in this method? As well, if one has to re-install, this procedure would then have to be reinstalled to renew it as I understand the idea.

I would expect this procedure would reduce the access time to internal pacman data when upgrading pacman pkgs.

I have to assume the .. db ... extension applies to all elements in the directory, namely /var/lib/pacman/current/extra/community and/ocal if there is no reference to a particular repository.

If a new repository is added at some future time, will that be included as well without requiring intervention?

Don't need to mess up a good pacman system ya kno!
Thanks for your reply.....

ody · 2006-04-01 12:37:45

The fact that the loopback file has a .db extention was purly so it didn't clash with its mount point when mounted: /var/lib/pacman. I could have called it "pacman" but would have had to put it someplace else than /var/lib. the "pacman.db" effects none of the workings of pacman, is is nothing more than a file that holds the filesystem that is mounted to /var/lib/pacman.. Does that make sense?

I'll explain the setup step by step incase:

Create a file to store our pacman database via loop back.

dd if=/dev/zero of=/var/lib/pacman.db bs=1M count=150

Create a ext3 filesystem on that file.

yes | mkfs.ext3 /var/lib/pacman.db

Create a temorary mount point for the file.

mkdir /mnt/tmp-pacman

Mount it using -o loop to /mnt/tmp-pacman

mount -o loop /var/lib/pacman.db /mnt/tmp-pacman

Copy the pacman database to this new looped backed filesystem.

cp -R /var/lib/pacman/. /mnt/tmp-pacman

Umount the new filesystem.

umount /mnt/tmp-pacman

Remove the temporary mount point.

rmdir /mnt/tmp-pacman

Move the original database to pacman-bk incase we need it in the future.

mv /var/lib/pacman /var/lib/pacman-bk

Make a new mount point for the loopback file.

mkdir /var/lib/pacman

Create the entry in /etc/fstab so the new pacman database is mounted at boot time.

echo "/var/lib/pacman.db /var/lib/pacman ext3 loop,defaults 0 0" >> /etc/fstab

Mount everything.

mount -a

Test it.

pacman -Sy

lilsirecho · 2006-04-01 14:41:58

ody;

Many thanks for your descriptive comments which are helpful in fully understanding the operations performed.

Your patient reply is appreciated It will be a help to other users as well.

It seems it should provide a speedier pacman function.

I am about to program it into my system.

Thanks again.

Ray

lilsirecho · 2006-04-01 15:20:57

ody;
Performed the procedure you outlined and have the following to report;
==================================================

Result obtained at ...mount-a..........................

This filesystem will be automatically checked every 31 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
[root@n6re ~]# mkdir /mnt/tmp-pacman
[root@n6re ~]# mount -o loop /var/lib/pacman.db /mnt/tmp-pacman
[root@n6re ~]# cp -R /var/lib/pacman/. /mnt/tmp-pacman
[root@n6re ~]# umount /mnt/tmp-pacman
[root@n6re ~]# rmdir /mnt/tmp-pacman
[root@n6re ~]# mv /var/lib/pacman /var/lib/pacman-bk
[root@n6re ~]# mkdir /var/lib/pacman
[root@n6re ~]# echo "/var/lib/pacman.db /var/lib/pacman ext3 loop,defaults 0 0" >> /etc/fstab
[root@n6re ~]# mount -a
mount: usbfs already mounted or /proc/bus/usb busy
mount: according to mtab, none is already mounted on /proc/bus/usb
mount: sysfs already mounted or /sys busy
mount: according to mtab, none is already mounted on /sys
mount: wrong fs type, bad option, bad superblock on /dev/sda1,
missing codepage or other error
In some cases useful info is found in syslog - try
dmesg | tail or so

mount: special device /dev/sdb1 does not exist
mount: special device /dev/scd1 does not exist

================================================

The result obtained at the .....pacman -Sy.....step;
================================================

[root@n6re ~]# pacman -Sy
:: Synchronizing package databases...
current [################] 100% 66K 40.5K/s 00:00:01
extra [################] 100% 237K 46.7K/s 00:00:05
community [################] 100% 54K 35.4K/s 00:00:01
[root@n6re ~]# pacman -Sy
:: Synchronizing package databases...
:: current is up to date
:: extra is up to date
:: community is up to date
[root@n6re ~]#

===============================================

The second entry of ...pacman -Sy .... was very fast, probably because the system was pre-exercised by the first entry and all elements aligned in the ext3 file.

/etc/fstab has the listing for the /var/lib/pacman.db ext3 loop file sys.

I assume these results are correct.

ody · 2006-04-01 20:05:42

mount: special device /dev/scd1 does not exist

You have a problem in your /etc/fstab that is trying to use scd1 which does not exist. Don't wory it doesnt at first glance seem to be a big problem and is nothing todo with the script you ran I posted(that would have completed with no problems ala why pacman is faster for you now) you will need to investigate the scd1 problem yourself. Feel free to send me a msg with a dump of your fstab and I will tell you the line that is at fault.

lilsirecho · 2006-04-01 20:59:13

ody;

Aware of the scd1 statement since it isn't connected in the system.

I use a USB to IDE interface which is presently connected to an IDE hdd. When I enable the CD on that interface, SCD1 activates. I do this by applying the +5v to the appropriate device which acts as a selector of the device to be utilized. Hence, fstab includes items not connected. All devices ( up to 5) are selected by a 5 position switch and all elements are external to the computer case. All are connected to the IDE cable at the same time as primary devices. That should define reason for the scd1 error message.

I have no problem with the pacman adaptation and do appreciate your script and helpful description. It seems to work just fine so far!

The /var/lib/pacman files indicate the present status and hold the pacman-bk as well.

Thanks again for the aid and comfort!!!

MAC!EK · 2006-04-01 22:50:36

I check this and no performance... or mayby a little but i can't see it.
I checked how much fragments does db have:

# filefrag /var/lib/pacman.db 
/var/lib/pacman.db: 109 extents found, perfection would be 2 extents

But i don't know how to defragment this file...

kr · 2006-04-01 22:55:42

It works. Faster. Much faster.

MAC!EK · 2006-04-01 22:56:42

kr wrote:

It works. Faster. Much faster.

How much fragments does your file have?

syamajala · 2006-04-01 23:49:59

what about using FUSE for pacman?

iphitus · 2006-04-02 02:02:59

syamajala wrote:

what about using FUSE for pacman?

thats about as generic as it gets, what do you exactly mean by that? fuse can do a lotta things.

James

syamajala · 2006-04-02 02:21:20

well maybe making a high performance file system just for storing the pacman db... i don't really know ;-p

kr · 2006-04-02 08:29:19

MAC!EK wrote:

kr wrote:
It works. Faster. Much faster.
How much fragments does your file have?

/var/lib/pacman.db: 128 extents found

codemac · 2006-04-02 09:31:46

The only thing this does is eliminate the need for pacman-optimize. If you run pacman-optimize, then run a pacman command, then compare that with the time on a seperate filesystem like this, you'll find very little difference. The key here is that you never have to run pacman-optimize if you have this setup. Very cool.

[edit]

Just did this on my system, 63 extents on the file, but noticeably faster for me.

MAC!EK · 2006-04-02 10:34:25

Don't run pacman-optimize after using this Tip!

# pacman-optimize 
==> md5sum'ing the old database...
==> copying /var/lib/pacman...
==> md5sum'ing the new database...
mv: nie można przenieść `/var/lib/pacman' do `/var/lib/pacman.bak': Urządzenie lub zasoby zajęte
pacman-optimize: error renaming /var/lib/pacman

I think a better performance can be seen when searching for smth the DB not updating.

EDIT:
To reduce DB fragmentation do this:

# umount /var/lib/pacman
# cp /var/lib/pacman.db /
# filefrag /var/lib/pacman.db /pacman.db

If the new copied file has less "extents found" do this:

# rm /var/lib/pacman.db
# mv /pacman.db /var/lib/pacman.db
# mount /var/lib/pacman

You will get a little more performance.

If it has more fragments do this:

# rm /pacman.db
# mount /var/lib/pacman

You won't get a performance increase.

syamajala · 2006-04-02 12:49:45

does enabling dir_index do anything?

jakob · 2006-04-02 13:03:55

I would think so, but I have another thing to say: Just executed the script and wondered about the different filesizes of the dirs in /mnt/tmp-pacman and /var/lib/pacman (with the old database, not the mounted yet):

The original /var/lib/pacman:

jakob@pacman $ duh1
7,5M ./local
616K ./shadowhand
2,2M ./testing
4,5M ./community
6,1M ./current
20M ./extra
41M .

After the "cp -R /var/lib/pacman/. /mnt/tmp-pacman"
mnt/tmp-pacman looks like follows:

jakob@tmp-pacman $ duh1
3,7M ./local
156K ./shadowhand
5,1M ./extra
du: „./lost+found": Keine Berechtigung
554K ./testing
1,2M ./community
1,6M ./current
12M .

from 41MiB to 12MiB... strange?

ody · 2006-04-02 14:32:12

It is likely you are using reiserfs as your root filesystem. An example below is of me creating a small 10byte file, now a filesystem will always allocate in set chunks, in my case for my root that is using reiserfs 4k, and for ext3 it uses 1k. Because the pacman db uses lots of files with little bits of information in you will save loads of space using default ext3 over default reiserfs.

[root@developer ~]# cd /var/lib/pacman
[root@developer pacman]# echo "1234567890" > test
[root@developer pacman]# du -sh test
1.0K    test
[root@developer pacman]# cd /
[root@developer /]# echo "1234567890" > test
[root@developer /]# du -sh test
4.0K    test
[root@developer /]#

jakob · 2006-04-02 15:14:58

Hm, unfortunately, I'm having a ext3 on my root

I already cp'd /var/lib/pacman to $HOME and there it has 41MiB, too.

But I know found it:

tune2fs -l for pacman.db

Block size: 1024
Fragment size: 1024
Blocks per group: 8192
Fragments per group: 8192
Inodes per group: 1992
Inode blocks per group: 249

my root:

Block size: 4096
Fragment size: 4096
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 16224
Inode blocks per group: 507

I created the ext3-fs on my root with the pacman setup, is the block size of 4k normal? You said 1k, which looks more sensible, is the default.. strange.

Thank you!

ody · 2006-04-02 15:50:30

Actualy I think most of the settings like block size, blocks per group etc etc are set dependant apon the size of the parititon you formating..

Arch Linux

#1 2006-03-31 20:45:08

Pacman Optimization

#2 2006-03-31 21:15:25

Re: Pacman Optimization

#3 2006-03-31 21:30:58

Re: Pacman Optimization

#4 2006-04-01 03:02:39

Re: Pacman Optimization

#5 2006-04-01 03:36:59

Re: Pacman Optimization

#6 2006-04-01 04:52:44

Re: Pacman Optimization

#7 2006-04-01 12:37:45

Re: Pacman Optimization

#8 2006-04-01 14:41:58

Re: Pacman Optimization

#9 2006-04-01 15:20:57

Re: Pacman Optimization

#10 2006-04-01 20:05:42

Re: Pacman Optimization

#11 2006-04-01 20:59:13

Re: Pacman Optimization

#12 2006-04-01 22:50:36

Re: Pacman Optimization

#13 2006-04-01 22:55:42

Re: Pacman Optimization

#14 2006-04-01 22:56:42

Re: Pacman Optimization

#15 2006-04-01 23:49:59

Re: Pacman Optimization

#16 2006-04-02 02:02:59

Re: Pacman Optimization

#17 2006-04-02 02:21:20

Re: Pacman Optimization

#18 2006-04-02 08:29:19

Re: Pacman Optimization

#19 2006-04-02 09:31:46

Re: Pacman Optimization

#20 2006-04-02 10:34:25

Re: Pacman Optimization

#21 2006-04-02 12:49:45

Re: Pacman Optimization

#22 2006-04-02 13:03:55

Re: Pacman Optimization

#23 2006-04-02 14:32:12

Re: Pacman Optimization

#24 2006-04-02 15:14:58

Re: Pacman Optimization

#25 2006-04-02 15:50:30

Re: Pacman Optimization

Board footer