You are not logged in.

#1 2006-06-24 17:26:25

_Gandalf_
Member
Registered: 2006-01-12
Posts: 735

[SCRIPT] pacman-drive

Hi,

I've made a little script to create/delete a loopback file for pacman database, Using a loopback file you can experience up to 60% additional speed since the partition (of the loopback file) contain only database files, even if the files get fragmented a lot, you can always delete and then re-create it back (like pacman-optimize does)..

The script takes one argument (required) which tells the script if you want to create the drive or remove it... it has been tested many times though no harm if anyone tested it too.

Download the script from here

Before any operation, the script will create a backup in /root/ of the database (and does not run if it already exists to make sure you don't accedently remove a good backup... )

Any feedback is appreciated...

[PLEASE READ]
P.S: The script makes a backup of your database, but Just in case, make a backup yourself, you only need the local subfolder, others can be recovered with pacman -Sy..

cd /var/lib/pacman
tar cjf pacman-local.tar.bz2 local
mv pacman-local.tar.bz2 /root

[/PLEASE READ]

Wael Nasreddine

Offline

#2 2006-06-25 02:20:47

twiistedkaos
Member
Registered: 2006-05-20
Posts: 666

Re: [SCRIPT] pacman-drive

Hmm I don't quite understand the purpose of this script. It creates a partition for pacman downlaoded files? What's the point in that and how exactly does it really make it any faster than just using pacman normally?

Offline

#3 2006-06-25 02:23:36

russianpirate
Member
Registered: 2006-05-17
Posts: 161

Re: [SCRIPT] pacman-drive

twiistedkaos wrote:

Hmm I don't quite understand the purpose of this script. It creates a partition for pacman downlaoded files? What's the point in that and how exactly does it really make it any faster than just using pacman normally?

it makes an fs image (iso, etc.) of the pacman database (the thing that has all the info about the packages in arch) and since its gonna be on its own virtual partition its not gonna be fragmented since the space is reserved for it already


Darkstar:
Athlon 64 3500+ (OC ~2.49ghz)
Leadtek GeForce 6800 128mb (unlocked/OC)
Rosewill Value DDR400 1GB
WDC SE 80GB
NEC 3550 DVDRW DL 16X

Offline

#4 2006-06-25 02:49:25

_Gandalf_
Member
Registered: 2006-01-12
Posts: 735

Re: [SCRIPT] pacman-drive

Exactly, if you haven't noticed, pacman by the time get slower and slower, this isn't because pacman sucks or anything, but because the database get fragmented a lot..

Go ahead and browse /var/lib/pacman, it contains a LOT of small files, and by time one file will be at the beginning of your HDD and the next at the end of it... By creating a loopback file (a partition, an FS call it whatever u want), the files stays within this 200MB file close to each other, so your PC will access the files much faster...

BEFORE
time pacman -Ss pacman
.....
real 0m48.018s
user 0m0.420s
sys 0m0.780s

AFTER
time pacman -Ss pacman
.....
real 0m1.313s
user 0m0.332s
sys 0m0.348s

:grin:

Offline

#5 2006-06-25 03:01:54

twiistedkaos
Member
Registered: 2006-05-20
Posts: 666

Re: [SCRIPT] pacman-drive

The files in  /var/lib/pacman can simply be removed can they not? Which will just force pacman to download the database again, which will also speed up pacmans process won't it?

Offline

#6 2006-06-25 03:23:50

_Gandalf_
Member
Registered: 2006-01-12
Posts: 735

Re: [SCRIPT] pacman-drive

Oh NO NO and NO!!!
If you delete /var/lib/pacman/local, pacman will lose all evidence of any installed package on your system, So if you delete /var/lib/pacman/local you have to either process /var/log/pacman.log carefully and install back every installed package or simply re-install your OS.

So that's why in my script, not only I double check the files, but also I back up the whole folder before touching it...

P.S: /var/lib/pacman contain also current/extra etc... which can be deleted safely but i suggest not to play with them unless u know what your're doing...

Offline

#7 2006-06-25 04:20:15

twiistedkaos
Member
Registered: 2006-05-20
Posts: 666

Re: [SCRIPT] pacman-drive

Hmm, interesting. I suppose I'll try out your script sometime tomarrow... or not. I have something to do tomarrow xD. Erm, I'll give it a try when I get back from my friends graduation party.

Offline

#8 2006-06-25 04:54:59

russianpirate
Member
Registered: 2006-05-17
Posts: 161

Re: [SCRIPT] pacman-drive

need check_root.. dont have that program.. is it in arch (i couldnt find it..)


Darkstar:
Athlon 64 3500+ (OC ~2.49ghz)
Leadtek GeForce 6800 128mb (unlocked/OC)
Rosewill Value DDR400 1GB
WDC SE 80GB
NEC 3550 DVDRW DL 16X

Offline

#9 2006-06-25 06:03:59

_Gandalf_
Member
Registered: 2006-01-12
Posts: 735

Re: [SCRIPT] pacman-drive

Oops forgot that function, This script is a part of a big project which i adapted to Arch actually, anyway fixed now, Sorry :oops:

Offline

#10 2006-06-25 14:50:04

sh__
Member
Registered: 2005-07-19
Posts: 272

Re: [SCRIPT] pacman-drive

So... what's the difference between this script and pacman-cage?

It might be worthwhile to change the bytes-per-inode value of the filesystem (see the last post on that thread). Since the files are so small, you are more likely to run out of inodes before running out of disk space.

Offline

#11 2006-06-25 16:13:22

_Gandalf_
Member
Registered: 2006-01-12
Posts: 735

Re: [SCRIPT] pacman-drive

Hmm I haven't seen that script before thx for pointing this out, anyway about inodes limit maybe you're right, I'll do some tests and adjust the script..
thx

Offline

#12 2006-06-25 23:10:49

_Gandalf_
Member
Registered: 2006-01-12
Posts: 735

Re: [SCRIPT] pacman-drive

sh__ wrote:

So... what's the difference between this script and pacman-cage?

After checking pacman-cage, this script seems more/less secure... read the code and see the difference...

sh__ wrote:

It might be worthwhile to change the bytes-per-inode value of the filesystem (see the last post on that thread). Since the files are so small, you are more likely to run out of inodes before running out of disk space.

Thanks for that, I Fixed it...

Offline

#13 2006-06-26 03:54:29

linfocito
Member
From: Gurupi - TO, Brasil
Registered: 2003-05-18
Posts: 82

Re: [SCRIPT] pacman-drive

Many people are stucked with reiserfs because pacman takes longer to be unbearable slow with that fs, compared to XFS and others. I think your script pretty much solves the problem.

Now I can have Archlinux+XFS without pacman-optimize every few days.

Thanks a lot for that!!!!


"...archoholism is a hard disease to cure..."
Archlinux Brasil

Offline

#14 2006-06-26 15:35:48

test1000
Member
Registered: 2005-04-03
Posts: 834

Re: [SCRIPT] pacman-drive

thaks alot for that aswell. This make pacman nearly instant compared to 50 seconds or more with XFS. This would also "fix" this bug: http://bugs.archlinux.org/task/4821


KISS = "It can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience." - Albert Einstein

Offline

#15 2006-06-27 15:03:09

neotuli
Lazy Developer
From: London, UK
Registered: 2004-07-06
Posts: 1,204
Website

Re: [SCRIPT] pacman-drive

You did those time tests back to back, didn't you?

The files were already cached in memory if you did...


The suggestion box only accepts patches.

Offline

#16 2006-06-27 18:39:25

russianpirate
Member
Registered: 2006-05-17
Posts: 161

Re: [SCRIPT] pacman-drive

This doesnt help me A LOT, but it doesnt slow anything down either so ill keep it.


Darkstar:
Athlon 64 3500+ (OC ~2.49ghz)
Leadtek GeForce 6800 128mb (unlocked/OC)
Rosewill Value DDR400 1GB
WDC SE 80GB
NEC 3550 DVDRW DL 16X

Offline

#17 2006-06-27 18:45:25

_Gandalf_
Member
Registered: 2006-01-12
Posts: 735

Re: [SCRIPT] pacman-drive

Well don't expect to have P17 with it lol, but with it I got rid of the irritating "wait to read database..."

it helps specially if my pc is on for like a week compiling (memory full, swap almost)...

Offline

#18 2006-06-27 19:35:20

twiistedkaos
Member
Registered: 2006-05-20
Posts: 666

Re: [SCRIPT] pacman-drive

Helped me out, took about 10 secs off the normal time. Just an estamated time though. But nice script, I really enjoy it smile Good work Gandalf!

Offline

#19 2006-06-29 13:27:55

test1000
Member
Registered: 2005-04-03
Posts: 834

Re: [SCRIPT] pacman-drive

ok, something fcked up. now my pacman is b0rked. How do I fix this manually Gandalf?

kris|~$ sudo ./pacman-drive delete
==> ERROR:  /root/pacman-database.tar.bz2 file already exists, please remove it first.

ofc i've tried removing it... and a bunch of other stuff aswell.. Now how do I get my old pacman back? Right now it remembers no entries...


KISS = "It can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience." - Albert Einstein

Offline

#20 2006-06-29 13:48:45

_Gandalf_
Member
Registered: 2006-01-12
Posts: 735

Re: [SCRIPT] pacman-drive

You haven't kept a backup before attempting to create/delete???

why ppl never read the post?? roll

Offline

#21 2006-07-01 14:08:32

alexpnx
Member
From: Nicosia, Cyprus
Registered: 2006-06-10
Posts: 47

Re: [SCRIPT] pacman-drive

this script is really awesome! before pacman took 50+sec and now it only takes 5sec to initialize!

Offline

#22 2006-07-10 06:29:58

holst
Member
Registered: 2006-03-25
Posts: 2

Re: [SCRIPT] pacman-drive

test1000 wrote:

ok, something fcked up. now my pacman is b0rked. How do I fix this manually Gandalf?

kris|~$ sudo ./pacman-drive delete
==> ERROR:  /root/pacman-database.tar.bz2 file already exists, please remove it first.

ofc i've tried removing it... and a bunch of other stuff aswell.. Now how do I get my old pacman back? Right now it remembers no entries...

So if you really have f**kup your pacman db _and_ got /root/pacman-local.tar.bz2 left, just unpack it in the /var/lib directory and you are good to go again.

cd /var/lib
mv pacman pacman.old
tar xfj /root/pacman-local.tar.bz2

After you have verified that everything works again you can remove the /var/lib/pacman.old directory.

An lession to be learned from all this.
Never trust a script to do 3 lines of manual labour.

This is how you setup an maximum optimized /var/lib/pacman to begin with.

1. Make a partition on 100MB during install or after if you have space left.
2. Create it with xfs or jfs that are good with many small files.
3. Edit fstab to mount /dev/thatpartition in /var/lib/pacman

Another tip for us with many installations of arch linux (sharing same cpu architecture). Mount /var/cache/pacman on a nfs share. That way you save much bandwidth. Since arch has a lot of package updates it becomes a unnecessary large download if you must download all packages 2-3 times each.

Offline

#23 2006-07-10 11:24:45

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,868

Re: [SCRIPT] pacman-drive

interesting idea. I get a VHCS error :

Error 403!
/files/arch/scripts/pacman-drive/pacman-drive
Forbidden!
when i try to download it, so i can't check it.

Question : could the same results be achieved by making a separate partition (maybe with LVM) for var/lib/pacman ?


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#24 2006-07-10 11:32:09

_Gandalf_
Member
Registered: 2006-01-12
Posts: 735

Re: [SCRIPT] pacman-drive

holst wrote:

An lession to be learned from all this.
Never trust a script to do 3 lines of manual labour.

that's correct also you can use the script but read the 3 bolded lines wink

holst wrote:

This is how you setup an maximum optimized /var/lib/pacman to begin with.

1. Make a partition on 100MB during install or after if you have space left.
2. Create it with xfs or jfs that are good with many small files.
3. Edit fstab to mount /dev/thatpartition in /var/lib/pacman

Another tip for us with many installations of arch linux (sharing same cpu architecture). Mount /var/cache/pacman on a nfs share. That way you save much bandwidth. Since arch has a lot of package updates it becomes a unnecessary large download if you must download all packages 2-3 times each.

You're missing a major thing here, never use jfs/xfs for pacman database, pacman database is an ext2 partition not xfs/jfs read the topic man...

Offline

#25 2006-07-10 11:34:41

_Gandalf_
Member
Registered: 2006-01-12
Posts: 735

Re: [SCRIPT] pacman-drive

Lone_Wolf wrote:

interesting idea. I get a VHCS error :

Error 403!
/files/arch/scripts/pacman-drive/pacman-drive
Forbidden!
when i try to download it, so i can't check it.

Sorry i changed the path of my svn auto checkouts and forgot to update the symlink, Fixed now

Lone_Wolf wrote:

Question : could the same results be achieved by making a separate partition (maybe with LVM) for var/lib/pacman ?

it will give you the same result, a partion as a file or a physical partition will give the same (maybe very close) speed IMHO...

Offline

Board footer

Powered by FluxBB