You are not logged in.

#1 2005-04-26 03:53:19

T-Dawg
Forum Fellow
From: Charlotte, NC
Registered: 2005-01-29
Posts: 2,736

pacman-defrag [obsolete - use pacman-optimize]

After dp's discovery of a simple way to speed up pacman which can be found here I've come up with this little program. It should be pretty safe because it checks for a running instance of pacman before executing and locks pacman (2>/tmp/pacman.lck) during use so it won't corrupt files. I threw in the other options because I thought they might be usefull. A log will be created as needed in the /var/log directory as pacman-defrag.log.

Move the file into /usr/bin/ , chown it root, and chmod 755.

file and sums can also be found here

#!/bin/bash

## CHANGE THESE VALUES ONLY IF THE PROGRAM COMPLAINS TO  ##
###### SHOULD BE UNIQUE AN MUST NOT EXIST ALREADY! ########
newtemp="/var/lib/new_temp/"
oldtemp="/var/lib/old_temp/"
############################################################

ver='1.5'
RED="33[1;31m"
NORMAL="33[0;39m"
log="/var/log/pacman-defrag.log"
d=`date +"[%b %d %H:%M %Y]"`
smes="Successful database defrag."
fmes="ERROR: Unsuccessful database defrag, pacman lock detected."
wuser="You must be root user to run this program."
screenlog="New log file created as /var/log/pacman-defrag.log"
notargetmes="ERROR: Unsuccessful database defrag, database directory did not exist."
newtmpexistsmes="ERROR: Unsuccessful database defrag, temp target $newtemp already exists."
oldtmpexistsmes="ERROR: Unsuccessful database defrag, temp target $oldtemp already exists."
badmd5mes="ERROR: Unsuccessful database defrag, md5sums did not match. Old database restored."

usage() {
echo "  pacman-defrag $ver"
echo
echo "  pacman-defrag is a utility that safely defrags database"
echo "  files in /var/lib/pacman for Arch Linux's package manager,"
echo "  'pacman'."
echo
echo "  A log file will automatically created and can be found in"
echo "  /var/log/ as 'pacman-defrag.log'."
echo
echo "  usage: $0 [options]"
echo
echo "  {-h  --help}"
echo
echo "  {-v  --version}"
echo
echo "  {-d  --defrag}   Defrags database safely by checking for"
echo "                   locks created by pacman while locking"
echo "                   itself from it so both can't run together."
echo "                   This should be your default option."
echo
echo "  {-f  --force}    Force the defrag ignoring pacman lock. If"
echo "                   no lock is found, one will be created to"
echo "                   protect pacman. BE CAREFULL WITH THIS!"
echo
echo "  {-r  --rforce}   Same as force except the lock will be"
echo "                   removed after use."
echo
exit 0
}

version() {
echo "version $ver"
exit 0
}

idcheck(){
[ `id -u` -ne 0 ] && {
echo "$wuser"
exit 1
}
}

check(){
if [ ! -d /var/lib/pacman ]; then
        echo "Error: database directory does not exist."
        echo "aborting."
        smes=$notargetmes
        logger
        exit 1
fi
if [ -d $newtemp ]; then
        echo "Error: $newtemp already exists. Please reset newtemp= near the top of"
        echo "this script to a unique directory that does not already exist."
        echo "aborting."
        smes=$newtmpexistsmes
        logger
        exit 1
fi
if [ -d $oldtemp ]; then
        echo "Error: $oldtemp already exists. Please reset oldtemp= near the top of"
        echo "this script to a unique directory that does not already exist."
        echo "aborting."
        smes=$oldtmpexistsmes
        logger
        exit 1
fi
}

movecopy(){
echo
cd /var/lib
tar -cf old.tar pacman/
md5sum old.tar | sort > old.sums
cp -rp pacman/ $newtemp
mv -f pacman/ $oldtemp
mv -f $newtemp pacman/
echo -e "checking integrity... c"
tar -cf new.tar pacman/
md5sum new.tar | sort > new.sums
olddb=`cat old.sums | cut -d' ' -f 1`
newdb=`cat new.sums | cut -d' ' -f 1`
echo $olddb > olddb.sums
echo $newdb > newdb.sums
diff olddb.sums newdb.sums
if [ $? -ne 0 ]; then
        echo "FAILED."
        echo -e "restoring old databases... c"
        rm -r pacman/
        mv -f $oldtemp pacman/
        smes=$badmd5mes
else
        echo "OK."
        ##### UNCOMMENT TO SEE SUMS ####
        #echo $olddb
        #echo $newdb
        ################################
        echo -e "completing defrag... c"
        rm -r $oldtemp
fi
rm -f old.tar
rm -f new.tar
rm -f old.sums
rm -f new.sums
rm -f olddb.sums
rm -f newdb.sums
}
logger() {
if [ -f $log ]; then
        echo -e "Logging event... c"
        echo -n "$d  " >> $log
        echo "$smes" >> $log
        echo "done."
else
        echo -n "$d  " >> $log
        echo "$smes" >> $log
echo "$screenlog"
fi
exit 0
}

defrag() {
idcheck
check
if [ -f "/tmp/pacman.lck" ]; then
        echo -e "${RED}Pacman lock detected!${NORMAL}"
        echo "If you are sure pacman is not being used, run this program again using -r."
        echo "aborting."
        smes=$fmes
else
        2>/tmp/pacman.lck
        movecopy
        rm -f /tmp/pacman.lck
        echo "done."
fi
logger
}

force() {
idcheck
check
if [ -f "/tmp/pacman.lck" ]; then
        movecopy
else
        2>/tmp/pacman.lck
        movecopy
        rm -f /tmp/pacman.lck
fi
logger
}

rforce() {
idcheck
check
if [ -f "/tmp/pacman.lck" ]; then
        movecopy
        rm -f /tmp/pacman.lck
else
        2>/tmp/pacman.lck
        movecopy
        rm -f /tmp/pacman.lck
fi
echo "done."
logger
}
 case $1 in
 -h | --help) usage ;;
 -d | --defrag) defrag ;;
 -v | --version) version ;;
 -f | --force) force ;;
 -r | --rforce) rforce ;;
 *) usage
  exit 0
  ;;
esac
# end of file

Offline

#2 2005-04-26 07:41:20

arooaroo
Member
From: London, UK
Registered: 2005-01-13
Posts: 1,268
Website

Re: pacman-defrag [obsolete - use pacman-optimize]

I love this forums' ability to knock up a bash script at every oppurtunity. smile Nice one Penguin.

I better start swotting up on bash scripting - I'm beginning to feel inadequate...

Offline

#3 2005-04-26 08:19:58

cmp
Member
Registered: 2005-01-03
Posts: 350

Re: pacman-defrag [obsolete - use pacman-optimize]

some simple hints:
you may want to create the directory pacman_tmp before copying, or at least check for its presence.
but I would recommend using "mktemp -p /var/lib/ -d" anyways.

Offline

#4 2005-05-12 03:30:04

markw
Member
From: US
Registered: 2005-03-17
Posts: 14
Website

Re: pacman-defrag [obsolete - use pacman-optimize]

Very nice script! I made a cron job that executes this script every week. (I am insane about automation; I just want to leave my computer and have it maintain itself).


"power corrupts, and absolute power corrupts absolutely" -Lord Acton

Offline

#5 2005-05-12 10:21:35

Moo-Crumpus
Member
From: Hessen / Germany
Registered: 2003-12-01
Posts: 1,487

Re: pacman-defrag [obsolete - use pacman-optimize]

I would prefer to copy and remove each file step by step, and not copy the whole tree at once. I am no well coder, but it should be something like "for each file in /var/lib/pacman do copy to temp - check if it is the same - delete source - copy to source - check if it is the same - delete copy".


Frumpus addict
[mu'.krum.pus], [frum.pus]

Offline

#6 2005-05-12 14:50:57

T-Dawg
Forum Fellow
From: Charlotte, NC
Registered: 2005-01-29
Posts: 2,736

Re: pacman-defrag [obsolete - use pacman-optimize]

UPDATE:
* Took cmp's advice about checking the temporary folder exists before copying back. Failure will be logged and echoed to the user.

*Added ability for user to define their own unique temp folder (near the top).
         NOTE: This folder should be unique and must not     exist already!

*Shortened  code

pink chick wrote:

I would prefer to copy and remove each file step by step, and not copy the whole tree at once. I am no well coder, but it should be something like "for each file in /var/lib/pacman do copy to temp - check if it is the same - delete source - copy to source - check if it is the same - delete copy".

I will look into this.

Offline

#7 2005-05-13 16:50:17

phrakture
Arch Overlord
From: behind you
Registered: 2003-10-29
Posts: 7,879
Website

Re: pacman-defrag [obsolete - use pacman-optimize]

Pink Chick wrote:

I would prefer to copy and remove each file step by step, and not copy the whole tree at once. I am no well coder, but it should be something like "for each file in /var/lib/pacman do copy to temp - check if it is the same - delete source - copy to source - check if it is the same - delete copy".

I don't really understand what you're saying.  But copying files individually won't do the same thing... the point is to copy the directory as a whole, in order to line up the inodes (well, cluster them or something, "line up" doesn't make sense)...

If you're saying you want something to validate that the "cp -r" succeeded, then that's a tad easier.... instead of "rm -f pacman" how about "mv pacman pacman-bak" then a nice little echo that says "old db has been moved here, go ahead and validate it..." and you can recursively apply md5sums from file to file...

Offline

#8 2005-05-15 03:18:14

T-Dawg
Forum Fellow
From: Charlotte, NC
Registered: 2005-01-29
Posts: 2,736

Re: pacman-defrag [obsolete - use pacman-optimize]

UPDATE:
* Now checks integrity using md5sums on the entire directory as a whole. It will automatically restore if bad sums are found.

Offline

#9 2005-06-03 18:13:40

FoPref
Member
From: Erlangen / Germany
Registered: 2004-03-24
Posts: 96
Website

Re: pacman-defrag [obsolete - use pacman-optimize]

[Jun 03 20:12 2005]  ERROR: Unsuccessful database defrag, md5sums did not match. Old database restored.


As you see, your checksum test worked. But I wonder what got wrong?

Offline

#10 2005-06-03 23:33:28

T-Dawg
Forum Fellow
From: Charlotte, NC
Registered: 2005-01-29
Posts: 2,736

Re: pacman-defrag [obsolete - use pacman-optimize]

Does it happen every time?

I copied and pasted the code and compared it with my original using diff and found no differences. You should do the same, it could be a copy and paste error.

If it does happen frequently I would do the following:
1  reinstall tar
2  reinstall coreutils
3  reinstall bash
4  reinstall your kernel and reboot.

Also, I would not advise running it during heavy system load

Hopefully it was just a freak thing. It's never happen to me before (ran it a millions times testing) so I suspect that's what it was.

Offline

#11 2005-06-04 00:00:48

FoPref
Member
From: Erlangen / Germany
Registered: 2004-03-24
Posts: 96
Website

Re: pacman-defrag [obsolete - use pacman-optimize]

Hi,

it happens every time and the checksums are the same ones every time. So I guess this is not a kind of "random" error.


Regards,
Ford Prefect

Offline

#12 2005-06-04 00:18:07

T-Dawg
Forum Fellow
From: Charlotte, NC
Registered: 2005-01-29
Posts: 2,736

Re: pacman-defrag [obsolete - use pacman-optimize]

did you try the other things I mentioned?

Are you seeing the sums and the filename to the right? If so, that's the problem.

I ran what I copied and pasted and I still can't simulate that error, even after new updates.

If it still gives you problems, you can pm me an email address where I can send you the file digitally signed.

Offline

#13 2005-06-04 21:00:52

Dusty
Schwag Merchant
From: Medicine Hat, Alberta, Canada
Registered: 2004-01-18
Posts: 5,986
Website

Re: pacman-defrag [obsolete - use pacman-optimize]

As a notice, I included this script in the new scripts section of the wiki:
http://wiki2.archlinux.org/index.php/pacman-defrag

Penguin: If you don't like it being world editable, feel free to remove it. You can also improve the text if you like.

Dusty

Offline

#14 2005-06-05 13:55:36

Dreameen
Member
From: Poland
Registered: 2004-09-06
Posts: 252

Re: pacman-defrag [obsolete - use pacman-optimize]

I was just wondering...the pacman-defrag script works well and it actually speeds things up considerably.

Maybe the same idea would work for all the other system directories like /usr, /opt, /lib etc.. making the system run a lot faster.

From what I've heard there are no defrag tools in linux, because they are unnecessary, well...the script that we have here proves that this argument is not entirely true. What is your opinion on this topic? Feel free to move it to a seperate thread if the need arises, I've posted it here in case the answer to my doubts is short&sweet wink

Offline

#15 2005-06-05 21:54:03

T-Dawg
Forum Fellow
From: Charlotte, NC
Registered: 2005-01-29
Posts: 2,736

Re: pacman-defrag [obsolete - use pacman-optimize]

Well, there's alot to explain about what causes files to defrag but to keep it simple, we'll say its because we're dealing with many small 100B files. All filesystems have a hard time managing them, it the way they were engineered. To manage them well would mean undesireable sacrafices elsewhere.
Yes linux filesystems don't get nearly as defraged as NTFS or FAT  on average so, to answer your question, a defrag utility wouldn't be necessary for those particular directories. The filesystem should be able to manage it quite well.
I used to have bookmarks on the matter but I've lost them. A google search would explain it more in depth.

Offline

#16 2005-06-05 22:12:09

i3839
Member
Registered: 2004-02-04
Posts: 1,185

Re: pacman-defrag [obsolete - use pacman-optimize]

It's a not about fragmentation of files or the filesystem, it is that Pacman reads a lot tiny files after eachother, and when they are scattered around the disk (not fragmented in the fs meaning, just spread), the disk needs to seek a lot to read them all, which is very slow compared to when they would be next to eachother on the disk. It is hard to avoid, as the fs can't know beforehand that it should leave enough space so new files can be added close to the old ones. It does try to put files in the same dir close to eachother in general, but if those files are in subdirs it becomes less clear.

Offline

#17 2005-06-06 04:42:09

deficite
Member
From: Augusta, GA
Registered: 2005-06-02
Posts: 693

Re: pacman-defrag [obsolete - use pacman-optimize]

Thank you so much for this script. It only took like 20 seconds to run it and pacman gives me search results near instantaneously instead of making Amarok's playback skip, and other bad things (tm). I haven't tried installing anything with pacman after "pacman-defrag"'ing, but I assume I will be VERY happy with the results.
</suck up>  tongue

Offline

#18 2005-06-06 08:08:08

IceRAM
Member
From: Bucharest, Romania
Registered: 2004-03-04
Posts: 772
Website

Re: pacman-defrag [obsolete - use pacman-optimize]

The script has probably cached into memory all pacman files, that's why pacman is so responsitive. Try searching using pacman after a reboot. Compare times then.

Offline

#19 2005-06-06 11:23:32

T-Dawg
Forum Fellow
From: Charlotte, NC
Registered: 2005-01-29
Posts: 2,736

Re: pacman-defrag [obsolete - use pacman-optimize]

IceRAM wrote:

The script has probably cached into memory all pacman files.....

A possible side effect but that's not how it works. The cp command is whats bringing the files together.
There really shouldn't be too much of a "permanent" notable difference for those with fairly new installations. The sluggishness occurs after some time....dp recommended doing something like this once a year.

Offline

#20 2005-06-06 11:54:58

iphitus
Forum Fellow
From: Melbourne, Australia
Registered: 2004-10-09
Posts: 4,927

Re: pacman-defrag [obsolete - use pacman-optimize]

IceRAM wrote:

The script has probably cached into memory all pacman files, that's why pacman is so responsitive. Try searching using pacman after a reboot. Compare times then.

After reboot it's still improved.

To start with, your pacman db is going to be around a similar spot on the hard drive. but over time as it changes, packages come and go and whatever, it gradually gets spread across the hard drive, with latter things ending up as far as the other end of the hard drive.
this causes the hard drive's head to have to whip back and forth between platters and all around the place.

the idea of this, is that it makes a copy, which is generally all together, or close together, so then the hard drives head doesnt have to go nuts everywhere, and it does make a difference smile

iphitus

Offline

#21 2005-06-06 12:47:54

Moo-Crumpus
Member
From: Hessen / Germany
Registered: 2003-12-01
Posts: 1,487

Re: pacman-defrag [obsolete - use pacman-optimize]

phrakture wrote:

But copying files individually won't do the same thing... the point is to copy the directory as a whole, in order to line up the inodes (well, cluster them or something, "line up" doesn't make sense)...

You are right. It is about copying all files on one step, while I thought it is about copying each file to defrag them. So I thought it was better to just copy   file by file, so you would'nt need the double of pacmans hd usage. My fault, I missunderstood the script's intention.

It could be worth to store all pacman files in a seperate partition, then. Ore, to say the bad word(C), into a real database.

----
Word : Copyright (C) 1983-2003 Microsoft Corporation. All rights reserved.


Frumpus addict
[mu'.krum.pus], [frum.pus]

Offline

#22 2005-06-06 20:08:56

i3839
Member
Registered: 2004-02-04
Posts: 1,185

Re: pacman-defrag [obsolete - use pacman-optimize]

Or to keep it in the tar and use the libtar interface to access them instead of the fs...

Offline

#23 2005-06-06 22:11:17

IceRAM
Member
From: Bucharest, Romania
Registered: 2004-03-04
Posts: 772
Website

Re: pacman-defrag [obsolete - use pacman-optimize]

iphitus wrote:

After reboot it's still improved.

To start with, ...

Well, I did understand how it works, I just wanted to point out that any activity on the pacman's db files can increase pacman's speed, even if you don't copy from one place to another, just because of the kernel cache.

I wanted to make sure this situation is taken out from the discussion and increase in performance is evaluated correctly.

Offline

#24 2005-06-06 23:08:59

cactus
Taco Eater
From: t͈̫̹ͨa͖͕͎̱͈ͨ͆ć̥̖̝o̫̫̼s͈̭̱̞͍̃!̰
Registered: 2004-05-25
Posts: 4,622
Website

Re: pacman-defrag [obsolete - use pacman-optimize]

I cant believe we are still talking about defragging a filesystem in this day and age.. *sigh*
Does this only happen with reiserfs? I was led to believe that ext3, jfs, and others were not in need of defrags, because they somehow placed consecutively accessed blocks closer together as a general rule..


"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍

Offline

#25 2005-06-06 23:15:11

deficite
Member
From: Augusta, GA
Registered: 2005-06-02
Posts: 693

Re: pacman-defrag [obsolete - use pacman-optimize]

I'm using ext3 right now and it's helped me. I have rebooted and pacman is still being the little installer slave it was before I installed KDE.

Offline

Board footer

Powered by FluxBB