You are not logged in.

#26 2005-06-07 07:34:21

alexmat
Member
Registered: 2004-12-31
Posts: 100

Re: pacman-defrag [obsolete - use pacman-optimize]

Running 40gig ext3 filesystem, had ~200 packages installed, ran defrag, pacman is much quicker for updates and searching... even after multiple reboots. Nice job smile

Edit: forgot to mention, my install before the defrag was 6 months old, also I noticed a significant decrease in hard drive thrashing(??) when running pacman.

Offline

#27 2005-06-07 16:37:15

Moo-Crumpus
Member
From: Hessen / Germany
Registered: 2003-12-01
Posts: 1,487

Re: pacman-defrag [obsolete - use pacman-optimize]

cactus wrote:

I cant believe we are still talking about defragging a filesystem in this day and age.. *sigh*
Does this only happen with reiserfs? I was led to believe that ext3, jfs, and others were not in need of defrags, because they somehow placed consecutively accessed blocks closer together as a general rule..

No, this is not the problem. The problem is to keep several hundred - not defraged - files together, so pacman can read them real fast instead of doing disc-hopping. It is more a dog-keeper-problem.


Frumpus addict
[mu'.krum.pus], [frum.pus]

Offline

#28 2005-06-07 16:58:39

phrakture
Arch Overlord
From: behind you
Registered: 2003-10-29
Posts: 7,879
Website

Re: pacman-defrag [obsolete - use pacman-optimize]

cactus wrote:

I cant believe we are still talking about defragging a filesystem in this day and age.. *sigh*
Does this only happen with reiserfs? I was led to believe that ext3, jfs, and others were not in need of defrags, because they somehow placed consecutively accessed blocks closer together as a general rule..

it's a tad different here... under windows, fragmentation happens when individual files get their nodes scattered across the disk.
that doesn't happen with the FSes you mentioned... the inodes stay consecutive, or as close as possible.  However, that is for an individual file.  A single file will never get fragmented, whereas a series of multiple files which are accessed in order can suffer from the same fragmentation that a single file on a FAT partition can suffer from.

So, like i3839 said, if we were to tar the whole bloody thing, and read from the tar file, it will never become fragmented, because the FS sees it as one file.

Hmmm, it might be possible to archive the thing up and mount it with some funky loop-back device at /var/lib/pacman.... *scratches chin*

Offline

#29 2005-06-07 21:43:47

Vinny
Member
From: the fifth floor
Registered: 2005-04-25
Posts: 29

Re: pacman-defrag [obsolete - use pacman-optimize]

A loop-mounted file will work for this ; no need to use tar.

cd /var/lib
dd if=/dev/zero of=Paclib.img bs=1024k count=60
mke2fs -m 0 -b 1024 -i 1024 -O dir_index -v -L VLPimg Paclib.img
mv pacman pacman.old
mkdir pacman
mount -o loop Paclib.img pacman
cd pacman.old ; cp -a * ../pacman

(edit) nvm about inode size, I was remembering something unrelated.

Offline

#30 2005-06-07 22:34:28

cactus
Taco Eater
From: t͈̫̹ͨa͖͕͎̱͈ͨ͆ć̥̖̝o̫̫̼s͈̭̱̞͍̃!̰
Registered: 2004-05-25
Posts: 4,622
Website

Re: pacman-defrag [obsolete - use pacman-optimize]

A loopback filesystem for storing application specific information?
This is what databases were built for people...

simply...amazing..


"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍

Offline

#31 2005-06-07 22:57:35

Vinny
Member
From: the fifth floor
Registered: 2005-04-25
Posts: 29

Re: pacman-defrag [obsolete - use pacman-optimize]

Well, yes it's not the ideal solution but it can be done today
not when(if) pacman DB support is ready.
YMMV smile

Offline

#32 2005-06-08 12:03:06

Dreameen
Member
From: Poland
Registered: 2004-09-06
Posts: 252

Re: pacman-defrag [obsolete - use pacman-optimize]

Thanks for all the responses to my initial question about linux fs defragmentation. Now I see that that the files don't get fragmented, they are just scattered around the hdd in a random manner.

Still, I dare say that linux needs some kind of tool that would deal with the scattered files and put them close to each other in a logical way. All the libraries(/usr/lib) for example, could be reordered in the same way as pacman database files are. All the binaries, headers, icons could be reorganised similarly. The problem with this kind of tool is that it would need to know which groups of files should be put together, therefore the files would need some kind of a tag(lib, icon, header etc..) that groups them in meaningful chunks. Anyway, I don't know if it makes any sense to you lol I also doubt if the speed increase would make up for all the hard work, I mean linux filesystems are already fast.

Offline

#33 2005-06-09 03:51:53

phrakture
Arch Overlord
From: behind you
Registered: 2003-10-29
Posts: 7,879
Website

Re: pacman-defrag [obsolete - use pacman-optimize]

Dreameen wrote:

Still, I dare say that linux needs some kind of tool that would deal with the scattered files and put them close to each other in a logical way. All the libraries(/usr/lib) for example, could be reordered in the same way as pacman database files are. All the binaries, headers, icons could be reorganised similarly. The problem with this kind of tool is that it would need to know which groups of files should be put together, therefore the files would need some kind of a tag(lib, icon, header etc..) that groups them in meaningful chunks. Anyway, I don't know if it makes any sense to you lol I also doubt if the speed increase would make up for all the hard work, I mean linux filesystems are already fast.

hmmm that won't help anything.... libraries (ideally) are only located once on the disk, then cached (ld.so.cache or something) - binaries are usually only run one at a time, and smaller programs usually stay loaded in memory anyway...

think about the binary stuff this way: if I run firefox and then run thunderbird, it doesn't matter where they are, they still have to be individually found because they're not being loaded at the same time (in the case of pacman, it's bascially saying "open /lib/pacman/*")... it'd be like driving to Bill's house, driving home, then driving to Fred's house - it doesn't matter if Bill and Fred live next door, you still need to make the whole trek.

Offline

#34 2005-06-19 16:59:28

T-Dawg
Forum Fellow
From: Charlotte, NC
Registered: 2005-01-29
Posts: 2,736

Re: pacman-defrag [obsolete - use pacman-optimize]

I've made this script available as a package with a PKGBUILD. It and my other projects can be found here

Offline

#35 2005-07-02 15:04:13

FoPref
Member
From: Erlangen / Germany
Registered: 2004-03-24
Posts: 96
Website

Re: pacman-defrag [obsolete - use pacman-optimize]

I now tried the actual version on your website and it still doesn't work for me.

packages tar, coreutils and bash were re-installed.


I just don't know what to do. I hade errors on my filesystem but resolved them lately.


cu
Ford Prefect

Offline

#36 2005-07-02 16:20:51

T-Dawg
Forum Fellow
From: Charlotte, NC
Registered: 2005-01-29
Posts: 2,736

Re: pacman-defrag [obsolete - use pacman-optimize]

damn, dude I have no clue whats wrong. Have you tried reinstalling diffutils? What kernel are you using?

Has anyone else had this problem?

Offline

#37 2005-07-02 16:40:11

FoPref
Member
From: Erlangen / Germany
Registered: 2004-03-24
Posts: 96
Website

Re: pacman-defrag [obsolete - use pacman-optimize]

didn't help.

Linux lanrules 2.6.9 #1 Thu Oct 28 02:15:28 CEST 2004 i686 AMD Athlon(tm) XP 1900+ AuthenticAMD GNU/Linux

selfmade kernel 2.6.9

Offline

#38 2005-07-02 16:51:46

T-Dawg
Forum Fellow
From: Charlotte, NC
Registered: 2005-01-29
Posts: 2,736

Re: pacman-defrag [obsolete - use pacman-optimize]

in the script where it says UNCOMMENT TO SEE SUMS exactly what does it show?
It may be worth a try to install the stock arch kernel (and reboot) to elminate that from the picture.

If nothing else works you can comment lines 90 to 91, 95-116, and 118-124. That will take out the intregrity check all together. Thats of course based off my latest (1.6-2). I haven't tested it so it'll be a good a idea to back up /var/lib/pacman/local in case it craps out.

EDIT: do a diff on the same file like so:

diff foo.file foo.file; [ $? -ne 0 ] && echo "You shouldn't see this"

tell me if you see the message.

Offline

#39 2005-07-02 20:33:12

FoPref
Member
From: Erlangen / Germany
Registered: 2004-03-24
Posts: 96
Website

Re: pacman-defrag [obsolete - use pacman-optimize]

uncommenting doesn't change anything (ie nothing additionally shown)

my diff works correct


won't start installing and booting other kernel images (too lazy, sorry)...
think it's risky to try without the check, as it fails of different checksums.

better want to try other things first.

cu
Ford Prefect

Offline

#40 2005-07-02 21:54:06

T-Dawg
Forum Fellow
From: Charlotte, NC
Registered: 2005-01-29
Posts: 2,736

Re: pacman-defrag [obsolete - use pacman-optimize]

It must not be extracting the correct data to diff against. Try this:

replace lines 98 and 99 with

olddb=$(cat old.sums | sed 's|s*old.tar||g')
newdb=$(cat new.sums | sed 's|s*new.tar||g')

Offline

#41 2005-07-02 23:31:42

FoPref
Member
From: Erlangen / Germany
Registered: 2004-03-24
Posts: 96
Website

Re: pacman-defrag [obsolete - use pacman-optimize]

didn't change anything.
same different checksums were given with the old and the new lines 98/99

Offline

#42 2005-07-03 00:25:16

T-Dawg
Forum Fellow
From: Charlotte, NC
Registered: 2005-01-29
Posts: 2,736

Re: pacman-defrag [obsolete - use pacman-optimize]

FoPref wrote:

same different checksums were given with the old and the new lines 98/99

I thought you said earlier on page 1 that the sums appeared to be indentical but the script thought they were wrong? If thats the case, there's very little chance its this but more likely a major problem elsewhere.
Anyways, I reviewed and played around with my code many times and I still can't see how it could be falsely crapping out on you with the integrity check. I'm sorry but I'm completely out of ideas.

In the process of all this I've found some minor cosmetics and coding improvements which I will finish up and probably upload onto the server later today.

Offline

#43 2005-07-03 06:14:52

lucke
Member
From: Poland
Registered: 2004-11-30
Posts: 4,018

Re: pacman-defrag [obsolete - use pacman-optimize]

Mmm, I don't know if you're aware guys, that Judd already included such script in the newest pacman. Probably inspired by and based on Penguin's work.

Just to point it out, as I see that you're still fighting with it ;-)

Offline

#44 2005-07-03 08:46:22

Cam
Member
From: Brisbane, Aus
Registered: 2004-12-21
Posts: 658
Website

Re: pacman-defrag [obsolete - use pacman-optimize]

I had a problem with pacman-optimize but I just did manually what the script did and it worked no worries. I've done some mucking around with my pacman DB files though so I wasn't going to report it as a bug unless other people got similar results.

Offline

#45 2005-07-03 11:47:43

T-Dawg
Forum Fellow
From: Charlotte, NC
Registered: 2005-01-29
Posts: 2,736

Re: pacman-defrag [obsolete - use pacman-optimize]

lucke wrote:

Mmm, I don't know if you're aware guys, that Judd already included such script in the newest pacman.

Well that figures. roll I see its very similar to mine except he tared the folder and piped the sums in a much more sensible fashion.

Cam: Just curious, what problems were you having with pacman-optimize? Did my script do the same?

At any rate, people should not use this anymore. I'll remove it from my site and the wiki when I get a chance.

Offline

#46 2005-07-03 12:15:00

Cam
Member
From: Brisbane, Aus
Registered: 2004-12-21
Posts: 658
Website

Re: pacman-defrag [obsolete - use pacman-optimize]

I never tried yours, I planeed on doing it myself since it's not really a physically demanding job but I never got around do it. Seeing that pacman 2.9.6 came with a similar script I gave it a shot but crapped out on the integrity check. I manually ran diff /tmp/pacsums* and they do differ so beats me whats happening but yeah, just cp'd, rm'd and mv'd it myself this arvo for the same results smile

Offline

#47 2005-07-03 13:42:28

T-Dawg
Forum Fellow
From: Charlotte, NC
Registered: 2005-01-29
Posts: 2,736

Re: pacman-defrag [obsolete - use pacman-optimize]

Ok. I removed this script from my website and will do the same to the wiki entry once its up and running again.

I'm going to lock this thread and advise everyone to use pacman-optimize included in pacman 2.9.2-2 instead.

Offline

#48 2005-07-04 03:39:37

rasat
Forum Fellow
From: Finland, working in Romania
Registered: 2002-12-27
Posts: 2,293
Website

Re: pacman-defrag [obsolete - use pacman-optimize]

Congratulation!
Nice to see an idea of user contribution applied in the official.


Markku

Offline

Board footer

Powered by FluxBB