You are not logged in.

#1 2004-12-27 00:40:57

Pajaro
Member
Registered: 2004-04-21
Posts: 884

why does not pacman use a db?

I'm wondeing why pacman stores all information in many diferent files so loading this info takes while everytime you run pacman, instead of using a single file db?

I've been thinking about for a very long while and the only answer I find is that it allows you to easily modify the db. I don't think this is a good reason for it. I'm tired of avoidable time wasting.

Offline

#2 2004-12-27 01:06:52

contrasutra
Member
From: New Jersey
Registered: 2003-07-26
Posts: 507

Re: why does not pacman use a db?

Well, the reasons I can see (this has been discussed):

1. It's easier to modify - don't discount that
2. It's less vulnerable to corruption
3. It's easier to change the DB format
4. It's more compatible. Easier to work with text files than with a DB, therefore:
5. More futureproof
6. It's simpler, code wise (opinion, I guess).

I don't have an opinion either way, but Pacman seems fine the way it is, so is there a very good reason to switch, apart from speed?


"Contrary to popular belief, penguins are not the salvation of modern technology.  Neither do they throw parties for the urban proletariat."

Offline

#3 2004-12-27 02:02:46

Pajaro
Member
Registered: 2004-04-21
Posts: 884

Re: why does not pacman use a db?

My opinions to:

1. (0) I've never modified it and I think I'll never do
2. (0.5) Completely true. So why not just use file db only with local? current, extra and turs are a copy of the db in the server, so you can recover them if they get corrupted.
3. (0) a "hashes system" has the same structure than a filesystem, so you can change the structure with the same advantage
4. (1) yes, it's more compatible.
5. (1) mmm... futureproof? I don't see where this point comes from.
6. (0) depends on the way you store your data and the capabilities of the language. In my experience, scripting languages are the best ones for this.

You win 2.5 points, but substracting 1 point comming from speed, you win me for 1.5 points that we could discuss, but I think that this different point of view is mainly subjective, so why wasting time?

Thanks for your info

Offline

#4 2004-12-27 02:58:00

iphitus
Forum Fellow
From: Melbourne, Australia
Registered: 2004-10-09
Posts: 4,927

Re: why does not pacman use a db?

It's faster. I've been using Mandrake for years and sorry dude, but pacman is much faster resolving deps, finding packages, installing and updating. Debian wasnt much better either.

and it agrees with unix philosophy tongue

keep it simple stupid!

iphitus

Offline

#5 2004-12-27 15:14:40

Pajaro
Member
Registered: 2004-04-21
Posts: 884

Re: why does not pacman use a db?

simpler is relative. We won't go anywhere  :shock:

Offline

#6 2004-12-27 16:18:08

i3839
Member
Registered: 2004-02-04
Posts: 1,185

Re: why does not pacman use a db?

Pacman needs to read its whole database each time it does something, once the files are in memory and cached things are fast. But before it's at that point I need to wait 5 seconds (-Ss takes 15 seconds). Well, granted, compared to the startup time of Firefox that's a joke, but still. The system isn't scalable, so if there will come much more packages then it will become too slow, if the number of packages more or less stays the same then it's not needed to make Pacman faster, as it's not yet too slow.

As a side note, the single file db can be generated from the current pkg database, so no need to change any format. There are also other ways to make pacman faster.

Offline

#7 2004-12-27 17:46:00

dp
Member
From: Zürich, Switzerland
Registered: 2003-05-27
Posts: 3,378
Website

Re: why does not pacman use a db?

i3839 wrote:

Pacman needs to read its whole database each time it does something, once the files are in memory and cached things are fast. But before it's at that point I need to wait 5 seconds (-Ss takes 15 seconds). Well, granted, compared to the startup time of Firefox that's a joke, but still. The system isn't scalable, so if there will come much more packages then it will become too slow, if the number of packages more or less stays the same then it's not needed to make Pacman faster, as it's not yet too slow.

As a side note, the single file db can be generated from the current pkg database, so no need to change any format. There are also other ways to make pacman faster.

good spoken - i second this

and about futureproofity: db's have standards


The impossible missions are the only ones which succeed.

Offline

#8 2004-12-27 18:03:23

vacant
Member
From: downstairs
Registered: 2004-11-05
Posts: 816

Re: why does not pacman use a db?

contrasutra wrote:

4. It's more compatible. Easier to work with text files than with a DB, therefore:

After years working in IT i've concluded this is crucial - K.I.S.S.

Offline

#9 2004-12-28 06:06:49

sarah31
Member
From: Middle of Canada
Registered: 2002-08-20
Posts: 2,975
Website

Re: why does not pacman use a db?

why is it that these questions keep coming up?

why don't people do searches here  and on the ML to prevent filling up the forum db with multiple copies of the same question every two months?

Why do people keeps asking questions that the majority of us users cannot answer because we are not responsible for the infrastructure of arch?

I don't mean to sound like some grump but hell it is so monotinous. People should know that the developers that are responsible for pacman and some of the other key continuous bitching points don't hang here.


AKA uknowme

I am not your friend

Offline

#10 2004-12-28 08:42:50

Pajaro
Member
Registered: 2004-04-21
Posts: 884

Re: why does not pacman use a db?

well, sarah, I think you are right, but I wouldn't have said this in this topic.

If people makes the same question every 2 months then people makes the same question every 2 months: you cannot change people roll

Keep the answer somewhere to copy-paste it when the next asks.

About pacman db, I'll try to make a faster version, just because I have to wait 40 seconds for pacman -Ss. Maybe just making an script that creates a ramdrive and mounts it in /var/lib/pacman will be enough (with local kept in hd, of course)

Offline

#11 2004-12-28 17:58:18

cactus
Taco Eater
From: t͈̫̹ͨa͖͕͎̱͈ͨ͆ć̥̖̝o̫̫̼s͈̭̱̞͍̃!̰
Registered: 2004-05-25
Posts: 4,622
Website

Re: why does not pacman use a db?

Pajaro wrote:

just because I have to wait 40 seconds for pacman -Ss.

wow..i never have to wait that long. Most of the wait time is the screen echo for me...as I watch everything scroll by.
For pacman -Ss packagename, it does take a little bit...but by no means 40 seconds. Maybe 2 or 3...if that.

My box is of average speed..an athlon 1800 (1.57 Ghz). Being that arch is 686, it is aimed at a bit higher performance systems..check archstats and you will see that the average system is pretty close to mine.

I have never really had pacman take a long time, unless the repo had difficulty being contacted. I don't expect Pacman to magically speed up network queries.


"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍

Offline

#12 2004-12-28 19:31:12

dp
Member
From: Zürich, Switzerland
Registered: 2003-05-27
Posts: 3,378
Website

Re: why does not pacman use a db?

i have similar times for -Ss:

[damir@Asteraceae /]$ date && pacman -Ss libvisual && date
Tue Dec 28 20:30:05 CET 2004
extra/libvisual 0.1.7-1
    abstraction library that comes between applications and audio visualisation
    plugins
extra/libvisual-plugins 0.1.7-1
    xmms plugin for libvisual
extra/libvisual-xmms 0.1.7-1
    xmms plugin for libvisual
Tue Dec 28 20:30:42 CET 2004

second run:

[damir@Asteraceae /]$ date && pacman -Ss libvisual && date
Tue Dec 28 20:31:55 CET 2004
extra/libvisual 0.1.7-1
    abstraction library that comes between applications and audio visualisation
    plugins
extra/libvisual-plugins 0.1.7-1
    xmms plugin for libvisual
extra/libvisual-xmms 0.1.7-1
    xmms plugin for libvisual
Tue Dec 28 20:32:07 CET 2004

The impossible missions are the only ones which succeed.

Offline

#13 2004-12-28 19:35:03

Mr Green
Forum Fellow
From: U.K.
Registered: 2003-12-21
Posts: 5,893
Website

Re: why does not pacman use a db?

time pacman -Ss libvisual
extra/libvisual 0.1.7-1
    abstraction library that comes between applications and audio visualisation
    plugins
extra/libvisual-plugins 0.1.7-1
    xmms plugin for libvisual
extra/libvisual-xmms 0.1.7-1
    xmms plugin for libvisual

real    0m5.180s
user    0m0.187s
sys     0m0.470s

 time pacman -Ss libvisual
extra/libvisual 0.1.7-1
    abstraction library that comes between applications and audio visualisation
    plugins
extra/libvisual-plugins 0.1.7-1
    xmms plugin for libvisual
extra/libvisual-xmms 0.1.7-1
    xmms plugin for libvisual

real    0m0.483s
user    0m0.179s
sys     0m0.295s

Whats the problem  :?:


Mr Green

Offline

#14 2004-12-28 19:52:01

i3839
Member
Registered: 2004-02-04
Posts: 1,185

Re: why does not pacman use a db?

The problem is that you run it when (most) files are already in ram. Then it takes less than one second, no matter what pc you have. With no database files cached it takes 18 second to do a search for me. How long it takes for you depends mostly on how fast your hd is and how many files are already/still in ram. 40 seconds is extremely long, sure you have dma enabled? (E.g. do hdparm -i or cat /proc/ide/hdc/settings)

Offline

#15 2004-12-28 20:01:03

kth5
Member
Registered: 2004-04-29
Posts: 657
Website

Re: why does not pacman use a db?

yup, i don't see something wrong with speed either:

Athlon 2200Xp+ (1.8Ghz)

[kth5@subhime ~]$ time pacman -Ss libvisual
extra/libvisual 0.1.7-1
    abstraction library that comes between applications and audio visualisation plugins
extra/libvisual-plugins 0.1.7-1
    xmms plugin for libvisual
extra/libvisual-xmms 0.1.7-1
    xmms plugin for libvisual

real    0m1.149s
user    0m0.180s
sys     0m0.322s
[kth5@subhime ~]$ time pacman -Ss libvisual
extra/libvisual 0.1.7-1
    abstraction library that comes between applications and audio visualisation plugins
extra/libvisual-plugins 0.1.7-1
    xmms plugin for libvisual
extra/libvisual-xmms 0.1.7-1
    xmms plugin for libvisual

real    0m0.856s
user    0m0.166s
sys     0m0.240s

Pentium 3 (700Mhz)

[kth5@julia ~]$ time pacman -Ss libvisual
extra/libvisual 0.1.7-1
    abstraction library that comes between applications and audio visualisation plugins
extra/libvisual-plugins 0.1.7-1
    xmms plugin for libvisual
extra/libvisual-xmms 0.1.7-1
    xmms plugin for libvisual

real    0m1.921s
user    0m0.403s
sys     0m0.782s
[kth5@julia ~]$ time pacman -Ss libvisual
extra/libvisual 0.1.7-1
    abstraction library that comes between applications and audio visualisation plugins
extra/libvisual-plugins 0.1.7-1
    xmms plugin for libvisual
extra/libvisual-xmms 0.1.7-1
    xmms plugin for libvisual

real    0m0.867s
user    0m0.301s
sys     0m0.493s

both machines were running X.org + KDE 3.3.2 at the same time and were rebooted especially to make sure how muchf caching affects this.


I recognize that while theory and practice are, in theory, the same, they are, in practice, different. -Mark Mitchell

Offline

#16 2004-12-28 21:32:23

cactus
Taco Eater
From: t͈̫̹ͨa͖͕͎̱͈ͨ͆ć̥̖̝o̫̫̼s͈̭̱̞͍̃!̰
Registered: 2004-05-25
Posts: 4,622
Website

Re: why does not pacman use a db?

time pacman -Ss libvisual
extra/libvisual 0.1.7-1
    abstraction library that comes between applications and audio visualisation
    plugins
extra/libvisual-plugins 0.1.7-1
    xmms plugin for libvisual
extra/libvisual-xmms 0.1.7-1
    xmms plugin for libvisual
 
real    0m0.343s
user    0m0.139s
sys     0m0.199s

*shrug*


"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍

Offline

#17 2004-12-28 22:46:35

dp
Member
From: Zürich, Switzerland
Registered: 2003-05-27
Posts: 3,378
Website

Re: why does not pacman use a db?

the short times result, because you have some files already in ram

try stopping time of a -Ss just after boot of the system ... this is slow!


The impossible missions are the only ones which succeed.

Offline

#18 2004-12-28 23:31:31

cactus
Taco Eater
From: t͈̫̹ͨa͖͕͎̱͈ͨ͆ć̥̖̝o̫̫̼s͈̭̱̞͍̃!̰
Registered: 2004-05-25
Posts: 4,622
Website

Re: why does not pacman use a db?

Directed to dp:

dp wrote:

the short times result, because you have some files already in ram
try stopping time of a -Ss just after boot of the system ... this is slow!

kth5 wrote:

yup, i don't see something wrong with speed either:
Athlon 2200Xp+ (1.8Ghz)

....
real    0m1.149s
user    0m0.180s
sys     0m0.322s

....
real    0m0.856s
user    0m0.166s
sys     0m0.240s

Pentium 3 (700Mhz)

....
real    0m1.921s
user    0m0.403s
sys     0m0.782s

....
real    0m0.867s
user    0m0.301s
sys     0m0.493s

both machines were running X.org + KDE 3.3.2 at the same time and were rebooted especially to make sure how muchf caching affects this.

I can't personally test the caching issue right now, because my live server is running fine and I don't need to reboot it right now..
But I have never had to wait that long even immediately following a boot.

I understand if you are doing a pacman -Ssy packagename
but not a regular pacman -Ss packagename

Isn't the cache persistent over reboots? I thought pacman stored the contents of reponame.db.tar.gz locally, and it was only a process of re-reading those local files.
Doing a pacman -Ssy WOULD take longer, especially if poor networking conditions were present (bad connection, slow connection, etc), but that should not be blamed on the app.

But Just be glad you aren't forced to use yum! LOL


"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍

Offline

#19 2004-12-28 23:48:32

dp
Member
From: Zürich, Switzerland
Registered: 2003-05-27
Posts: 3,378
Website

Re: why does not pacman use a db?

.... were rebooted especially to make sure how muchf caching affects this.

sorry, i didn't realized this part  :oops:


The impossible missions are the only ones which succeed.

Offline

#20 2004-12-29 05:16:18

kth5
Member
Registered: 2004-04-29
Posts: 657
Website

Re: why does not pacman use a db?

could be that thenumber and size of misc. repos you have beside current and extra available per default in arch. i personally only use current and extra synched on a daily basis.


I recognize that while theory and practice are, in theory, the same, they are, in practice, different. -Mark Mitchell

Offline

#21 2004-12-29 05:51:07

iBertus
Member
From: Greenville, NC
Registered: 2004-11-04
Posts: 2,228

Re: why does not pacman use a db?

It would seem that the vast majority of time on my machine is spent doing the screen echos. In fact, pacman -Ss runs in 0m0.217s when doing this. Also, 0m3.422s is my runtime while printing to the terminal. It should be noted that I haven't rebooted in awhile but do a -Syu everyday.

Offline

#22 2004-12-29 19:33:21

apeiro
Daddy
From: Victoria, BC, Canada
Registered: 2002-08-12
Posts: 771
Website

Re: why does not pacman use a db?

Pacman can be slow to start when its databases are not cached.

This is mostly due to fragmentation, I think.  Pacman's "database" uses many small files, which can be easily fragmented (especially with reiserfs, IIRC -- correct me if I'm wrong).

Pacman doesn't have to stick to the filesystem db layout forever, but it does come in handy sometimes.  One idea was to use a gdbm database as the working cache of the real filesystem db.  This could minimize startup times, and serve as a persistent db cache over reboots.

Just an idea at this point, though...

Offline

#23 2004-12-29 23:04:27

dp
Member
From: Zürich, Switzerland
Registered: 2003-05-27
Posts: 3,378
Website

Re: why does not pacman use a db?

apeiro wrote:

Pacman can be slow to start when its databases are not cached.

This is mostly due to fragmentation, I think.  Pacman's "database" uses many small files, which can be easily fragmented (especially with reiserfs, IIRC -- correct me if I'm wrong).

this would explain it for me: i'm using reiserfs and have about 1200 pkgs installed - using 7 repos


The impossible missions are the only ones which succeed.

Offline

#24 2004-12-29 23:53:30

cactus
Taco Eater
From: t͈̫̹ͨa͖͕͎̱͈ͨ͆ć̥̖̝o̫̫̼s͈̭̱̞͍̃!̰
Registered: 2004-05-25
Posts: 4,622
Website

Re: why does not pacman use a db?

:shock:


"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍

Offline

#25 2004-12-30 00:58:20

iBertus
Member
From: Greenville, NC
Registered: 2004-11-04
Posts: 2,228

Re: why does not pacman use a db?

cactus wrote:

:shock:

Which part? The part about reiserfs being more prone to fragmentation or the part about dp's 1200 pkgs?  lol

Offline

Board footer

Powered by FluxBB