You are not logged in.

#1 2005-03-19 07:15:31

cactus
Taco Eater
From: t͈̫̹ͨa͖͕͎̱͈ͨ͆ć̥̖̝o̫̫̼s͈̭̱̞͍̃!̰
Registered: 2004-05-25
Posts: 4,622
Website

Pacman with db support?

What are the pros/cons of pacman with database support?
What databases would be viable and "the right choice"?

I am not convinced that db support is the answer to "omg pacman is slow" woes. I am concenred with people thinking db support is a panacea. I believe it has the potential to cuase as many problems as it sovles: Database corruption, locking issues, etc.

If a db WAS decided to be the best solution, I would probably recommend something like berkDB (berkeley db). It is more tested, and provides better cross platform support, than sqlite seems to.

Please, try to keep the discussion civil. I realize this is a subject potentially wrought with strong feelings and opinions.

thanks.


"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍

Offline

#2 2005-03-19 07:18:11

miqorz
Member
Registered: 2004-12-31
Posts: 475

Re: Pacman with db support?

I think it's a good IDEA that we should keep in the back of our minds but there's really no need to adopt this idea NOW or any time soon.

Pacman is way faster than most solutions out there (portage, anyone?) and using SQ Lite for this solution with threats of corrupt databases just seems very pre-mature.


http://wiki2.archlinux.org/

Read it. Love it. Live it. Or die.

Offline

#3 2005-03-19 07:27:11

cactus
Taco Eater
From: t͈̫̹ͨa͖͕͎̱͈ͨ͆ć̥̖̝o̫̫̼s͈̭̱̞͍̃!̰
Registered: 2004-05-25
Posts: 4,622
Website

Re: Pacman with db support?

yeah. the thing that scared me...I looked at the sqlite website, to see what kind of features they have..
Two most recent updates on their site both were issues that cause corrupted databases..
*shudders*

Also, what would the migration strategy be for the users currently using pacman in its current form? Automigration?
In the even of a corrupted database, how would it be fixed? Rebuilt?


"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍

Offline

#4 2005-03-19 07:31:54

Eugenia
Member
From: Bay Area, CA, USA
Registered: 2005-03-08
Posts: 74
Website

Re: Pacman with db support?

Pacman is fast when few packages are installed. But it gets significantly slower on a more 'normal' desktop installation, e.g. a desktop that has most gnome or kde packages installed. With the course of time, pacman is becoming slower not because its code is slow, but because the amount of tracked packages in the system becomes big.

For example, I am writing this from the PC where only XFce is installed (no gnome or kde). I haven't installed more than the bare minimum I would need: pacman is fast.

Move to my other two PCs, one with gnome and quite a few gnome third party packages,  and one with both gnome & kde and some of their third party packages. Pacman is too slow on these machines because of the amount of packages in them (which is significantly smaller than a stock fedora or mdk or suse installation btw). The hdd is overused, it pretty much brings the machine into a stall. Pretty much you can't use the PC very effectively when pacman is installing stuff, a problem that it's not present on most other distros or even other OSes when they do updates.

There are optimizations to be done to the current pacman architecture, but at some point in the future the limit will again be hit. Flat files can only do so much for speed when you get more and more packages weekly. Be it 1 year, or 2 years, the problem will still be there.

However, using an SQL backend, that can handle thousands of entries in milliseconds, I believe it's a sensible solution, given that  the SQL engine used is indeed lite, small, secure, fast, stable and it's only used to track deps/package descriptions and not big files. I advocate that SQLite3 (or BDB4) are good solutions given the needs, but I would avoid mySQL or PostgreSQL because these are full-pledge DB solutions, they are a bit overkill for what pacman would need.

Bug reports that discuss the matter:

http://bugs.archlinux.org/index.php?do= … a=comments
http://bugs.archlinux.org/index.php?do= … a=comments

Offline

#5 2005-03-19 07:35:29

Eugenia
Member
From: Bay Area, CA, USA
Registered: 2005-03-08
Posts: 74
Website

Re: Pacman with db support?

>Two most recent updates on their site both were
>issues that cause corrupted databases..

Yup, sometimes they have bugs too. This is why I suggested to Judd --if he would go with SQLite--, to use a very specific version of SQLite that is known to be really stable, and statically link pacman3 to it. In other words, the wise thing to do here, for this specific case, it would be to only upgrade SQLite for pacman once a year or so (only when really needed), and only after extensive testing. This is one case where statically linking would be a good idea and extensive testing with the version at  linked.

Offline

#6 2005-03-19 07:39:34

miqorz
Member
Registered: 2004-12-31
Posts: 475

Re: Pacman with db support?

If.. -IF- this becomes a reality (I'm not even saying it SHOULD be a reality) we should absolutely use berkDB. It seems to be the solution over-all.

But adding this just because it seems a little slow to a couple people seems silly to me. I have a full blown desktop system and pacman runs just as great as when I first installed it in December.

The current pacman solution is MORE than okay for now and far into the future.

We should just keep the idea open for sometime later in the next couple years.

Oh and I've NEVER experienced a slowness in my desktop when installing alot of packages.. I have alot of gnome libs installed and during the 2.10.0 push my system seemed just as fast as usual.


http://wiki2.archlinux.org/

Read it. Love it. Live it. Or die.

Offline

#7 2005-03-19 07:49:25

cactus
Taco Eater
From: t͈̫̹ͨa͖͕͎̱͈ͨ͆ć̥̖̝o̫̫̼s͈̭̱̞͍̃!̰
Registered: 2004-05-25
Posts: 4,622
Website

Re: Pacman with db support?

I think sql support would be overkill. All we are talking about is basically string hashing right?
So, something like gdbm would probably be good enough. gdbm has locking support, if I remember correctly, so you could ensure that only one instance of pacman is accessing it at once.

Granted, I am still not convinced db support is needed. Cleanup of pacman IS needed, but that is pretty well understood, and the "libidization" supposedly had this as a focus already.

You keep saying SQL backend. DB backend is one thing, but SQL? If db support was decided a "must have feature", then I would suggest a simple, well tested, and highly efficient file db solution.
something like dbm, gdbm, or berkdb.

Saying, "yup, sometimes they have bugs too" is slightly indecorous, and not for promoting reasonable discussion. I still don't see much in your argument other than "omg. pacman is slow. SQLite will make it roxor!"

I want sound arguments for/against. Sound arguments supporting one db over another.
Is that too much to ask for?


"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍

Offline

#8 2005-03-19 07:53:09

miqorz
Member
Registered: 2004-12-31
Posts: 475

Re: Pacman with db support?

cactus,

You're the best moderator ever.  8)


http://wiki2.archlinux.org/

Read it. Love it. Live it. Or die.

Offline

#9 2005-03-19 08:16:27

iphitus
Forum Fellow
From: Melbourne, Australia
Registered: 2004-10-09
Posts: 4,927

Re: Pacman with db support?

as far as I know berkelydb is accessed and used via sql too....

Offline

#10 2005-03-19 08:19:20

cactus
Taco Eater
From: t͈̫̹ͨa͖͕͎̱͈ͨ͆ć̥̖̝o̫̫̼s͈̭̱̞͍̃!̰
Registered: 2004-05-25
Posts: 4,622
Website

Re: Pacman with db support?

iphitus wrote:

as far as I know berkelydb is sql based too....

It doesn't appear to be.

From the supposed "Key features" list on the berkeley db site (http://www.sleepycat.com/products/data.shtml)

Stores data in the application's native representation, eliminating the need to translate into a foreign object or relational format.


"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍

Offline

#11 2005-03-19 08:32:50

iphitus
Forum Fellow
From: Melbourne, Australia
Registered: 2004-10-09
Posts: 4,927

Re: Pacman with db support?

cactus wrote:
iphitus wrote:

as far as I know berkelydb is sql based too....

It doesn't appear to be.

From the supposed "Key features" list on the berkeley db site (http://www.sleepycat.com/products/data.shtml)

Stores data in the application's native representation, eliminating the need to translate into a foreign object or relational format.

I musta edited my comment as you were posting, sql is a syntax for accessing databases, not storing them. I have been under the impression that you can access berkelydb via sql.

Offline

#12 2005-03-19 08:34:48

Eugenia
Member
From: Bay Area, CA, USA
Registered: 2005-03-08
Posts: 74
Website

Re: Pacman with db support?

Personally, I don't mind if something is SQL-based or not (I think I didn't make this very clear), what I was discussing was a db-based system that is fast, stable and lite that doesn't fragment as much as most file systems do. BDB4 is a good solution indeed, and I believe SQLite is a good one too. I have heard that Metakit is good as well, but I have never used it in any way so I can't really comment about it. Gbdm and dbm Cactus mentioned, I don't really know them.

Offline

#13 2005-03-19 09:51:50

miqorz
Member
Registered: 2004-12-31
Posts: 475

Re: Pacman with db support?

I'm still not even remotely convinced this is needed at all.


http://wiki2.archlinux.org/

Read it. Love it. Live it. Or die.

Offline

#14 2005-03-19 09:55:10

iphitus
Forum Fellow
From: Melbourne, Australia
Registered: 2004-10-09
Posts: 4,927

Re: Pacman with db support?

miqorz wrote:

I'm still not even remotely convinced this is needed at all.

well should arch expand to become much larger, and the number of packages increase substantially, it would speed it up a bit.

Thats pretty much the only argument going for it. Getting pacman a bit more 'futureproofed'

Offline

#15 2005-03-19 09:56:38

Celti
Member
From: Phoenix, AZ, USA
Registered: 2004-02-28
Posts: 43
Website

Re: Pacman with db support?

Nor am I. I have had a metric arse-ton of packages installed, including both KDE and Gnome, and pacman took maybe 15 seconds longer than usual.  My system is nothing special, an Athlon XP 2600, with a gig of RAM. I don't see the need. It's still faster than portage, or apt, or a bunch of others. One of the advantages of the directories full of files pacman has is that scripts can go through them too, without having to use some kludgy library of SQL access for bash. Switching to a db would remove this advantage, for a minor speed increase. I do not see the need.

~Celti


“A stupid person can make only certain, limited types of errors; the mistakes open to a clever fellow are far broader. But to the one who knows how smart he is compared to everyone else, the possibilities for true idiocy are boundless.”  —S.K.Z. Brust

Offline

#16 2005-03-19 11:07:57

cmp
Member
Registered: 2005-01-03
Posts: 350

Re: Pacman with db support?

my system is under heavy load, when I install packages, but it could be due to my lack of RAM (only 190mb SDRAM), but the real problem is the startup time, because it takes half a minute before pacman does anything, if I install some more packages.
I don't know who, but someone said, that in other systems the package manager would consume less resources, at least for fedora the basic package manager consumes even more than arch and yum is crappy all the way, it takes long to start and then is freaking resource hungry.

Offline

#17 2005-03-19 11:28:26

Moo-Crumpus
Member
From: Hessen / Germany
Registered: 2003-12-01
Posts: 1,487

Re: Pacman with db support?

All I want to do running pacman is install something here and then without any hussle. Who cares if it takes 1 minute or 1 minute and 20 seconds to do a query for some apps. Pacman is famous for it's ease.

I would hate to bundle pacman with a database system. I don't want to run a db just to install packages. Wasn't it an arch philosophy to keep it as simple and easy as possible. Furthermore I would hate to see pacman bundled with a database system I would never choose as a user. And I would not like to run several databases, one for pacman, another one for my own databases, and who knows, maybe eugenia will tell us the next day syslog messages have to be stored in another database, too, to be really really cool and fancy.

Speaking for myself, this is not the style of archlinux.


Frumpus addict
[mu'.krum.pus], [frum.pus]

Offline

#18 2005-03-19 13:59:12

zeppelin
Member
From: Athens, Greece
Registered: 2004-03-05
Posts: 807
Website

Re: Pacman with db support?

Celti wrote:

I don't see the need.~Celti

Celti and other guys. Did you read Raven's post in bugs or not? I would like to hear you comments after reading it and not before. migorz I hope you don't attack me too

http://bugs.archlinux.org/index.php?do= … ments#4407

thank you

Offline

#19 2005-03-19 14:22:02

Moo-Crumpus
Member
From: Hessen / Germany
Registered: 2003-12-01
Posts: 1,487

Re: Pacman with db support?

zeppelin wrote:
Celti wrote:

I don't see the need.~Celti

Celti and other guys. Did you read Raven's post in bugs or not? I would like to hear you comments after reading it and not before. migorz I hope you don't attack me too

http://bugs.archlinux.org/index.php?do= … ments#4407

thank you

I read it and can't see how a db backend should help if code is a little buggy.


Frumpus addict
[mu'.krum.pus], [frum.pus]

Offline

#20 2005-03-19 14:57:25

beniro
Member
From: St. Petersburg, FL, USA
Registered: 2002-12-31
Posts: 313

Re: Pacman with db support?

One thing I have been noticing recently is alot of disk usage when I pacman -Syu (which is mentioned in the bug discussion linked to above).  I'm not sure that I'm arguing for DB support in pacman, but it the high disk usage is a bit...distressing.

Using a light, statically-linked DB (as per Eugenia's suggestion) would help in with the disk usage, correct?  I'm asking this question independent of whether or not anyone is in favor of DB support in pacman.  Or is the disk usage from some other check pacman is running?

The benefit of DB support is simply speed, speed, speed.  Which could be useful, especially as the repos (official and user) grow in size.

The drawback is we could end up with something that is just one more thing to maintain.  Any addition should be considered carefully, because of Arch's KISS ideology.  Which is what we're doing, of course.

Wouldn't it be easier to wait for the libidized version of pacman (still happening, right?), adn then write a frontend with DB support???

I'm not as technically proficient as some here, so feel free to trample me under foot.  smile

Offline

#21 2005-03-19 18:10:49

cactus
Taco Eater
From: t͈̫̹ͨa͖͕͎̱͈ͨ͆ć̥̖̝o̫̫̼s͈̭̱̞͍̃!̰
Registered: 2004-05-25
Posts: 4,622
Website

Re: Pacman with db support?

Well, the bug report mentions that the code rereads all the flat files, every time after a package is installed. This is far too much. After the first read, some type of caching should be taking place.

I think it is hard to qualify the benefits of a database solution, given the current state as reported in the bugtracker. I still  have not been convinced, as nobody has provided any evidence beyond their opinion.

So far, what I seem to be hearing is:

con-
KISS philosophy of arch
Possible db corruption issues
small potential to cause porting issues to alternate platforms, due to endianness, or other issues (a good choice in db would negate this con)

pro-
*probable* speed increase (benchmarking not possible without some implementation)


"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍

Offline

#22 2005-03-19 18:24:45

Dusty
Schwag Merchant
From: Medicine Hat, Alberta, Canada
Registered: 2004-01-18
Posts: 5,986
Website

Re: Pacman with db support?

Again, because I've had no problems with pacman, I'm finding it hard to justify the need for this. I read the bug report; as cactus says, it sounds like it would be possible to optimize the existing pacman code WITHOUT introducing a third party database dependency.  In my mind, this should be done first; optimize pacman to run better using the current system. If that doesn't speed it up enough, then investigate other solutions, including a database.

Since aperio knows this code better than anyone, he can probably identify bottlenecks that haven't even been seen. From past examples of how he applies a patch (look at the patch, rewrite it to be better), it wouldn't surprise me if he can resolve this issue without adding a db. If he does choose to add a database, I'll guarantee it'll be the right decision, he hasn't been wrong yet. ;-)

Dusty

Offline

#23 2005-03-19 18:50:00

beniro
Member
From: St. Petersburg, FL, USA
Registered: 2002-12-31
Posts: 313

Re: Pacman with db support?

I agree...I trust 100% that A) the situation can be improved with some optimization and B) that Apeiro knows WTF he's doing.

And...

Pacman is still going to be libidized at some point???  In this case, the flexibility for a frontend to use a DB to store pkg lists and info could be provided.  This would make this whole thing a bit of a moot point, because then everyone could use whichever frontend they'd like (provided it exists...duh!  smile  ), everything from simple console all the way to a full GUI + DB.

Offline

#24 2005-03-19 19:13:54

phrakture
Arch Overlord
From: behind you
Registered: 2003-10-29
Posts: 7,879
Website

Re: Pacman with db support?

Oi, after reading the other locked thread, I am reluctant to discuss here... but I'd just like to say, to those who thing DB == faster pacman, I'd like you to prove it.  Back in the earlier days, I used to work on VMS systems... the OS stored database info in structured flat files.  At one point we purchased some obscure database software that implemented a real simple database for VMS... needless to say, it increased or codebase of all the little progams by alot... and there was no noticable speed difference.  Some times the flat file lookup was faster.
I think what people are pining for is not really a full balls-to-the-wall database, but indexing support.  It wouldn't be that hard to create an index file from the current info, used only for searching... hell slocate does it...

Offline

#25 2005-03-19 19:25:56

beniro
Member
From: St. Petersburg, FL, USA
Registered: 2002-12-31
Posts: 313

Re: Pacman with db support?

hell slocate does it...

Really?  I thought it used a true DB...slocate is obviously fast as all get out...and I have many, many more files than pacman will have to deal with anytime soon.

I guess pacman's current excess of disk usage isn't due to a lack of some extra technology...it just needs a bit of cleanup.

Offline

Board footer

Powered by FluxBB