Pacman + DB & External scriptability

awalk · 2005-11-04 06:41:49

Howdy.

I've been thinking about this one for quite some time - how to have Pacman work with a database (e.g. sqlite), whilst allowing the current functionality (namely, ease of interfacing with Pacman info via external scripts).

I know the idea of using a DB with Pacman has been debated before, but I never saw any good answers to the age old external script issue. I'm going to briefly explain two approaches one could take, and criticise them (before others do!)

Description

The way I see it, there are two ways this can be approached.

· Store all package information in a sqlite (or comparable) database, and have Pacman interface with this directly. Create a FUSE-based pseudo filesystem to generate this information in the same hierchical filesystem format as Pacman deals with now.
· Duplicate all data. One lot goes in the database, to be used by Pacman. The other lot in the hierarchical filesystem (standard filesystem, not a FUSE pseudo FS).

Criticism

The first problem with the first approach is dependencies. It would require a database, and it would require FUSE. FUSE is now in the 2.6.14 kernel, but libfuse is still required.

The second problem (this covers both approaches) is complexity. Both add a fair bit of complexy. Especially in the first case, as a FS would need to be written (albeit a relatively simple one).

With the second approach, you might think it sounds brittle and susceptible to synchronisation issues (by which I mean the two data sets becoming out of synch). This could be rectified by having some sort of daemon using Inotify to keep an eye on one dataset, and update the other accordingly. Again, more complexity.

I think the first approach would be pretty neat to go with, but again, probably too much complexity for too little gain. Any comments?

cactus · 2006-01-15 06:18:30

since sqlite comes with a command line interface, I dont see why there would even need to be a filesystem component. Just have scripts interface directly with the sqlitedb for reading and such...

awalk · 2006-01-15 06:26:20

Aye, you could do that easily enough. But it's not transparent, which is what I was aiming for. i.e. people could reuse their existing scripts with no changes.

phrakture · 2006-01-15 09:23:53

I like the idea of a fuse filesystem - however, please not that pacman 3.0 may or may not change the database situation.

test1000 · 2006-01-22 16:43:10

well, if it needed fuse I couldn't have chroot'ed my arch linux installation with my INSERT rescue-cd and downloaded packages which fixed my install after i unintentionally b0rked the install It's not KISS, and therefore i don't like it..

iphitus · 2006-01-22 23:54:26

Aye, so fuse breaks recovery situations... whjch very effectively rules it out.

Leaving sqlite.. oh lord this has been argued to death. Why oh why do we not just leave the current system and leave it up to judd. He isnt going to change it any sooner because we start another thread about it, he'll change it when he can and want to.

iphitus

arooaroo · 2006-01-23 09:46:14

iphitus wrote:

... Why oh why do we not just leave the current system and leave it up to judd. He isnt going to change it any sooner because we start another thread about it, he'll change it when he can and want to.

I think it's worth starting up Yet Another Thread on this issue. Judd's success in creating an excellent OS around his excellent package manager means that there's a lot of packages available for Arch now.

The pacman lib has been a worthy "next step" in the evolution for pacman, but I do think that the next issue to address is back-end db. I'm not cussing Judd for keeping things simple - I'm a developer too, and I know I would have gone down a similar route. I suppose that I personally would now revisit this area if I were the maintainer because of the obvious bottleneck.

I don't think this is controversial, as Judd as commented about this himself in previous debates, and IIRC he said that his original long term aim was to have a proper dbms backend, rather than all the files that are there now.

awalk · 2006-01-23 10:01:18

test1000 wrote:

well, if it needed fuse I couldn't have chroot'ed my arch linux installation with my INSERT rescue-cd and downloaded packages which fixed my install after i unintentionally b0rked the install It's not KISS, and therefore i don't like it..

In the worst case, yes. However, having a FUSE interface to the db doesn't preclude the use of some other, more primitive interface. I mean, just because there's a FUSE interface (which was mainly aimed at custom scripts), doesn't mean the Pacman frontend would have to use it to install new (or replace "b0rked") packages. In fact, it would likely be preferable not to...

But, as you, and I have said, the complexity has certainly been increased beyond "KISS".

iphitus wrote:

Leaving sqlite.. oh lord this has been argued to death. Why oh why do we not just leave the current system and leave it up to judd. He isnt going to change it any sooner because we start another thread about it, he'll change it when he can and want to.

Yes, I'm aware there's been great debate (see first post), but I never saw any decent solutions. I've always felt the issue was never really concluded (http://bugs.archlinux.org/task/1151), so this was an attempt to spur some action (which I would be happy to be a part of), or bury it.

demonus · 2006-01-23 10:46:24

why not use berkeley db, it's installed by default anyway + the accompanying API is quite simple and well documented

jakob · 2006-01-23 12:02:01

hm i certainly have no idea of databases and such, and you may call my suggestion stupidity it it's best, but why not store the information in a simple txt-File?

cactus · 2006-01-23 17:11:55

In the past I was an ardent decrier of the sqlite camp, realizing that a db backend of some kind was needed, but thinking another solution would be best.

I have since changed my mind. I believe that sqlite would make a fine db backend for pacman.

awalk · 2006-01-23 23:56:46

hellwoofa wrote:

hm i certainly have no idea of databases and such, and you may call my suggestion stupidity it it's best, but why not store the information in a simple txt-File?

It's not a stupid suggestion: that is what is currently being done It's limited though, as the number of packages in the database has a significant impact on how long it takes to perform procedures like updates. This seems to be mainly due to the number of files that need to be opened and parsed.

Romashka · 2006-01-24 08:23:43

demonus wrote:

why not use berkeley db, it's installed by default anyway + the accompanying API is quite simple and well documented

Berkeley DB is not a good choice for large and constantly changing tatabase. SQLite is much better for this job. And because it is SQL engine it will do many low-level functions itself, leaving more high-level functions to Pacman.

Arch Linux

#1 2005-11-04 06:41:49

Pacman + DB & External scriptability

#2 2006-01-15 06:18:30

Re: Pacman + DB & External scriptability

#3 2006-01-15 06:26:20

Re: Pacman + DB & External scriptability

#4 2006-01-15 09:23:53

Re: Pacman + DB & External scriptability

#5 2006-01-22 16:43:10

Re: Pacman + DB & External scriptability

#6 2006-01-22 23:54:26

Re: Pacman + DB & External scriptability

#7 2006-01-23 09:46:14

Re: Pacman + DB & External scriptability

#8 2006-01-23 10:01:18

Re: Pacman + DB & External scriptability

#9 2006-01-23 10:46:24

Re: Pacman + DB & External scriptability

#10 2006-01-23 12:02:01

Re: Pacman + DB & External scriptability

#11 2006-01-23 17:11:55

Re: Pacman + DB & External scriptability

#12 2006-01-23 23:56:46

Re: Pacman + DB & External scriptability

#13 2006-01-24 08:23:43

Re: Pacman + DB & External scriptability

Board footer