You are not logged in.

#101 2005-03-30 16:23:34

phrakture
Arch Overlord
From: behind you
Registered: 2003-10-29
Posts: 7,879
Website

Re: Pacman with db support?

berkus wrote:

(About the "unmaintained" question someone raised here - cdb is mature enough and not being updated for a long time means it didn't take the "creeping featuritis" route.)

and if anything, it's very small - I'm sure some here could fork it if pacman were to require it and the project died....(*looks at i3839*, heh)

I'll vote for cdb... judd, you listening?  lol

Offline

#102 2005-03-30 18:14:30

c0ldevil
Member
Registered: 2004-10-10
Posts: 22

Re: Pacman with db support?

I just raised the "unmaintained" question in order to clear up the stability and maturity of the db's presented here.

I go for simplicity, we should choose a db that fits perfectly our needs, no more no less smile That way it's easier to maintain the whole thing.

Offline

#103 2005-03-30 19:45:41

i3839
Member
Registered: 2004-02-04
Posts: 1,185

Re: Pacman with db support?

I think any big changes to Pacman's code should be done in conjunction with Aurelien Foret's work on the libified Pacman. Perhaps instead of splitsing Pacman in two parts (frontend and the lib), it could be modularized into multiple parts where each part is easily replaced with something else, if needed. To begin with into three parts: The frontend, the core pacman and the db part. Downloading could also be a seperate module, but with the XferCommand option that isn't really needed.

cdb looks interesting, but it's read only, and we'll probably want a db of all installed packages too which requires writing often to it, so not sure.

As a sidenote, using a simple filesystem based db may be fast enough if its usage is optimized to avoid unnecessary actions.

Offline

#104 2005-03-30 19:52:01

cactus
Taco Eater
From: t͈̫̹ͨa͖͕͎̱͈ͨ͆ć̥̖̝o̫̫̼s͈̭̱̞͍̃!̰
Registered: 2004-05-25
Posts: 4,622
Website

Re: Pacman with db support?

or choose none at all.
wink


"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍

Offline

#105 2005-03-30 19:55:40

phrakture
Arch Overlord
From: behind you
Registered: 2003-10-29
Posts: 7,879
Website

Re: Pacman with db support?

i3839 wrote:

I think any big changes to Pacman's code should be done in conjunction with Aurelien Foret's work on the libified Pacman. Perhaps instead of splitsing Pacman in two parts (frontend and the lib), it could be modularized into multiple parts where each part is easily replaced with something else, if needed. To begin with into three parts: The frontend, the core pacman and the db part. Downloading could also be a seperate module, but with the XferCommand option that isn't really needed.

hmmm, I think of it more like this: if the exported library functions are done well enough, database changes to the backend won't matter... there shouldn't be library functions to "get_record_from_db" but instead "get_package_info"... in this way the way the data is stored can change at will and not disrupt front ends..

i3839 wrote:

cdb looks interesting, but it's read only, and we'll probably want a db of all installed packages too which requires writing often to it, so not sure.

it's read only? i guess I missed that... how does one generate a cdb hash then? is there a tool?
actually being read only, it may work to an advantage.... gensync could be modified to generate a cdb database, and that could be pulled by pacman and left as is.... an "-Sy" operation would be a simple download then... (well maybe with some md5 summing and whatnot)

Offline

#106 2005-03-30 20:12:15

i3839
Member
Registered: 2004-02-04
Posts: 1,185

Re: Pacman with db support?

cactus wrote:

or choose none at all.
wink

That's my preference, I'd like a system where the Pacman just queries the server for package info. (maybe I'll even implement it one day..) To me it looks rather silly to download the whole db each time when you don't use most of it.

Though in that case you'll still need a local db of all installed packages (no matter how that db looks like, can be just a list).

To clear up possible confusions: with modules I mean source code modules, that don't have to be seperate libraries.

I didn't look into the cdb source documentation yet, so perhaps it's possible to add data to an existing db without making a new one, but the first impression is that that isn't possible. There is a tool to make a db for cdb yes, as wel as some other tools.

Offline

#107 2005-03-30 20:24:14

phrakture
Arch Overlord
From: behind you
Registered: 2003-10-29
Posts: 7,879
Website

Re: Pacman with db support?

i3839 wrote:
cactus wrote:

or choose none at all.
wink

That's my preference, I'd like a system where the Pacman just queries the server for package info. (maybe I'll even implement it one day..) To me it looks rather silly to download the whole db each time when you don't use most of it.

Though in that case you'll still need a local db of all installed packages (no matter how that db looks like, can be just a list).

hmmm, interesting, but I have a feeling some operations would be rather slow... like searching... that may be rough... though besides that stuff, I like the idea of just a list of installed packages and basically doing a "ls" on the server to check for updates.... info is a different story...

Offline

#108 2005-03-30 20:29:42

berkus
Member
From: Tallinn, Estonia
Registered: 2005-03-29
Posts: 65
Website

Re: Pacman with db support?

i3839 wrote:

cdb looks interesting, but it's read only, and we'll probably want a db of all installed packages too which requires writing often to it, so not sure.

As stated in answer to question number two, cdb will be updated only by sync operation (either while syncing to remote repo or rescanning local files for changes).


duh, i have a dejavu... i think i wrote the same text in gentoo forum a while ago smile


keep in touch.

Offline

#109 2005-03-30 21:04:21

i3839
Member
Registered: 2004-02-04
Posts: 1,185

Re: Pacman with db support?

berkus wrote:

As stated in answer to question number two, cdb will be updated only by sync operation (either while syncing to remote repo or rescanning local files for changes).

It depends on how fast it is to generate a new db when adding only a bit data to it. If it's fast enough, then it's virtually read/write, but if it's too slow then it's practically read only. Testing should find out.

As for the no db, query based approach: Searching won't be slow because the server will do it (with all data in cache) and return the result (with a max on the data, e.g. first 50 results or max 1K of data, with an option to get everything if you really want). Compare it to DNS.

The most complex operation would be to sync, then the app will ask the server all new packages since date X, which can be quite a lot if it was long ago. The other approach would be to send a list of packages and ask for which ones there isan update. Or, horrors of horrors, keep a list of installed packages on the server so the server knows what you need (though some people may dislike that because it pokes at their privacy, even if you can't see from the list from who it is, not to mention that it doesn't contain any sensitive data in the first place, no IP's are logged or something).

Anyway, if Pacman is modular enough it should be very easy to add different backends to see what is best or most liked.

I'll contact Aurelien Foret when I have more time later this or next week, to see how far he is and if he wants any help. Anyone happen to know his forum/irc nickname? Otherwise it'll be email.

Offline

#110 2005-04-15 07:28:28

nuopus
Member
From: Mesa, AZ
Registered: 2005-03-09
Posts: 60

Re: Pacman with db support?

miqorz wrote:

I think it's a good IDEA that we should keep in the back of our minds but there's really no need to adopt this idea NOW or any time soon.

Pacman is way faster than most solutions out there (portage, anyone?) and using SQ Lite for this solution with threats of corrupt databases just seems very pre-mature.

And definately faster than that peace of crap yum that fedora uses.

Offline

#111 2005-04-15 09:19:46

sweiss
Member
Registered: 2004-02-16
Posts: 635

Re: Pacman with db support?

i3839 wrote:
berkus wrote:

As stated in answer to question number two, cdb will be updated only by sync operation (either while syncing to remote repo or rescanning local files for changes).

It depends on how fast it is to generate a new db when adding only a bit data to it. If it's fast enough, then it's virtually read/write, but if it's too slow then it's practically read only. Testing should find out.

As for the no db, query based approach: Searching won't be slow because the server will do it (with all data in cache) and return the result (with a max on the data, e.g. first 50 results or max 1K of data, with an option to get everything if you really want). Compare it to DNS.

The most complex operation would be to sync, then the app will ask the server all new packages since date X, which can be quite a lot if it was long ago. The other approach would be to send a list of packages and ask for which ones there isan update. Or, horrors of horrors, keep a list of installed packages on the server so the server knows what you need (though some people may dislike that because it pokes at their privacy, even if you can't see from the list from who it is, not to mention that it doesn't contain any sensitive data in the first place, no IP's are logged or something).

Anyway, if Pacman is modular enough it should be very easy to add different backends to see what is best or most liked.

I'll contact Aurelien Foret when I have more time later this or next week, to see how far he is and if he wants any help. Anyone happen to know his forum/irc nickname? Otherwise it'll be email.

I also had a similar idea in mind. Are there any drawbacks to accessing the servers directly?

Offline

#112 2005-04-15 11:21:06

i3839
Member
Registered: 2004-02-04
Posts: 1,185

Re: Pacman with db support?

sweiss wrote:

I also had a similar idea in mind. Are there any drawbacks to accessing the servers directly?

The package db server can be different than the actual file server.

Perhaps the main drawback is that you need such db server, though it could use http if people dislike a more elegant special purpose protocols. Another drawback is that it is slightly harder for people to have a public repo with this system, as they either need to run their own db server or be known by the default one (which is trivial, but still). But now AUR is here this problem is smaller because the intent is that all packages are more centralized anyway.

Most actions can be done with just FTP, like finding new packages (-Sy), but there will be always at least the search action which must be done by some sort of package db server if the user has no local pkg db (other than the installed packages of course). Though the number of packages and their description doesn't change that often, so perhaps that can be kept on the client.

(I totally forgot about this whole thing, slightly more urgent things are keeping me busy now. I'll probably hack something up this summer..)

Offline

#113 2005-04-19 14:49:51

i3839
Member
Registered: 2004-02-04
Posts: 1,185

Re: Pacman with db support?

turbidostato wrote:

Then you would loose access as fast as your network does.

Not only that, but you would pay for big latencies whenever you need to ask for any info to the server, not to talk about comptention on such servers (soustaining a lot of queries is not only a matter of having enough bandwidth).

Things you can do with pacman can be split up into three groups:

1) Local things, like -Qx and -R.

It is clear that this case won't change at all.


2) Things which need internet, like -Sy, -Su, and most other -S.

My way -Sy won't be needed anymore, so that saves at least 250 KB for base + extra syncing (that much?? Something strange happened to the db recently or something?). The list of new packages of the last month is 5K uncompressed. That list gives enough info to know which packages must be updated, and then only the package info of the packages you need to upgrade are needed, which isn't that much info.

With the query system the worst case is that you download all package info, which can only happen if you have all packages installed and all need upgrading. The current system downloads all info all the time. To make it worse, if you sync more often than strictly needed then the bandwidth waste is even worse.

As this group is the main purpose of Pacman, installing and updating packages, I think optimizing this usage is important. Also now it may be workable, but the current system isn't scalable, while the query system is. Obviously you need internet anyway for this actions.


3) Searching packages with -Ss.

This is the only thing that becomes harder with the query system. There are multiple possible ways to implement searching, to name a few:

a) Let the server do it.

Pro: Saves bandwidth as not all package info has to be sent to the client nor kept up to date. Probably easiest to implement.

Con: Need an internet connection for just searching. If the connection is very bad it will be slower too (if the server is busy enough then all files will be in memory and the searching will be much faster than locally, depending on the db). Puts an additional load on the server.

b) Let the client do it. The client will need a list of al package info then. This list won't be updated that often as packages don't change often. Mostly info of new packages will be added.

Pro: No internet needed, easier for the server.

Con: must keep a db of package info synced, requiring extra bandwidth and it may complicate things a bit.

Also 2) and 3b) can be implemented by just using FTP, without need for a special server, so that's probably the thing I'll try first.

Offline

#114 2005-05-25 23:02:05

berkus
Member
From: Tallinn, Estonia
Registered: 2005-03-29
Posts: 65
Website

Re: Pacman with db support?

Heh look what i found

http://www.ohse.de/uwe/dyndb.html


keep in touch.

Offline

#115 2005-05-25 23:07:27

phrakture
Arch Overlord
From: behind you
Registered: 2003-10-29
Posts: 7,879
Website

Re: Pacman with db support?

berkus wrote:

interesting... might be worth a look... but it's also way dead:
0.5.2 - 2000-08-21

Offline

#116 2005-05-26 00:08:33

berkus
Member
From: Tallinn, Estonia
Registered: 2005-03-29
Posts: 65
Website

Re: Pacman with db support?

phrakture wrote:
berkus wrote:

interesting... might be worth a look... but it's also way dead:
0.5.2 - 2000-08-21

I believe it have just reached its maturity. You know, there's nothing else left to remove.


keep in touch.

Offline

#117 2005-05-26 00:14:35

miqorz
Member
Registered: 2004-12-31
Posts: 475

Re: Pacman with db support?

Hardly.

The beginning and Ending dates kind of denies that theory.


http://wiki2.archlinux.org/

Read it. Love it. Live it. Or die.

Offline

#118 2005-05-26 00:57:04

berkus
Member
From: Tallinn, Estonia
Registered: 2005-03-29
Posts: 65
Website

Re: Pacman with db support?

miqorz wrote:

Hardly.

The beginning and Ending dates kind of denies that theory.

Mind you, it is not PostgreSQL and therefore does not need 10 years to reach acceptable stability level, does it?


keep in touch.

Offline

#119 2005-05-26 03:16:08

miqorz
Member
Registered: 2004-12-31
Posts: 475

Re: Pacman with db support?

No. But a few months 5 years ago isn't very promising at all.


http://wiki2.archlinux.org/

Read it. Love it. Live it. Or die.

Offline

#120 2005-05-26 12:39:52

berkus
Member
From: Tallinn, Estonia
Registered: 2005-03-29
Posts: 65
Website

Re: Pacman with db support?

miqorz wrote:

No. But a few months 5 years ago isn't very promising at all.

I'm not insisting.


keep in touch.

Offline

Board footer

Powered by FluxBB