why do we need packages in pacman? why not download files one by one?

cypa · 2023-12-09 11:41:21

Hi, what about alternative software distribution?
now every server/device totally online then why do we need packages? why don't download the very files individually?
packages are basically archives of files which is useful for dial-up download but now we are always online ...

schard · 2023-12-09 12:34:46

A package is a collection of files that represents a cohesive unit of software.
Software is not just single files. It's more often that not multiple files belonging together.
Also such software packages may depend on other packages, so that's something to take into account, too.
So, no, a package is not just an archive of files. The archive is just a convenient way to keep all files of that package together.
The package is more about the metadata of what those collections of files represent and their relation to other such collections.

mpan · 2023-12-09 13:02:38

This is a thing decided by Arch developers and pacman developers. So the question should go to them. Unless one of them happen to come across your question (low chances), asking this on a public forum is likely to give you a mix of non-authoritative guesses and unproductive arguing between people having their strong opinions on how things should be done.

For actual, productive proposals the right route is filing a PR through the means mentioned on either of the pages I mentioned above.

But mind that your suggestion:

Requires an enormous effort: redesigning the entire release process of a dozen different and independent projects, and verifying/changing the workflow/configuration of hundreds of elements.⁽¹⁾ You did not mention any benefits outweighting that gargantuan cost.
Does not mention, how making hundreds to thousands requests per package is better than making one. Please explain, how do you go around P(file|package) = 1.
Does not explain, how do you want to deal with signatures, when the signature size (relative to content) suddenly becomes big.
Does not explain, how do you deal with decreased compression ratios.
Does not explain, how do you deal with increased load on the users’ file system.
Does not discuss privacy concerns, given that this is increasing metadata leaks through TLS by three orders of magnitude.

____
⁽¹⁾ Did you thought about e.g. 2nd tier mirrors, which must ensure not facing issues with i-node counts, fragmentation and btrfs metadata size.

Allan · 2023-12-09 13:17:17

mpan wrote:

This is a thing decided by Arch developers and pacman developers. So the question should go to them. Unless one of them happen to come across your question (low chances)

Higher than you think.

And the answer is because there is near-zero conceivable advantage of downloading packages one file at at time.

I say near zero, because there are files that do not change (apart from file time metadata) between package updates and that could be used to reduce download sizes. But in a rolling release distribution such as Arch, the savings are much less than you would think.

twelveeighty · 2023-12-09 15:21:31

cypa wrote:

why don't download the very files individually

You left out the most important part of that argument: your motivation. Why do you think that is "better"? What would be your counter to Allen's objection that there is "near-zero conceivable advantage of downloading [...] one file at at time"?

cypa · 2023-12-09 16:04:33

schard wrote:

The archive is just a convenient way to keep all files of that package together.

indeed it was many years ago back then we had dial-up downloads ... but now we are always online, servers are and terminal devices almost always online - so why do we need archive of files and not download files one by one?
leave aside dependencies, requeriments and other metadata which do not require archiving files actually

cypa · 2023-12-09 16:31:52

mpan wrote:

This is a thing decided by Arch developers and pacman developers. So the question should go to them. Unless one of them happen to come across your question (low chances), asking this on a public forum is likely to give you a mix of non-authoritative guesses and unproductive arguing between people having their strong opinions on how things should be done.
For actual, productive proposals the right route is filing a PR through the means mentioned on either of the pages I mentioned above.
But mind that your suggestion:
Requires an enormous effort: redesigning the entire release process of a dozen different and independent projects, and verifying/changing the workflow/configuration of hundreds of elements.⁽¹⁾ You did not mention any benefits outweighting that gargantuan cost.
Does not mention, how making hundreds to thousands requests per package is better than making one. Please explain, how do you go around P(file|package) = 1.
Does not explain, how do you want to deal with signatures, when the signature size (relative to content) suddenly becomes big.
Does not explain, how do you deal with decreased compression ratios.
Does not explain, how do you deal with increased load on the users’ file system.
Does not discuss privacy concerns, given that this is increasing metadata leaks through TLS by three orders of magnitude.
____
⁽¹⁾ Did you thought about e.g. 2nd tier mirrors, which must ensure not facing issues with i-node counts, fragmentation and btrfs metadata size.

thank you for the thorough analysis, it's an idea and I'd like to discuss it before it would be ready to PR

I think that

the effort would be fruitfull if we find a better simpler way to replace package archives with bunch of files in subdirectory tree
what exactly do you mean P(file|package) = 1? modern OS could transfer many files flawless, not sure if it is a problem with ftp, anyway one can pack temporary archives to transfer files over network where needed
not sure about signature but is it possible to sign a few files with single signature?
afaik transparent compression options are available also, assuming installed files do not need compression
I think user filesystem load would be lower since it's (software files) vs (package files + software files) on user filesystem
I'm not aware of metadata leaks through TLS, shall explore this

waiting for your comments ...

Last edited by cypa (2023-12-09 16:33:24)

cypa · 2023-12-09 16:38:10

Allan wrote:

And the answer is because there is near-zero conceivable advantage of downloading packages one file at at time.
I say near zero, because there are files that do not change (apart from file time metadata) between package updates and that could be used to reduce download sizes. But in a rolling release distribution such as Arch, the savings are much less than you would think.

most problems with archlinux on my laptop and servers were many sorts of pacman and packages fails, so I guess we should eliminate the packaging system complications to live better,
and haven't you seen any problems with packages?

cypa · 2023-12-09 16:45:18

twelveeighty wrote:

You left out the most important part of that argument: your motivation. Why do you think that is "better"? What would be your counter to Allen's objection that there is "near-zero conceivable advantage of downloading [...] one file at at time"?

KISS principle - the package archives is another entity which could be eliminated due to communications technology progress, as we are always online now

I guess the dependency database and processing could be simplified with plain file sets instead of package archives
and possibly it would be another step to unified linux software packaging ...

Trilby · 2023-12-09 16:46:01

Archiving has nothing to do with network availability or bandwidth. I have no idea why you are (trying to) make that connection. Why would individual files be harder to download than archived groups of files on dial up systems*? That's just a complete non-sequitur, and then so to is the converse that there's no need for archives of multiple files with broadband connections. Dial-up connections were in no way the reason for packages ... not at all.

Now you may be conflating package archives and compression methods. But these are different things. Archives can be uncompressed, individual files could be compressed. We just happen to use compressed archives.

If you want to argue that compression is no longer needed with modern network connections you'd have some ground to stand on, but still really not much. As (de)compression algorithms have improved we've come to a point where in many cases it's still most efficient overall to compress data transferred over networks.

* in fact, while it has it's own problems, the opposite argument seems more logical: in less reliable network conditions, sending individual files as needed rather than full packages would be better.

Last edited by Trilby (2023-12-09 16:47:53)

Scimmia · 2023-12-09 16:46:45

cypa wrote:

Allan wrote:
And the answer is because there is near-zero conceivable advantage of downloading packages one file at at time.
I say near zero, because there are files that do not change (apart from file time metadata) between package updates and that could be used to reduce download sizes. But in a rolling release distribution such as Arch, the savings are much less than you would think.
most problems with archlinux on my laptop and servers were many sorts of pacman and packages fails, so I guess we should eliminate the packaging system complications to live better,
and haven't you seen any problems with packages?

No, I haven't seen any problems this would fix at all. The only advantage I can see would be smaller data transfer, and that's not anywhere near worth the enormous efforts and resources this would require.

If you see some other advantage, you need to make that case.

Last edited by Scimmia (2023-12-09 16:48:01)

cypa · 2023-12-09 16:57:27

Trilby wrote:

Archiving has nothing to do with network availability or bandwidth. I have no idea why you are (trying to) make that connection. Why would individual files be harder to download than archived groups of files on dial up systems*? That's just a complete non-sequitur, and then so to is the converse that there's no need for archives of multiple files with broadband connections. Dial-up connections were in no way the reason for packages ... not at all.
Now you may be conflating package archives and compression methods. But these are different things. Archives can be uncompressed, individual files could be compressed. We just happen to use compressed archives.
If you want to argue that compression is no longer needed with modern network connections you'd have some ground to stand on, but still really not much. As (de)compression algorithms have improved we've come to a point where in many cases it's still most efficient overall to compress data transferred over networks.
* in fact, while it has it's own problems, the opposite argument seems more logical: in less reliable network conditions, sending individual files as needed rather than full packages would be better.

I'm a bit confused if you answered my comment or other,

but I strongly disagree, dial-up connection is transient then syncing integrity of bunch of files is complicated while single archive file download is much easier, moreover in the dial-up era most hardware and OSes were weak and single-file archive reduces resources consumed, and last thing is storing for self and distributing the packages to friends by handheld disks and flash more convenient with single-file archives,

on compression, I do not deny any compression option you like

Last edited by cypa (2023-12-09 17:05:12)

cypa · 2023-12-09 17:03:27

No, I haven't seen any problems this would fix at all. The only advantage I can see would be smaller data transfer, and that's not anywhere near worth the enormous efforts and resources this would require.

well, if you agree it's smaller data transfer then you should also agree it's less files on user filesystem, and less install/update operations, shouldn't you? then it's a fewer scripts with less errors in total, or not?

Scimmia · 2023-12-09 17:05:49

No, it's not less files on the user filesystem. The potential smaller downloads would come from not redownloading files that haven't changed. A package is the files needed for the program, they will be there no matter what. You would not have less install/update operations, as a program update is a program update.

cypa · 2023-12-09 17:10:07

plus another thing, very easy add local user-defined packages to the database - list full path of your files as new package, that's it!

it's +9000 to package operation simplicity

Last edited by cypa (2023-12-09 18:32:52)

Trilby · 2023-12-09 17:10:39

Perhaps I misunderstood. It initially seemed you were referring to upgrading individual files while not having packages at all. But now you seem to be alluding to a system where there are packages at least conceptually, but the contents of a given package could be downloaded one file at a time.

I could see how there could be some potential benefits of the latter approach: i.e., when only one file in a package changes, users could download just that file rather than the full package. But if this was the logic, "file" is still not really the right level of abstraction and package deltas or diffs would be: why download the full new file if only one line in it changed, just download the diff / patch.

In any case, the unit of download didn't seem to be your initial topic as you referred vaguely to issues you've faced do to packages / packaging. And these issues certainly weren't due to the fact that package data was downloaded as a single unit, but rather was almost certainly due to version mismatches, dependency issues, or the like which are all due to the package concept not the downloaded unit (and so changing the downloadable unit while maintaining a package concept would not address even the one vague benefit you've pointed to).

EDIT: no, nevermind, you are now further conflating the conceptual unit of a package with the downloadable unit of the archive. You seem to be conflating these deliberately to dodge questions / critiques. So either you are being deliberately disingenuous or you are not even thinking through your own point. Either way, there's no value in trying to understand someone who either A) doesn't want to be understood, or B) doesn't even understand themselves.

Last edited by Trilby (2023-12-09 17:13:40)

cypa · 2023-12-09 17:22:24

Scimmia wrote:

No, it's not less files on the user filesystem. The potential smaller downloads would come from not redownloading files that haven't changed. A package is the files needed for the program, they will be there no matter what. You would not have less install/update operations, as a program update is a program update.

wait, have you forgot the package archive files itself and temporary files/directories needed to unpack the archives?
is unpacking an additional operation itself?

for example, take the core/linux package with the kernel - are you aware what pacman update scripts do?
- remove vmlinuz and initramfs boot files
- do some processing of update packages (and all the processing time your system is not bootable at all - if power fails then you could not boot OS!)
- then place new files vmlinuz and initramfs

but if you do not need process whole package archives then it's easy to use replace file atomic operation for boot files ... isn't it better?

cypa · 2023-12-09 17:23:54

Trilby wrote:

Perhaps I misunderstood. It initially seemed you were referring to upgrading individual files while not having packages at all. But now you seem to be alluding to a system where there are packages at least conceptually, but the contents of a given package could be downloaded one file at a time.

indeed, we totally need package metadata with dependencies requirements etc.

cypa · 2023-12-09 17:29:40

Trilby wrote:

I could see how there could be some potential benefits of the latter approach: i.e., when only one file in a package changes, users could download just that file rather than the full package. But if this was the logic, "file" is still not really the right level of abstraction and package deltas or diffs would be: why download the full new file if only one line in it changed, just download the diff / patch.

files diff download is another interesting thing, but I'd like to detach it to avoid complication

cypa · 2023-12-09 17:36:31

Trilby wrote:

In any case, the unit of download didn't seem to be your initial topic as you referred vaguely to issues you've faced do to packages / packaging. And these issues certainly weren't due to the fact that package data was downloaded as a single unit, but rather was almost certainly due to version mismatches, dependency issues, or the like which are all due to the package concept not the downloaded unit (and so changing the downloadable unit while maintaining a package concept would not address even the one vague benefit you've pointed to).

look, in general it's package system development question and I see ways for better dependency resolution on plain files instead of two levels of dependencies (one for packages and one for files),
e.g. if another package update consists of one single file would it be easier to resolve? I'm not sure here ...

cypa · 2023-12-09 17:39:27

Trilby wrote:

EDIT: no, nevermind, you are now further conflating the conceptual unit of a package with the downloadable unit of the archive. You seem to be conflating these deliberately to dodge questions / critiques. So either you are being deliberately disingenuous or you are not even thinking through your own point. Either way, there's no value in trying to understand someone who either A) doesn't want to be understood, or B) doesn't even understand themselves.

sorry, my english is not so good, mayby I didn't express my thoughts well,
but feel free to ask more detailed questions

thank you for discussion anyway

seth · 2023-12-09 17:43:32

most problems with archlinux on my laptop and servers were many sorts of pacman and packages fails

This thread is a trainwreck of an https://en.wikipedia.org/wiki/XY_problem

What kind of problem do you actually have there? Maybe focus on that instead…

cypa · 2023-12-09 18:07:16

seth wrote:

most problems with archlinux on my laptop and servers were many sorts of pacman and packages fails
This thread is a trainwreck of an https://en.wikipedia.org/wiki/XY_problem
What kind of problem do you actually have there? Maybe focus on that instead…

well it's now one single problem ... it general experience and my with to optimize things on my computers, my curiosity ...

mpan · 2023-12-09 18:38:42

cypa wrote:

waiting for your comments ...

My intention was to point you to the right recipients of the suggestion and avoid creating a topic-going-nowhere here. And the list are the things to consider before sending any request to the developers. I hope we agree, that it would be bad idea to ask for their attention, if the idea has many unaddressed issues.

Since it’s clear the problem is elsewhere, I don’t think it’s worth using that route. Redesigning the entire ecosystem of multiple projects is not a proper response to a single person experiencing some specific problem.

Your problem is “most problems with archlinux on my laptop and servers were many sorts of pacman and packages fails”. Please provide the description of what has happened. With exact errors. These can be addressed and fixed. I find it unlikel that a single issue you encountered was caused by the current distribution model.

seth · 2023-12-09 20:24:46

it general experience and my with to optimize things on my computers, my curiosity

So you found some solution and are now looking for a problem it might solve?

What are you optimizing here? You're making a point that we're no longer using dial up connections, therefore don't need tarballs.
Leaving aside whether that makes any sense, you're obviously not concerned w/ traffic.
The packages collect "stuff" that is required for "feature" and sometimes do "install action" to make it work.
Are you suggesting it would be better if every user had to figure every single file they need to make gnome run - and where to put it?

I seriously don't understand what you're envisioning here or what problem you're trying to solve or what is being optimized how.

Arch Linux

#1 2023-12-09 11:41:21

why do we need packages in pacman? why not download files one by one?

#2 2023-12-09 12:34:46

Re: why do we need packages in pacman? why not download files one by one?

#3 2023-12-09 13:02:38

Re: why do we need packages in pacman? why not download files one by one?

#4 2023-12-09 13:17:17

Re: why do we need packages in pacman? why not download files one by one?

#5 2023-12-09 15:21:31

Re: why do we need packages in pacman? why not download files one by one?

#6 2023-12-09 16:04:33

Re: why do we need packages in pacman? why not download files one by one?

#7 2023-12-09 16:31:52

Re: why do we need packages in pacman? why not download files one by one?

#8 2023-12-09 16:38:10

Re: why do we need packages in pacman? why not download files one by one?

#9 2023-12-09 16:45:18

Re: why do we need packages in pacman? why not download files one by one?

#10 2023-12-09 16:46:01

Re: why do we need packages in pacman? why not download files one by one?

#11 2023-12-09 16:46:45

Re: why do we need packages in pacman? why not download files one by one?

#12 2023-12-09 16:57:27

Re: why do we need packages in pacman? why not download files one by one?

#13 2023-12-09 17:03:27

Re: why do we need packages in pacman? why not download files one by one?

#14 2023-12-09 17:05:49

Re: why do we need packages in pacman? why not download files one by one?

#15 2023-12-09 17:10:07

Re: why do we need packages in pacman? why not download files one by one?

#16 2023-12-09 17:10:39

Re: why do we need packages in pacman? why not download files one by one?

#17 2023-12-09 17:22:24

Re: why do we need packages in pacman? why not download files one by one?

#18 2023-12-09 17:23:54

Re: why do we need packages in pacman? why not download files one by one?

#19 2023-12-09 17:29:40

Re: why do we need packages in pacman? why not download files one by one?

#20 2023-12-09 17:36:31

Re: why do we need packages in pacman? why not download files one by one?

#21 2023-12-09 17:39:27

Re: why do we need packages in pacman? why not download files one by one?

#22 2023-12-09 17:43:32

Re: why do we need packages in pacman? why not download files one by one?

#23 2023-12-09 18:07:16

Re: why do we need packages in pacman? why not download files one by one?

#24 2023-12-09 18:38:42

Re: why do we need packages in pacman? why not download files one by one?

#25 2023-12-09 20:24:46

Re: why do we need packages in pacman? why not download files one by one?

Board footer