You are not logged in.

#1 2013-01-13 21:57:43

nevempzub
Member
Registered: 2013-01-12
Posts: 1

Saving bandwidth, "the future of package management"

I just imagine an "optimized" world, where each small change in a source code results only a small change in a binary "object" (which I imagine far smaller building elements than nowaday average packages -- but this thread is not about that), and only the changelog (not the whole object) will be streamed to the users. Every small piece of change gets transmitted only once. New users first download a "snapshot" of the object, then a connection will remain between the author of the object and the users, streaming the changelog (keeping in sync, auto-updating). 1 byte change in source results in 1 byte of stream transfer.

Last edited by nevempzub (2013-01-13 21:59:22)

Offline

#2 2013-01-13 22:02:26

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 7,130

Re: Saving bandwidth, "the future of package management"

Sounds awful. Try to think of something more cheerful.


CLI Paste | How To Ask Questions

Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L

Offline

#3 2013-01-13 22:50:09

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Saving bandwidth, "the future of package management"

Binary deltas do something similar, also http://dev.chromium.org/developers/desi … -courgette

Offline

#4 2013-01-13 22:55:03

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 30,424
Website

Re: Saving bandwidth, "the future of package management"

Moving to GNU/Linux Discussion...


Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

Offline

#5 2013-01-14 01:34:53

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 7,130

Re: Saving bandwidth, "the future of package management"

karol wrote:

Binary deltas do something similar, also http://dev.chromium.org/developers/desi … -courgette

But that lacks the nightmarish feature of auto-updating your software for you while you are in the middle of giving a presentation during a job interview and therefore would not constitute the "optimised" scenario originally envisaged?


CLI Paste | How To Ask Questions

Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L

Offline

#6 2013-01-14 02:07:47

ngoonee
Forum Fellow
From: Between Thailand and Singapore
Registered: 2009-03-17
Posts: 7,354

Re: Saving bandwidth, "the future of package management"

cfr wrote:
karol wrote:

Binary deltas do something similar, also http://dev.chromium.org/developers/desi … -courgette

But that lacks the nightmarish feature of auto-updating your software for you while you are in the middle of giving a presentation during a job interview and therefore would not constitute the "optimised" scenario originally envisaged?

Please try not to troll anyone (even if the idea is half-baked and/or already accomplished in part).

OP - see the deltas mentioned earlier. Also consider the scalability of your approach, which seems to be directly connecting (clueless) users with upstream. There's a reason distros exist.


Allan-Volunteer on the (topic being discussed) mailn lists. You never get the people who matters attention on the forums.
jasonwryan-Installing Arch is a measure of your literacy. Maintaining Arch is a measure of your diligence. Contributing to Arch is a measure of your competence.
Griemak-Bleeding edge, not bleeding flat. Edge denotes falls will occur from time to time. Bring your own parachute.

Offline

#7 2013-01-14 02:27:54

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 7,130

Re: Saving bandwidth, "the future of package management"

ngoonee wrote:

Please try not to troll anyone (even if the idea is half-baked and/or already accomplished in part).

Sorry. I think it just reminded me so much of the way our machines are configured in work. You are about to start teaching a class but when you log in, the network insists on updating everything regardless of the fact that you need to actually use something now. And then it insists on rebooting. Or sometimes it waits until you actually launch the programme you need. Or sometimes it does both. Even if the update doesn't break something, the clock is ticking and an entire class is staring at you as if it was something you did. And there's nothing you can do - these are machines in classrooms and there's usually only 10 minutes change over time between classes (or no change over time for one department I work for).

The idea of automatic updates just hit a raw nerve.


CLI Paste | How To Ask Questions

Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L

Offline

#8 2013-01-14 02:59:34

ngoonee
Forum Fellow
From: Between Thailand and Singapore
Registered: 2009-03-17
Posts: 7,354

Re: Saving bandwidth, "the future of package management"

cfr wrote:
ngoonee wrote:

Please try not to troll anyone (even if the idea is half-baked and/or already accomplished in part).

Sorry. I think it just reminded me so much of the way our machines are configured in work. You are about to start teaching a class but when you log in, the network insists on updating everything regardless of the fact that you need to actually use something now. And then it insists on rebooting. Or sometimes it waits until you actually launch the programme you need. Or sometimes it does both. Even if the update doesn't break something, the clock is ticking and an entire class is staring at you as if it was something you did. And there's nothing you can do - these are machines in classrooms and there's usually only 10 minutes change over time between classes (or no change over time for one department I work for).

The idea of automatic updates just hit a raw nerve.

I understand as an educator forced to work with certain operating systems as well (hence why I just use my own laptop for the most part) but that's quite off-topic here, obviously smile.

Note also that 'automatic' updating isn't stated in the OP (which is to be fair quite sparse).


Allan-Volunteer on the (topic being discussed) mailn lists. You never get the people who matters attention on the forums.
jasonwryan-Installing Arch is a measure of your literacy. Maintaining Arch is a measure of your diligence. Contributing to Arch is a measure of your competence.
Griemak-Bleeding edge, not bleeding flat. Edge denotes falls will occur from time to time. Bring your own parachute.

Offline

#9 2013-01-14 03:29:25

cfr
Member
From: Cymru
Registered: 2011-11-27
Posts: 7,130

Re: Saving bandwidth, "the future of package management"

Maybe I misunderstood. I read "keeping in sync, auto-updating" to mean autoupdating the software. Do you think the OP just meant autoupdating the changelog?

[Also off-topic: I do the same with my laptop but sometimes my laptop has been unusable. And lately I have this randomly-shut-down-for-no-reason phenomenon. Which has led me to tell them they have to give me rooms away from the greatest danger zone, though now it has happened at home... but that's in another thread... Also, the OS updates are the least of it. It is mostly the network management updates which won't let you do anything else.]

EDIT: Or was the idea just that the available UPDATE would be automatically generated at the source?

Last edited by cfr (2013-01-14 03:35:32)


CLI Paste | How To Ask Questions

Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L

Offline

#10 2013-01-14 08:55:41

litemotiv
Forum Fellow
Registered: 2008-08-01
Posts: 5,026

Re: Saving bandwidth, "the future of package management"

cfr wrote:

Maybe I misunderstood. I read "keeping in sync, auto-updating" to mean autoupdating the software. Do you think the OP just meant autoupdating the changelog?

[Also off-topic: I do the same with my laptop but sometimes my laptop has been unusable. And lately I have this randomly-shut-down-for-no-reason phenomenon. Which has led me to tell them they have to give me rooms away from the greatest danger zone, though now it has happened at home... but that's in another thread... Also, the OS updates are the least of it. It is mostly the network management updates which won't let you do anything else.]

EDIT: Or was the idea just that the available UPDATE would be automatically generated at the source?

As far as i could tell the whole idea was about transferring binary changes to users instead of complete package updates which are unpacked and overwritten. So instead of downloading a 50MB kernel update with pacman, only the 100KB of code that was actually changed between revisions would be updated, through a GIT-like system.

The auto-update thing which you freaked out about seems like a trivial part of the OP's story, and would be simple to make optional.


ᶘ ᵒᴥᵒᶅ

Offline

#11 2013-01-14 09:27:25

ShionjiYuuko
Member
Registered: 2011-07-31
Posts: 67

Re: Saving bandwidth, "the future of package management"

Perhaps this needs a large scale compiler redesigning... smile

Most of package deltas are about 10% or more of the original package size. What if we apply deltas per package file, not on the whole package archive?


始まりの荒野を独り もう歩き出してるらしい、僕は灰になるまで僕で有り続けたい
http://about.me/nnhnkn | http://identi.ca/nnhzkn

Offline

#12 2013-01-14 09:31:52

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,365
Website

Re: Saving bandwidth, "the future of package management"

ShionjiYuuko wrote:

What if we apply deltas per package file, not on the whole package archive?

It would get worse.

Offline

#13 2013-01-14 12:39:07

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,442
Website

Re: Saving bandwidth, "the future of package management"

I don't know much (aka anything) about binary diffs, and I may well have used the wrong tool.  But I just installed xdelta to see how small a binary diff might be based on the assumption that a small code change may have substantial alignment-type changes on the resulting binary.  I took a program I've been working on, compiled it, commented out two lines which implement a minor feature, then recompiled.

Both versions are about the same size at 23K.  The xdelta produced patch is 7K.

So, in one sense, that is a 3 fold savings.  But this would require the patches to be made upstream.  It would also require that one has a perfect always-on internet connection: what if you miss an update?  If you miss an update, the subsequent diffs will be against a version you don't have, so they will fail.  The only solution to this problem is that online repositories contain the history of all such diffs.

For my example, if this repository had to contain just 3 sequential diffs, this would eliminate the benefit.  Also for any user who had not updated since the 3 changes previous, they'd still be downloading something the size of the original package but then they'd have the additional cost of needing to apply each binary diff.

So, using only my single test case as an example, one might have to download three 7K diffs, then apply each one, or they could just download the new 23K binary itself and be done with it.  What if there were 4 changes since their last update?  five? etc ...

This would make it absolute hell to try to update any rapidly developing project - especially when it's pitched as grabbing every trivial change a developer makes rather than being based on versioned releases.  I make changes to my code daily.  Do you want a daily diff of probably more than that 7K example above, or would you prefer to just update every week or so with the newest 23K version of the program?

Last edited by Trilby (2013-01-14 12:39:35)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#14 2013-01-24 03:18:40

andjeng
Member
From: Indonesia
Registered: 2012-08-30
Posts: 148

Re: Saving bandwidth, "the future of package management"

what if just update your system once a year or so.
i think it will save alot of bandwidth.
assuming a software updated twice or thrice a year.


just looking around. wink

Offline

#15 2013-01-24 05:01:47

Trent
Member
From: Baltimore, MD (US)
Registered: 2009-04-16
Posts: 990

Re: Saving bandwidth, "the future of package management"

cd /usr; git init

big_smile

Okay, it would be a bit more complex with minimal binary diffs and the ability to selectively pull only the packages you wanted. Still...

Last edited by Trent (2013-01-24 05:02:14)

Offline

#16 2013-01-24 06:53:41

Janarto
Member
From: Paris
Registered: 2008-09-23
Posts: 80

Re: Saving bandwidth, "the future of package management"

I think the best way to improve package managing from a technical standpoint would be to use a bittorrent/aria2 kind of decentralised base for software distribution, but that brings a lot of hurdles in terms of security and privacy.

I'd like to see something that would allow for software diffusion on mesh networks for example, where you can't always point to specific servers

Offline

#17 2013-01-24 14:39:49

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Saving bandwidth, "the future of package management"

andjeng wrote:

what if just update your system once a year or so.

And in the meantime use software that might have security issues or bugs ;P

Offline

Board footer

Powered by FluxBB