You are not logged in.

#1 2014-03-06 11:12:55

matthiaskrgr
Member
Registered: 2012-04-03
Posts: 28
Website

[SOLVED] official delta mirror?

Hi, to make it short, I'd like to have an official delta mirror.

My bandwidth has been limited to mostly ~8kb/s in the last months and I have still been able to keep my arch system up to date with deltas. smile

(How this works: basically you only download the differences between your version and the up-to-date version of the package ("libarchive-3.1.2-4_to_3.1.2-6-x86_64.delta)". Before installing, the libarchive-3.1.2-6-x86_64 package is then recreated from the "patch" and everything behaves as if you downloaded the original one.)

So far I have been using delta.archlinux.fr
Unfortunately, it seems to be the only delta mirror and provides no signatures for the package lists (because they are different from "original" arch ones and contain information about where to get deltas I guess?) which is kinda not-cool. If there was an official delta mirror, that might solve the signature problems.
Edit: I misunderstood here, apparently this is not about signatures of repo DBs

Why should you care about deltas?

The delta packages can be a lot smaller than the "regular" packages I have made a list of the packages updated since beginning of this month:

 name              delta      regular  saved:
glibc             202.8 K      8.3 M    97%
device-mapper                165.9 K
cryptsetup                   192.7 K
feh                          170.2 K
lib32-glibc                    2.6 M
libgphoto2        708.2 K      1.1 M    37%
libtirpc                     133.0 K
readline                     281.1 K
lvm2                         767.4 K
ppp                          279.7 K
python2-lxml                 657.0 K
raptor                       256.9 K
rasqal                       243.8 K
virtuoso-base       1.8 M      2.9 M   38%
gcr               329   K    777.9 K   57%
ldoc                          88.3 K
libpulse                     333.3 k
webkitgtk           7.7 M     20.8 M   37%
webkitgtk2                     8   M
inxi                          97.9 K
blender            17.4 M     39.5 M   56%
systemd            66.8 K      3   M   97%
chromium           17.2 M     30.6 M   43%
libunique                     26.6 K
xterm                        266.1 K
lib32-systemd                154   K
smpeg                         96.4 K
xbmc               570.2 K    21.7 M   97%
parted                       378.7 K
gnutls             362   K     1.9 M   81%
calligra-filters               3   M
calligra-libs                  2.9 M
calligra-plugins               2.9 M
calligra-words     234.9 K   956.7 K   75%
calligra-flow      460.7 K     5.9 M   92%
libburn                      245.9 K
libisofs                     197.1 K



packages with deltas only:
downloaded: 47 MB      actual (regular) size: 136 MB
=> 65  % saved, 89MB saved

all packages:
downloaded: 71 MB      actual (regular) size: 166 MB  
=> 57% saved, 95 MB saved

There might rarely be delta version of big binary packages (I have not yet seen any kernel delta packages for example), but if a package is just rebuild for a .so-bump and gets its pkgrel bumped (1.0-1 -> 1.0-2) I often can save quite some traffic with delta support. I also save a lot for packages that contain a lot of graphics (games for example) and only have their binaries changed, a few graphics.



There is also still a lot of room for optimization!
Yesterday I proposed a patch that increases xdelta3s compression level (so we might get smaller deltas with next pacman major release, if the patch is merged).

Also, bsdiff is able to create much smaller deltas than xdelta3 (but also needs more resources at generation).

Comparing /usr/bin/blender of the 2 latest blender releases:

xdelta3     10.4M # default
xdelta3 -9   9.9M # highest compression level
bsdiff       4.7M 

Edit: size of the binary: 47MB

However, while xdelta3 is clever enough to look into archives and compress against the differences of the files contained INSIDE the archive, bsdiff would just generate a delta BETWEEN the archives (does not look at the content), so an "xdelta3 using the bsdiff algorithm" still has to be made (by someone) smile

"But decompression of the packages is so slow!1!!"
This is kinda true, my machine is rather old (2008), so recreating the packages from the deltas might take some time, depending on the size of the package.
But it is still faster than downloading the entire package.
This could be further optimized by regenerating `getconf _NPROCESSORS_ONLN ` packages at a time.

P.S.: I hope there are no grave math errors, I was very tired when I compiled the list, and I and it would be great to have an official delta mirror in near future! big_smile
While researching I saw that there have been a few delta threads before suggesting this, but as of now, I could only find a single unofficial server.

Regards,  Matthias

Last edited by matthiaskrgr (2014-03-07 22:50:46)

Offline

#2 2014-03-06 12:21:59

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,365
Website

Re: [SOLVED] official delta mirror?

matthiaskrgr wrote:

Unfortunately, it seems to be the only delta mirror and provides no signatures for the package lists (because they are different from "original" arch ones and contain information about where to get deltas I guess?) which is kinda not-cool. If there was an official delta mirror, that might solve the signature problems.

You do know that no Arch mirrors have repo dbs signed?   Only packages.  The reconstituted package will still have its signature checked like usual.

Offline

#3 2014-03-06 12:25:08

matthiaskrgr
Member
Registered: 2012-04-03
Posts: 28
Website

Re: [SOLVED] official delta mirror?

Oops, I apologize to archlinux.fr then! smile
I saw something like this when updating

--2014-03-06 13:23:47--  http://delta.archlinux.fr/core/os/x86_64/core.db.sig
Resolving delta.archlinux.fr (delta.archlinux.fr)... 91.121.141.57
Connecting to delta.archlinux.fr (delta.archlinux.fr)|91.121.141.57|:80... connected.
HTTP request sent, awaiting response... 404 Not Found

and I though it was signatures for DBs ("core.db.sig")
So is this the package signatures which fail to download then?

Offline

#4 2014-03-06 14:49:48

drcouzelis
Member
From: Connecticut, USA
Registered: 2009-11-09
Posts: 4,092
Website

Re: [SOLVED] official delta mirror?

Wow, if I understand that Arch Wiki page correctly, deltas are an official feature of pacman. I had no idea! As someone who used to have a rather slow Internet connection, I can totally see the value in this. smile

matthiaskrgr wrote:

Unfortunately, it seems to be the only delta mirror and provides no signatures for the package lists (because they are different from "original" arch ones and contain information about where to get deltas I guess?) which is kinda not-cool. If there was an official delta mirror, that might solve the signature problems.

So, if this is not the case (according to Allan above) then is this thread [SOLVED]?

Offline

#5 2014-03-06 16:24:08

x33a
Forum Fellow
Registered: 2009-08-15
Posts: 4,587

Re: [SOLVED] official delta mirror?

drcouzelis wrote:

Wow, if I understand that Arch Wiki page correctly, deltas are an official feature of pacman. I had no idea! As someone who used to have a rather slow Internet connection, I can totally see the value in this. smile

Unfortunately, the value is nullified by the number of mirrors serving delta packages. I remember the few weeks I had used fedora for, that it had support for delta packages. At that time, it felt very alien to me (a long time back). I used to be surprised that the actual data downloaded was much less than what yum showed before download.

Offline

#6 2014-03-06 18:36:55

matthiaskrgr
Member
Registered: 2012-04-03
Posts: 28
Website

Re: [SOLVED] official delta mirror?

Well yes, if you are lucky, you can save a lot of traffic with delta packages.

If you ask me, the delta feature should be better "advertised" and more visible, which providing more delta servers would also lead to.
Maybe one could even just set up one "main" delta server which crunches the delta packages, and let the mirrors just pull the delta packages from there, not every server should need to regenerate them itself.

Offline

#7 2014-03-06 18:46:43

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: [SOLVED] official delta mirror?

Offline

#8 2014-03-06 23:23:18

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,365
Website

Re: [SOLVED] official delta mirror?

This is a zero priority for the Arch developers.  If people want it implemented, we need patches for our dbscripts (https://projects.archlinux.org/) to be supplied by interested people.

Offline

#9 2014-03-07 19:39:37

matthiaskrgr
Member
Registered: 2012-04-03
Posts: 28
Website

Re: [SOLVED] official delta mirror?

Mmh ok.. sad
The bugreport karol pointed at is closed, pointing to another one which is closed as well.
So I guess all this translates as "nope" (or at least "not now"?) for the idea to have an official delta mirror.

Feel free to mark this thread as [SOLVED] then.

Offline

#10 2014-03-07 22:48:17

cedeel
Member
From: ~
Registered: 2009-08-25
Posts: 176
Website

Re: [SOLVED] official delta mirror?

matthiaskrgr wrote:

Mmh ok.. sad
The bugreport karol pointed at is closed, pointing to another one which is closed as well.
So I guess all this translates as "nope" (or at least "not now"?) for the idea to have an official delta mirror.

Feel free to mark this thread as [SOLVED] then.

You're the one who should mark your thread. Just edit the title in the first post.

Offline

#11 2014-03-07 22:51:10

matthiaskrgr
Member
Registered: 2012-04-03
Posts: 28
Website

Re: [SOLVED] official delta mirror?

Oh, indeed, thanks. smile
Done.

Offline

#12 2014-03-07 23:37:10

progandy
Member
Registered: 2012-05-17
Posts: 5,184

Re: [SOLVED] official delta mirror?

If someone wants to implement a delta mirror, here are some ideas:
- First sync the mirror. Don't delete old packages, but move them in a backup.
- Use "xdelta3 encode" to create deltas.
- Delete obsolete package archives
- Use "xdelta3 merge" to build the new deltas for older versions,
- Delete obsolete (source of a merge) and too old delta files


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

#13 2014-03-08 01:02:52

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,365
Website

Re: [SOLVED] official delta mirror?

Or you could just use the tools provide by pacman (pkgdelta, repo-add --delta, cleanupdelta)

Offline

#14 2014-03-08 01:51:44

progandy
Member
Registered: 2012-05-17
Posts: 5,184

Re: [SOLVED] official delta mirror?

Allan wrote:

Or you could just use the tools provide by pacman (pkgdelta, repo-add --delta, cleanupdelta)

If I am not completely wrong this only allows one delta per package. If a package gets multiple updates per week it may be great to have deltas for a few more versions.


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

#15 2014-03-08 02:58:07

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,365
Website

Re: [SOLVED] official delta mirror?

pkgdelta allows you to create deltas between any package version, cleanupdelta removes deltas from the repo when they are no longer needed (due to total download size to get from old package version to current version)

Offline

#16 2014-03-08 03:56:00

progandy
Member
Registered: 2012-05-17
Posts: 5,184

Re: [SOLVED] official delta mirror?

Allan wrote:

pkgdelta allows you to create deltas between any package version, cleanupdelta removes deltas from the repo when they are no longer needed (due to total download size to get from old package version to current version)

Thanks. I finally understand the delta-logic.


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

Board footer

Powered by FluxBB