You are not logged in.
Pages: 1
Hi, to make it short, I'd like to have an official delta mirror.
My bandwidth has been limited to mostly ~8kb/s in the last months and I have still been able to keep my arch system up to date with deltas.
(How this works: basically you only download the differences between your version and the up-to-date version of the package ("libarchive-3.1.2-4_to_3.1.2-6-x86_64.delta)". Before installing, the libarchive-3.1.2-6-x86_64 package is then recreated from the "patch" and everything behaves as if you downloaded the original one.)
So far I have been using delta.archlinux.fr
Unfortunately, it seems to be the only delta mirror and provides no signatures for the package lists (because they are different from "original" arch ones and contain information about where to get deltas I guess?) which is kinda not-cool. If there was an official delta mirror, that might solve the signature problems.
Edit: I misunderstood here, apparently this is not about signatures of repo DBs
Why should you care about deltas?
The delta packages can be a lot smaller than the "regular" packages I have made a list of the packages updated since beginning of this month:
name delta regular saved:
glibc 202.8 K 8.3 M 97%
device-mapper 165.9 K
cryptsetup 192.7 K
feh 170.2 K
lib32-glibc 2.6 M
libgphoto2 708.2 K 1.1 M 37%
libtirpc 133.0 K
readline 281.1 K
lvm2 767.4 K
ppp 279.7 K
python2-lxml 657.0 K
raptor 256.9 K
rasqal 243.8 K
virtuoso-base 1.8 M 2.9 M 38%
gcr 329 K 777.9 K 57%
ldoc 88.3 K
libpulse 333.3 k
webkitgtk 7.7 M 20.8 M 37%
webkitgtk2 8 M
inxi 97.9 K
blender 17.4 M 39.5 M 56%
systemd 66.8 K 3 M 97%
chromium 17.2 M 30.6 M 43%
libunique 26.6 K
xterm 266.1 K
lib32-systemd 154 K
smpeg 96.4 K
xbmc 570.2 K 21.7 M 97%
parted 378.7 K
gnutls 362 K 1.9 M 81%
calligra-filters 3 M
calligra-libs 2.9 M
calligra-plugins 2.9 M
calligra-words 234.9 K 956.7 K 75%
calligra-flow 460.7 K 5.9 M 92%
libburn 245.9 K
libisofs 197.1 K
packages with deltas only:
downloaded: 47 MB actual (regular) size: 136 MB
=> 65 % saved, 89MB saved
all packages:
downloaded: 71 MB actual (regular) size: 166 MB
=> 57% saved, 95 MB saved
There might rarely be delta version of big binary packages (I have not yet seen any kernel delta packages for example), but if a package is just rebuild for a .so-bump and gets its pkgrel bumped (1.0-1 -> 1.0-2) I often can save quite some traffic with delta support. I also save a lot for packages that contain a lot of graphics (games for example) and only have their binaries changed, a few graphics.
There is also still a lot of room for optimization!
Yesterday I proposed a patch that increases xdelta3s compression level (so we might get smaller deltas with next pacman major release, if the patch is merged).
Also, bsdiff is able to create much smaller deltas than xdelta3 (but also needs more resources at generation).
Comparing /usr/bin/blender of the 2 latest blender releases:
xdelta3 10.4M # default
xdelta3 -9 9.9M # highest compression level
bsdiff 4.7M
Edit: size of the binary: 47MB
However, while xdelta3 is clever enough to look into archives and compress against the differences of the files contained INSIDE the archive, bsdiff would just generate a delta BETWEEN the archives (does not look at the content), so an "xdelta3 using the bsdiff algorithm" still has to be made (by someone)
"But decompression of the packages is so slow!1!!"
This is kinda true, my machine is rather old (2008), so recreating the packages from the deltas might take some time, depending on the size of the package.
But it is still faster than downloading the entire package.
This could be further optimized by regenerating `getconf _NPROCESSORS_ONLN ` packages at a time.
P.S.: I hope there are no grave math errors, I was very tired when I compiled the list, and I and it would be great to have an official delta mirror in near future!
While researching I saw that there have been a few delta threads before suggesting this, but as of now, I could only find a single unofficial server.
Regards, Matthias
Last edited by matthiaskrgr (2014-03-07 22:50:46)
Offline
Unfortunately, it seems to be the only delta mirror and provides no signatures for the package lists (because they are different from "original" arch ones and contain information about where to get deltas I guess?) which is kinda not-cool. If there was an official delta mirror, that might solve the signature problems.
You do know that no Arch mirrors have repo dbs signed? Only packages. The reconstituted package will still have its signature checked like usual.
Offline
Oops, I apologize to archlinux.fr then!
I saw something like this when updating
--2014-03-06 13:23:47-- http://delta.archlinux.fr/core/os/x86_64/core.db.sig
Resolving delta.archlinux.fr (delta.archlinux.fr)... 91.121.141.57
Connecting to delta.archlinux.fr (delta.archlinux.fr)|91.121.141.57|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
and I though it was signatures for DBs ("core.db.sig")
So is this the package signatures which fail to download then?
Offline
Wow, if I understand that Arch Wiki page correctly, deltas are an official feature of pacman. I had no idea! As someone who used to have a rather slow Internet connection, I can totally see the value in this.
Unfortunately, it seems to be the only delta mirror and provides no signatures for the package lists (because they are different from "original" arch ones and contain information about where to get deltas I guess?) which is kinda not-cool. If there was an official delta mirror, that might solve the signature problems.
So, if this is not the case (according to Allan above) then is this thread [SOLVED]?
Offline
Wow, if I understand that Arch Wiki page correctly, deltas are an official feature of pacman. I had no idea! As someone who used to have a rather slow Internet connection, I can totally see the value in this.
Unfortunately, the value is nullified by the number of mirrors serving delta packages. I remember the few weeks I had used fedora for, that it had support for delta packages. At that time, it felt very alien to me (a long time back). I used to be surprised that the actual data downloaded was much less than what yum showed before download.
Offline
Well yes, if you are lucky, you can save a lot of traffic with delta packages.
If you ask me, the delta feature should be better "advertised" and more visible, which providing more delta servers would also lead to.
Maybe one could even just set up one "main" delta server which crunches the delta packages, and let the mirrors just pull the delta packages from there, not every server should need to regenerate them itself.
Offline
There's e.g. https://bugs.archlinux.org/task/28021
Offline
This is a zero priority for the Arch developers. If people want it implemented, we need patches for our dbscripts (https://projects.archlinux.org/) to be supplied by interested people.
Offline
Mmh ok..
The bugreport karol pointed at is closed, pointing to another one which is closed as well.
So I guess all this translates as "nope" (or at least "not now"?) for the idea to have an official delta mirror.
Feel free to mark this thread as [SOLVED] then.
Offline
Mmh ok..
The bugreport karol pointed at is closed, pointing to another one which is closed as well.
So I guess all this translates as "nope" (or at least "not now"?) for the idea to have an official delta mirror.Feel free to mark this thread as [SOLVED] then.
You're the one who should mark your thread. Just edit the title in the first post.
Offline
Oh, indeed, thanks.
Done.
Offline
If someone wants to implement a delta mirror, here are some ideas:
- First sync the mirror. Don't delete old packages, but move them in a backup.
- Use "xdelta3 encode" to create deltas.
- Delete obsolete package archives
- Use "xdelta3 merge" to build the new deltas for older versions,
- Delete obsolete (source of a merge) and too old delta files
| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |
Offline
Or you could just use the tools provide by pacman (pkgdelta, repo-add --delta, cleanupdelta)
Offline
Or you could just use the tools provide by pacman (pkgdelta, repo-add --delta, cleanupdelta)
If I am not completely wrong this only allows one delta per package. If a package gets multiple updates per week it may be great to have deltas for a few more versions.
| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |
Offline
pkgdelta allows you to create deltas between any package version, cleanupdelta removes deltas from the repo when they are no longer needed (due to total download size to get from old package version to current version)
Offline
pkgdelta allows you to create deltas between any package version, cleanupdelta removes deltas from the repo when they are no longer needed (due to total download size to get from old package version to current version)
Thanks. I finally understand the delta-logic.
| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |
Offline
Pages: 1