http://archdelta.net/: An Archlinux delta repository (i686 only)

sabooky · 2010-03-27 13:00:36

@Allan

I think thats actually correct, when I looked at it at work I was kind of rushed. I think the reason that gzip is an exception is because of the timestamp issue, xz and others don't seem to have that problem.

for those that wanted the source I've pushed it to github:
http://github.com/sabooky/archdelta

Ramses de Norre · 2010-04-01 11:27:18

There seems to be a problem with the delta for openjdk6:

checking delta integrity...
applying deltas...
generating openjdk6-6.b17_1.7.2-1-i686.pkg.tar.xz with openjdk6-6.b17_1.7.1-1_to_6.b17_1.7.2-1-i686.delta... 
xdelta3: non-seekable source in decode: XD3_INTERNAL
failed.
error: failed to commit transaction (delta patch failed)
Errors occurred, no packages were upgraded.

Last edited by Ramses de Norre (2010-04-01 11:27:44)

lucke · 2010-04-01 16:10:48

Downgrade to 3.0v or 3.0v2. I posted about the issue to xdelta's bugtracker.

sabooky · 2010-04-02 00:37:05

Since there seems to be issues with the newer version of xdelta3. I've uploaded the 3.0v-2 version to archdelta.net

It can be found here:
http://archdelta.net/xdelta3-3.0v-2-i686.pkg.tar.gz

Ramses de Norre · 2010-04-02 10:18:50

sabooky wrote:

Since there seems to be issues with the newer version of xdelta3. I've uploaded the 3.0v-2 version to archdelta.net
It can be found here:
http://archdelta.net/xdelta3-3.0v-2-i686.pkg.tar.gz

Thanks And god, this is great on my low speed connection in the students dorm! Thank you a lot for setting up this repo

karol · 2010-04-02 11:03:36

[OT]
@Ramses de Norre
> low speed connection in the students dorm
I beg your pardon? Here the dorms get 100Mbps ... That's one of the reasons people study not 5 but more like 7 years (higher education in Poland is free i.e. everyone pays for it paying taxes).
[/OT]

Ramses de Norre · 2010-04-02 11:15:41

karol wrote:

[OT]
@Ramses de Norre
> low speed connection in the students dorm
I beg your pardon? Here the dorms get 100Mbps ... That's one of the reasons people study not 5 but more like 7 years (higher education in Poland is free i.e. everyone pays for it paying taxes).
[/OT]

60kbps here.. Sucks big time.

karol · 2010-04-02 11:51:06

> Sucks big time.
Depends how you see it ;P A phat pipe can suck more.

Ramses de Norre · 2010-04-15 09:36:42

Hi

Is it expected behaviour that some packages are not in the repo? (Noticed today: curl and aria2) Is it because maybe they haven't been updated since the repo came into existence?

shining · 2010-04-21 13:13:14

sabooky wrote:

pacman devs
Hmm.. I wonder if this move to xz is gonna break with the following logic in the pacman sourcecode
./lib/libalpm/sync.c
      if(endswith(to, ".gz")) {
        /* special handling for gzip : we disable timestamp with -n option */
        snprintf(command, PATH_MAX, "xdelta3 -d -q -R -c -s %s %s | gzip -n > %s", from, delta, to);
      } else {
        snprintf(command, PATH_MAX, "xdelta3 -d -q -s %s %s %s", from, delta, to);
      }
The following might need to be added.
else if (endswith(to,".xz")) {
snprintf(command, PATH_MAX, "xdelta3 -d -q -R -c -s %s %s | xz > %s", from, delta, to);
}
I'm at work right now so I can't look too much into it.. but this is definitely an interesting problem.. and I really hope that this xz change doesn't completely break archdelta.net for the time being :\.
I would really hate having to use a modified pacman to get archdelta to work.
Thanks for the feedback

If you had posted this on the ML, I would have read this immediately, I barely read the forums nowadays. So I found this thread only now, months after it was created.
But first kudos to all your work, that's awesome !
Ideally we would work together to improve the pacman, repo-add and similar tools, so that all the xdelta code is found either in pacman repo.
http://projects.archlinux.org/pacman.git/
And then look at the dbscripts side of things (I know nothing of that though) :
http://projects.archlinux.org/dbscripts.git/

xdelta handles compressed files by uncompressing/recompressing transparently :
http://code.google.com/p/xdelta/wiki/Ex … ompression
If it does not do that / does not support format compression, it is VERY inefficient in size.
xz support was added relatively recently, Oct 2009, so it might be only present in the last two releases (x and y) :
http://code.google.com/p/xdelta/issues/detail?id=87

I did not fully understand why you use the old v release, but I doubt it will work properly with xz.

Now the issue with this external recompression is that pacman (originally/historically) tracks md5sum of *compressed packages*.
This means that when xdelta applies a delta and recompress the final file, that compression must be EXACTLY like the original one. If there is the slightest difference, even just one bit change, pacman will report a corrupted package because of different md5.
The issue with gzip is that by default it applies a timestamp of the time the compression is made. So that clearly break things in our use case.
So we have two things to do :
- disable timestamp at package creation in makepkg : *tar.gz) gzip -f -n "$pkg_file" ;;
- do not let xdelta recompress the file on its own, and apply gzip with -f manually (in above pacman code).

AFAIK there is no timestamp or similar issue with xz, so it might work out of the box.
So the default case is supposed to handle bzip2/xz package.

The move from gzip to xz in repo was not a bad idea, a simple change to use a more efficient compression format, resulting in smaller packages / smaller download for everyone. xz is slower at compression, but reasonably fast at decompression. This seemed ideal : a package is compressed just once (by one developer), but decompressed many times (thousands of users).

Unfortunately this might suck a lot with xdelta :
- the package needs to be recompressed, so the above argument is lost
- xz package are already smaller, and delta ratio is less interesting than with gzip

If we want more flexibility on the recompression part, we might need a big rework of pacman code. We will need to work on uncompressed packages for everything : package integrity with md5, creation and application of delta
I am not sure how big this change would be, and whether it's really worth it.

rggjan · 2010-04-27 07:18:17

shining wrote:

I am not sure how big this change would be, and whether it's really worth it.

Well, in my opinion it would be really worth it. It is already a big improvement now, having only to download 2 MB instead of 70 MB when going from jdk-x-1 to jdk-x-2 for example.

Even though, I have to admit, it takes a lot of time now recompressing. Still, you can save much bandwith, on the client and on the server...

Also, isn't it a waste to keep all the old packages one downloads and never use them? (exept for reinstalling things, of course...)

adrianx · 2010-04-27 07:40:52

shining wrote:

Unfortunately this might suck a lot with xdelta :
- xz package are already smaller, and delta ratio is less interesting than with gzip

Even though Fedora uses xz, I still usually get a 70%+ reduction in bandwidth (with yum-presto). Just the other day, a 370MB update was reduced to around ±50MB. That is pretty interesting to me.

I admit that it is very possible that I'm misunderstanding.

karol · 2010-04-27 11:28:19

@adrianx
I think shining was worried about the complexity of the implementation and smaller benefits wrt to using xdelta3 + xz. Binary deltas give great improvements, so does xz compression, but combining the two might be tricky and

shining wrote:

I am not sure how big this change would be, and whether it's really worth it.

shining · 2010-04-29 09:50:32

Just to clarify what I said : someone complained that applying delta on xz packages took a long time.

And indeed, that's because applying delta on a compressed package does the following transparently :
1) decompress package (gzip fast, xz fast enough)
2) apply
3) re-compress package (gzip fast, xz slow)

If that's not a problem for you, then good, we already have all that working.

However, if we want to avoid the re-compression step 3) which is slow with xz, it will need a bigger rework than someone might think.
Because we currently do not have any controls on that, it's all done transparently by xdelta3.

muzenbaher · 2010-05-20 20:20:43

Hi

Is it possible to create locale mirror with http://archdelta.net using a rsync or something else?

sandsmark · 2010-06-12 17:32:10

I set up my own delta mirror, for x86_64: http://delta.mts.ms/

So far it hasn't generated any deltas, since I'm too lazy to use ARM. :-P

shinythings · 2010-06-24 09:04:28

This is great, it's been saving me lots of time on my excruciatingly slow connection.

I've noticed that the latest updates to wesnoth (1.8.2-2) and xulrunner (1.9.2.4-1) don't seem to be available. Especially wesnoth would be really useful since it's just a pkgrel bump for a boost rebuild, and 260mb download.

Does this mean the deltas on these didn't meet the 70% rule? Or is there some other reasons for leaving some packages out?

Ramses de Norre · 2010-07-08 17:45:12

It seems like the repo is not syncing any longer?

sabooky · 2010-07-15 02:55:26

Ramses de Norre wrote:

It seems like the repo is not syncing any longer?

Just noticed this post.. yeah you're right.. apparently my python and openssl got out of sync so I was getting the following error:

    import md5
  File "/usr/lib/python2.6/md5.py", line 10, in <module>
    from hashlib import md5
  File "/usr/lib/python2.6/hashlib.py", line 136, in <module>
    md5 = __get_builtin_constructor('md5')
  File "/usr/lib/python2.6/hashlib.py", line 63, in __get_builtin_constructor
    import _md5
ImportError: No module named _md5

Anyways, doing a pacman -Syu.. I'll leave it overnight since it's a huge upgrade:

Total Installed Size:   5840.26 MB

As long as nothing blows up on my desktop, I'll reboot tomorrow and hopefully everything will start syncing again properly.

Wish I had a server to run this on.. t's running on my desktop currently and.. well.. desktop isn't exactly the most stable machine in the world...

Thanks for bringing this up

karol · 2010-07-15 03:21:20

Maybe you could post some stats (size of that thing) and ask for a bit of server space / bandwidth?

sabooky · 2010-07-17 19:17:02

karol wrote:

Maybe you could post some stats (size of that thing) and ask for a bit of server space / bandwidth?

8.3G    community/os/i686
368M    community-testing/os/i686
259M    core/os/i686
9.0G    extra/os/i686
1.9G    testing/os/i686
407M    community/os/i686/deltas
8.6M    community-testing/os/i686/deltas
64M    core/os/i686/deltas
1.5G    extra/os/i686/deltas
87M    testing/os/i686/deltas
22G    total

The scripts are pretty small in size, however a full copy of the repository needs to be accessible to use for generating the diffs.

I wish I could give this project more time, but work killed the hobby side of IT for me

Anyways, if someone has good bandwith 24/7 uptime, and 30gigs (60gigs if they wanna do x64 also) free and wants to run this locally.. I'll be glad to help them get set up.

Also.. looking at the above, its only 2gigs (4gigs if x64 also) or so for the delta only portions (thats all a webserver would need to serve)

I'm sorry for not supporting this as well as I could.. Having Linux work be my day job really killed the hobby side for me .

Hopefully someone takes the code and improves on this project or it gets integrated into the official repos.

karol · 2010-07-17 19:34:28

> Having Linux work be my day job really killed the hobby side for me
Many people would be happy to work w/ Linux: https://bbs.archlinux.org/viewtopic.php?id=99761

> if someone has good bandwith 24/7 uptime
How much bandwidth is needed /month?
Were you creating deltas for all the packages? I don't see wesnoth and IIRC there were also some other big packages missing.

sandsmark provides 64-bit already http://delta.mts.ms/

sabooky · 2010-07-17 21:24:39

karol wrote:

> Having Linux work be my day job really killed the hobby side for me
Many people would be happy to work w/ Linux: https://bbs.archlinux.org/viewtopic.php?id=99761

Don't get me wrong, I love my job. That said, 40hrs/week is enough Linux for me. The "hobby" (fun little project here and there) died for the most part. I still love Linux.

karol wrote:

> if someone has good bandwith 24/7 uptime
How much bandwidth is needed /month?

Hmm.. actually the 24/7 uptime is only needed on the server hosting the deltas. The delta creating machine only needs to sync, create deltas, and then upload (I do this nightly, but others may prefer more/less).

karol wrote:

Were you creating deltas for all the packages? I don't see wesnoth and IIRC there were also some other big packages missing.

As for wesnoth and (possibly) other missing packages:

creating delta: wesnoth-1.8.2-1-i686.pkg.tar.xz -> wesnoth-1.8.2-2-i686.pkg.tar.xz => wesnoth-1.8.2-1_to_1.8.2-2-i686.delta
Delta size too big: /var/repos/extra/os/i686/deltas/wesnoth-1.8.2-1_to_1.8.2-2-i686.delta
deleting /var/repos/extra/os/i686/deltas/wesnoth-1.8.2-1_to_1.8.2-2-i686.delta
deleting /var/repos/extra/os/i686/wesnoth-1.8.2-1-i686.pkg.tar.xz

If one delta file is bigger than 70% that leaves 0 deltas.

From my first post:

sabooky wrote:

* I am only keeping up to 3 versions back for each package. Also, delta size combined must be less than %70 of the pkg size which is the pacman cutoff.

karol wrote:

sandsmark provides 64-bit already http://delta.mts.ms/

sandsmark,

Do you have the space/bandwidth to also do i686? If you decide to do so, let me know so I can change archdelta.net to mirror you rather than generating different sets of deltas (pretty sure md5sums will be different).

karol · 2010-07-17 21:37:40

If sandsmark doesn't respond, contact him directly: https://bbs.archlinux.org/profile.php?id=15883

> If one delta file is bigger than 70% that leaves 0 deltas.
<facepalm> I'm braindead sometimes, sorry.

josh.macdonald · 2010-08-03 02:25:47

Hi all,

xdelta3 author here, just following up on the bug reported in 3.0w, which was fixed in 3.0y. I'm trying to close issue 110, here:

http://code.google.com/p/xdelta/issues/detail?id=110

I couldn't determine whether the error was caused by applying pre-3.0w-generated deltas with a 3.0y binary, or whether it was a bug in 3.0y that still exists. In any case, 3.0z is released and I'd appreciate your help with any information. The newer code is faster than 3.0v in the way it handles compressed source files, it decompresses them through a pipe instead of making an uncompressed copy in a temporary file.

And by the way, on the subject of "bsdiff wins.. no surprise there", I wonder if that's a fair comparison. If you're running xdelta3 with its default settings, you're getting only 64MB of source buffer, meaning xdelta3 will not find any similarities that occur at a distance of greater than 64MB. With -B set to the size of the source file you should see an improvement, and you might also set -9 and -S djw if you really want a fair comparison to bsdiff. Thanks, Josh

Arch Linux

#26 2010-03-27 13:00:36

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#27 2010-04-01 11:27:18

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#28 2010-04-01 16:10:48

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#29 2010-04-02 00:37:05

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#30 2010-04-02 10:18:50

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#31 2010-04-02 11:03:36

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#32 2010-04-02 11:15:41

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#33 2010-04-02 11:51:06

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#34 2010-04-15 09:36:42

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#35 2010-04-21 13:13:14

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#36 2010-04-27 07:18:17

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#37 2010-04-27 07:40:52

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#38 2010-04-27 11:28:19

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#39 2010-04-29 09:50:32

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#40 2010-05-20 20:20:43

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#41 2010-06-12 17:32:10

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#42 2010-06-24 09:04:28

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#43 2010-07-08 17:45:12

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#44 2010-07-15 02:55:26

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#45 2010-07-15 03:21:20

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#46 2010-07-17 19:17:02

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#47 2010-07-17 19:34:28

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#48 2010-07-17 21:24:39

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#49 2010-07-17 21:37:40

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#50 2010-08-03 02:25:47

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

Board footer