http://archdelta.net/: An Archlinux delta repository (i686 only)

karol · 2010-08-03 03:01:57

josh.macdonald wrote:

Hi all,
xdelta3 author here, just following up on the bug reported in 3.0w, which was fixed in 3.0y. I'm trying to close issue 110, here:
http://code.google.com/p/xdelta/issues/detail?id=110
I couldn't determine whether the error was caused by applying pre-3.0w-generated deltas with a 3.0y binary, or whether it was a bug in 3.0y that still exists. In any case, 3.0z is released and I'd appreciate your help with any information. The newer code is faster than 3.0v in the way it handles compressed source files, it decompresses them through a pipe instead of making an uncompressed copy in a temporary file.
And by the way, on the subject of "bsdiff wins.. no surprise there", I wonder if that's a fair comparison. If you're running xdelta3 with its default settings, you're getting only 64MB of source buffer, meaning xdelta3 will not find any similarities that occur at a distance of greater than 64MB. With -B set to the size of the source file you should see an improvement, and you might also set -9 and -S djw if you really want a fair comparison to bsdiff. Thanks, Josh

I think that bsdiff v. xdelta3 was a very loose comparison: bsdiff is always much slower but often produces much smaller deltas. With 'xdelta -1 -S djw' you get the deltas really fast - I use it for my hourly backups :-)

About the bug: I'm quite sure the delta was created with the version 3.0v. As I have only 1GB RAM, I could go only that high with the -B switch and the higher values didn't help.

sabooky · 2010-08-16 11:08:22

josh.macdonald wrote:

And by the way, on the subject of "bsdiff wins.. no surprise there", I wonder if that's a fair comparison. If you're running xdelta3 with its default settings, you're getting only 64MB of source buffer, meaning xdelta3 will not find any similarities that occur at a distance of greater than 64MB. With -B set to the size of the source file you should see an improvement, and you might also set -9 and -S djw if you really want a fair comparison to bsdiff. Thanks, Josh

Setting -B to the size of the source gave me worse results (this was done with the 3.0v of the software).

Could you show an example of xdelta3 usage with the -B flag and some performance numbers?

Thanks

josh.macdonald · 2010-08-24 06:38:02

sabooky wrote:

josh.macdonald wrote:
And by the way, on the subject of "bsdiff wins.. no surprise there", I wonder if that's a fair comparison. If you're running xdelta3 with its default settings, you're getting only 64MB of source buffer, meaning xdelta3 will not find any similarities that occur at a distance of greater than 64MB. With -B set to the size of the source file you should see an improvement, and you might also set -9 and -S djw if you really want a fair comparison to bsdiff. Thanks, Josh
Setting -B to the size of the source gave me worse results (this was done with the 3.0v of the software).
Could you show an example of xdelta3 usage with the -B flag and some performance numbers?
Thanks

The problem with -B turned out to be a bug in the "source-from-fifo" code which was introduced (with a fresh set of bugs, apparently) in 3.0w. Since 3.0w, when xdelta encounters compressed inputs (e.g., a gzip or bzip2 source and target), it forks a process to decompress the input file and then reads the data directly from the pipe with no intermediate file. Raising -B gives xdelta more memory to buffer the source data in memory, and it can drastically improve performance in cases where large chunks of data have moved great distances between versions. If your files do not exhibit this characteristic, then raising -B will likely have a slight negative effect, since there is a marginal cost to raising -B (because it takes more bits to encode addresses). A contrived example where raising -B would help: the old version consists of 16 * 64MB chunks, the new version contains the same 16 chunks but shuffled so that they appear in a random order. If you run xdelta with -B$[512 * 1024 * 1024] then it will perform well. The problem with -B that you observed was caused by a bad optimization, I tried to explain in a comment here, http://code.google.com/p/xdelta/source/detail?r=324, in any case I've simplified the code and tossed a couple of minor optimizations that only matter for small files. The bug exists in 3.0z and I have it fixed in SVN 324. The bug is documented here, http://code.google.com/p/xdelta/issues/detail?id=113, and I've verified the results are now as expected. The above discussion is about raising -B to improve compression in the ENCODER.

Now -- back to the issue that started this conversation, http://code.google.com/p/xdelta/issues/detail?id=110. As explained above, the new code is much improved because it doesn't need to write a temporary file to decompress the source file prior to encoding/decoding a delta. The program decide at the beginning how much memory it will use to buffer data read from the source FIFO, this is the value set by -B. The problem being reported is that 3.0y and 3.0z cannot decode a delta created by 3.0v, which is true, but the problem is with 3.0v. I can't modify 3.0y and later to work around this problem, it's simply that when xdelta reads compressed data it sends the data through a pipe and it must know at the beginning how much buffer space it needs. The assumption is that so long as the encoder and decoder use the same setting for -B, it works. However, 3.0v has a bug in which it disregards the -B setting and allows itself to emit a copy that goes too far back (this has to do with the fact that 3.0v decompresses the source to a temporary file and then seeks around, it was not forced to respect the FIFO buffer discipline of later versions). Thus, 3.0v emits a copy instruction that goes further back than the (post-3.0w) decoder expects given its setting of -B. To decode a 3.0v delta using 3.0y and later, run the DECODER with a larger value of -B.

Another option would be for you to decode (using 3.0v) and re-encode all your deltas using a forthcoming release. The next version (3.0aa) will be released in a few days.

Does this make sense? Thanks, Josh

gtklocker · 2010-08-25 19:45:14

Hi guys, can I somehow rsync (or download) the delta repository locally, on my server?

Ramses de Norre · 2010-11-11 12:54:16

I have the impression that [community] isn't syncing, is this possible?

berot3 · 2010-12-15 19:52:51

so no x86_64 -repo at all at the moment?

http://delta.mts.ms/ doesnt seem to work:

Yeah, I'll have to get this back up.

sandsmark · 2010-12-23 17:46:18

fixed now, but no actual deltas yet.

sandsmark · 2010-12-31 13:42:54

I've been syncing it manually every time I needed it, I'll add it to cron if people actually need/want it.

gtklocker · 2011-01-18 19:56:54

sandsmark wrote:

I've been syncing it manually every time I needed it, I'll add it to cron if people actually need/want it.

/me needs

sabooky · 2011-01-24 11:04:13

Dreamhost was for $9 or so for the first year.. the second year charges that on a monthly basis.

Ayways, a year is almost up. I personally haven't been using the delta repositories for a while now.. but wondering if there are people who are still interested in archdelta.net.

Has anyone mirrored or improved on the code? I've been pretty MIA when it came to maintaining this project (just did enough to keep the site going).. wondering if anyone picked this idea up.

Anyone want to take over this project? I'm sure there are a lot of people who can do a better job than me at maintaining this.

karol · 2011-01-24 11:29:53

sabooky wrote:

I personally haven't been using the delta repositories for a while now.. but wondering if there are people who are still interested in archdelta.net.

Don't you have any kind of stats, e.g. how much bandwidth did the project use per month?

ataraxia · 2011-02-27 05:36:48

So, it looks like neither of these repos are active any longer (sabooky's is dead and sandsmark's is out of date). Any progress on getting this set up on the official repos? What can non-devs like myself do to help make this happen? I'd say that this is at least as big a deal as package signing, and as the support is already implemented in pacman for deltas, it would hopefully be closer to completion.

Cdh · 2011-03-09 19:48:04

I happened to think about this today when updating eclipse.

There is this bugreport. So far it only has my vote and probably the reporter's. I wouldn't feel motivated to implement something with that little interest on the bugtracker either...

https://bugs.archlinux.org/task/18590

berot3 · 2011-04-22 20:19:59

i wonder why it doesnt get more attention... it should be standard! i came to arch because the easy and really fast installing/updating, delta would make this LIGHTNING-fast! i think in fedora and opensuse its already standard. just ubuntu seems to take more time...
so the problem at the moment is the few support it gets?

karol · 2011-04-22 20:30:47

berot3 wrote:

i wonder why it doesnt get more attention... it should be standard! i came to arch because the easy and really fast installing/updating, delta would make this LIGHTNING-fast! i think in fedora and opensuse its already standard. just ubuntu seems to take more time...
so the problem at the moment is the few support it gets?

Patches welcome.
Both Fedora and opensuse have a couple more devs I think :-)

Many people have fast enough internet connection so they don't care.

Oh, and I'm not sure if deltas would make things faster: I think you need to create the packages locally (from previous version + deltas) in order to install them, so for people with slow cpus and fast internet connection it would mean the updates would actually be slower.

Last edited by karol (2011-04-22 20:34:22)

karol · 2011-06-11 12:27:24

https://wiki.archlinux.org/index.php/Deltup
Seems we have a delta repo :-)

sva_h4cky0 · 2011-07-28 09:49:59

karol wrote:

https://wiki.archlinux.org/index.php/Deltup
Seems we have a delta repo :-)

Thx Karol for the info, but unfortunately it's seem not all package have delta. Just curious

karol · 2011-07-28 13:32:32

sva_h4cky0 wrote:

karol wrote:
https://wiki.archlinux.org/index.php/Deltup
Seems we have a delta repo :-)
Thx Karol for the info, but unfortunately it's seem not all package have delta. Just curious

If the delta is almost as big as the regular package, it makes little sense to provide a delta, but it's just a guess and you have to contact the maintainers to get a definite answer.

Allan · 2011-07-29 09:01:48

karol wrote:

If the delta is almost as big as the regular package, it makes little sense to provide a delta

In fact it makes no sense. If the total deltas are greater than 70% of the package size, pacman will download the package.

Arch Linux

#51 2010-08-03 03:01:57

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#52 2010-08-16 11:08:22

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#53 2010-08-24 06:38:02

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#54 2010-08-25 19:45:14

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#55 2010-11-11 12:54:16

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#56 2010-12-15 19:52:51

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#57 2010-12-23 17:46:18

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#58 2010-12-31 13:42:54

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#59 2011-01-18 19:56:54

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#60 2011-01-24 11:04:13

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#61 2011-01-24 11:29:53

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#62 2011-02-27 05:36:48

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#63 2011-03-09 19:48:04

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#64 2011-04-22 20:19:59

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#65 2011-04-22 20:30:47

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#66 2011-06-11 12:27:24

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#67 2011-07-28 09:49:59

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#68 2011-07-28 13:32:32

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#69 2011-07-29 09:01:48

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

Board footer