http://archdelta.net/: An Archlinux delta repository (i686 only)

sabooky · 2010-02-27 01:47:32

Background:
I've been interested in pacman delta support for some time now, I made a post a few years back about it here, but never really followed up with it.

Anyways.. the arch developers have added support for delta packages, the only thing that is missing is a delta repository. So I've decided to make an arch linux delta mirror.

Usage:
1) edit /etc/pacman.d/mirrorlist adding the following as the top repository

#Only the i686 versions of core, extra, community repos are supported
Server = http://archdelta.net/$repo/os/i686

2) edit /etc/pacman.conf adding "UseDelta" to the [options] section

3) reply to this thread posting your `pacman -Syuw` "Total Download Size" with and without "UseDelta"

My results:

home: 635.20MB => 227.25MB  (36% of reg)
work: 739.77MB => 301.14MB  (40% of reg)

...
Thats all there is to it really

NOTES:
* This repository is a delta only repository.. which means if you want a package that has no delta files you will get an error message about the pkg not being found. However, pacman will automatically fall back to the next mirror and grab it from there. So please make sure that archdelta.net isn't the only mirror configured in pacman.
* I am only keeping up to 3 versions back for each package. Also, delta size combined must be less than %70 of the pkg size which is the pacman cutoff.
* The repository starts syncing at ~3:30am EST with the most recently synced mirrors. (if there's an official master repository I should be syncing to, please let me know)

Numbers and technical details...:
The compression used to generate the delta files is xdelta -9 + gzip 9 compression (using the python gzip library).
xdelta -9 + gzip seems to provide excellent compression and performance. The reason I went with gzip 9 is because it was the default in python.

I'm using xdelta3 (3.0v-2) because the 3.0w version has a bug where it doesn't decompress input files correctly (netting a delta thats almost the same size as the package).

Here's some compression results using different options (syntax name.<xdelta_opts>.delta.<compression_opts>.{gz,xz,bzip2}):

kernel:

2164    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.bsdiff            <= bsdiff wins.. no surprise there
2908    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.9.delta.9e.xz
2912    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.9.delta.9.xz
2912    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.9.delta.xz
2928    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.delta.9e.xz
2928    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.delta.9.xz
2928    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.delta.xz
2996    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.9.delta.9py.gz    <= this is what the repository uses
3008    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.9.delta.9.gz
3008    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.9.delta.gz
3024    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.delta.gz
3084    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.9djw.delta.9e.xz
3084    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.9djw.delta.9.xz
3084    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.9djw.delta.xz
3092    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.djw.delta.9.xz
3092    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.djw.delta.xz
3096    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.9djw.delta.gz
3096    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.djw.delta.9e.xz
3108    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.djw.delta.gz
3116    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.9.delta.bz2
3124    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.9djw.delta.bz2
3132    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.djw.delta.bz2
3136    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.9djw.delta
3136    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.delta.bz2
3148    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.djw.delta
3216    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.0.delta.9e.xz
3216    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.0.delta.9.xz
3216    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.0.delta.xz
3356    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.9.delta
3372    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.delta
3404    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.0.delta.gz
3588    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.0.delta.bz2
4304    kernel26-2.6.32.7-1_to_-2.6.32.8-1-i686.0.delta
29284   kernel26-2.6.32.7-1-i686.pkg.tar.gz
29288   kernel26-2.6.32.8-1-i686.pkg.tar.gz

evolution:

400     evolution-2.28.1-1_to_2.28.2-1-i686.bsdiff               <= o.O more bsdiff goodness
1120    evolution-2.28.1-1_to_2.28.2-1-i686.9.delta.9e.xz
1120    evolution-2.28.1-1_to_2.28.2-1-i686.9.delta.9.xz
1120    evolution-2.28.1-1_to_2.28.2-1-i686.9.delta.xz
1196    evolution-2.28.1-1_to_2.28.2-1-i686.9.delta.9py.gz       <= this is what the repository uses
1204    evolution-2.28.1-1_to_2.28.2-1-i686.9.delta.9.gz
1204    evolution-2.28.1-1_to_2.28.2-1-i686.9.delta.gz
1236    evolution-2.28.1-1_to_2.28.2-1-i686.9djw.delta.9.xz
1236    evolution-2.28.1-1_to_2.28.2-1-i686.9djw.delta.xz
1240    evolution-2.28.1-1_to_2.28.2-1-i686.9djw.delta.9e.xz
1256    evolution-2.28.1-1_to_2.28.2-1-i686.9.delta.bz2
1256    evolution-2.28.1-1_to_2.28.2-1-i686.9djw.delta.gz
1264    evolution-2.28.1-1_to_2.28.2-1-i686.9djw.delta.bz2
1272    evolution-2.28.1-1_to_2.28.2-1-i686.9djw.delta
1380    evolution-2.28.1-1_to_2.28.2-1-i686.delta.9e.xz
1384    evolution-2.28.1-1_to_2.28.2-1-i686.delta.9.xz
1384    evolution-2.28.1-1_to_2.28.2-1-i686.delta.xz
1440    evolution-2.28.1-1_to_2.28.2-1-i686.0.delta.9e.xz
1440    evolution-2.28.1-1_to_2.28.2-1-i686.0.delta.9.xz
1440    evolution-2.28.1-1_to_2.28.2-1-i686.0.delta.xz
1460    evolution-2.28.1-1_to_2.28.2-1-i686.9.delta
1484    evolution-2.28.1-1_to_2.28.2-1-i686.delta.gz
1520    evolution-2.28.1-1_to_2.28.2-1-i686.djw.delta.9e.xz
1520    evolution-2.28.1-1_to_2.28.2-1-i686.djw.delta.9.xz
1520    evolution-2.28.1-1_to_2.28.2-1-i686.djw.delta.xz
1540    evolution-2.28.1-1_to_2.28.2-1-i686.djw.delta.gz
1548    evolution-2.28.1-1_to_2.28.2-1-i686.delta.bz2
1552    evolution-2.28.1-1_to_2.28.2-1-i686.djw.delta.bz2
1564    evolution-2.28.1-1_to_2.28.2-1-i686.djw.delta
1684    evolution-2.28.1-1_to_2.28.2-1-i686.0.delta.gz
1712    evolution-2.28.1-1_to_2.28.2-1-i686.0.delta.bz2
1772    evolution-2.28.1-1_to_2.28.2-1-i686.delta
2832    evolution-2.28.1-1_to_2.28.2-1-i686.0.delta
28944   evolution-2.28.1-1-i686.pkg.tar.gz
28952   evolution-2.28.2-1-i686.pkg.tar.gz

NOTE: If you're wondering why I'm not using bsdiff even though it has the best compression, there's a few reasons:
1) pacman would need to be patched to support it (not difficult...)
2) bsdiff files don't contain metadata about the source/target files, so they would need to be packaged with some sort of metadata file. (again not difficult but requires repo-add to be modified)
3) bsdiff is much slower than xdelta to generate diffs, and requires INSANE amounts of ram (up to 17x the size of destination file)

That said.. if bsdiff ever gets supported by pacman.. Then it might not be a bad idea to use bsdiff for packages under a specific size.

Questions, todo, feedback, etc..:
I'll add notes to this section as they come..

I'll start with my question(s):
Is there an official archlinux repository that I should be syncing to? If so.. who do I need to talk to about that?

edit: If you are having xdelta3 errors, try the 3.0v-2 version of xdelta, I've posted a copy of the package at archdelta.net
http://archdelta.net/xdelta3-3.0v-2-i686.pkg.tar.gz

Last edited by sabooky (2010-04-02 00:38:10)

neofish · 2010-02-27 02:52:11

with delta:
:: Starting full system upgrade...
resolving dependencies...

Targets (10): avahi-0.6.25-2 dcron-4.4-2 e2fsprogs-1.41.10-1
heimdal-1.3.1-3 iputils-20100214-2 sdl_mixer-1.2.11-2
smbclient-3.4.6-1 squashfs-tools-4.0-3 syslog-ng-3.0.4-3
thunderbird-3.0.2-1

Total Download Size: 13.00 MB
Total Installed Size: 102.17 MB

and without:

:: Starting full system upgrade...
resolving dependencies...

Targets (10): avahi-0.6.25-2 dcron-4.4-2 e2fsprogs-1.41.10-1
heimdal-1.3.1-3 iputils-20100214-2 sdl_mixer-1.2.11-2
smbclient-3.4.6-1 squashfs-tools-4.0-3 syslog-ng-3.0.4-3
thunderbird-3.0.2-1

Total Download Size: 29.78 MB
Total Installed Size: 102.17 MB

Allan · 2010-02-27 02:59:36

sabooky wrote:

Is there an official archlinux repository that I should be syncing to? If so.. who do I need to talk to about that?

There is... but any good mirror should be fine. We encourage new mirrors to sync of another local mirror these days anyway.

Something that might also interest you:
http://code.toofishes.net/cgit/xavier/p … d=e46bb09f

Edit: good work BTW! This is one thing I want to get officially implemented but there are a couple of repo management things that need improved with a higher priority.

james · 2010-02-28 20:15:40

Great work!

My link is 32kb/s at most so this is exactly what I was hoping for.... Thank you.

with pacman -Syu I get this:

Proceed with installation? [Y/n] 
:: Retrieving packages from extra...
 cdrdao-1.2.3-2_to_1...     0.6K  433.9K/s 00:00:00 [################################################################] 100%
 unzip-6.0-4_to_6.0-...    41.1K   49.6K/s 00:00:01 [################################################################] 100%
checking delta integrity...                                                                                                
applying deltas...                                                                                                         
generating bind-9.6.1.P3-2-i686.pkg.tar.xz with bind-9.6.1.P3-1_to_9.6.1.P3-2-i686.delta... xdelta3: seek to 0 failed: /var/cache/pacman/pkg/bind-9.6.1.P3-1-i686.pkg.tar.gz: Illegal seek                                                            
xdelta3: non-seekable source: copy is too far back (try raising -B): XD3_INTERNAL                                          
failed.

Can I set this -B option somewhere?

James

Update:
I Changed to xdelta3.0v2.
I get 'success!' generating the packages (no 'success!' with zlib delta).
The packages seem to be corrupted:

Proceed with download? [Y/n] 
:: Retrieving packages from core...
error: failed retrieving file 'zlib-1.2.3.9-1-i686.pkg.tar.xz' from archdelta.net : Not Found
 zlib-1.2.3.9-1-i686...    90.9K   37.2K/s 00:00:02 [################################################################] 100%
checking package integrity...                                                                                              
:: File boost-1.42.0-1-i686.pkg.tar.gz is corrupted. Do you want to delete it? [Y/n] n                                     
:: File openssl-0.9.8m-1-i686.pkg.tar.xz is corrupted. Do you want to delete it? [Y/n] n                                   
:: File libmysqlclient-5.1.44-1-i686.pkg.tar.xz is corrupted. Do you want to delete it? [Y/n] n                            
:: File mysql-clients-5.1.44-1-i686.pkg.tar.xz is corrupted. Do you want to delete it? [Y/n] n

etc.

I looked at both 'filesystem-2010.02-4-any.pkg.tar.xz' on a server and in the local cache.
My local copy is not compressed. ('cat [file from server] | xz -d > [another file] ' creates the same as the file generated by pacman.)

anybody with same problem?

Monologue part 3:
I was able to use all the xz files by renaming them to *.tar and running xz -z on them.
gz files still corrupted with both gzip -9 and python gzip

I used this for the gz files:

import gzip
import sys

f_in = open(sys.argv[1], 'rb')
f_out = gzip.open(sys.argv[2], 'wb')
f_out.writelines(f_in)
f_out.close()
f_in.close()

I hope there will soon be official deltas in the repositories.
For now I'll do

#UseDelta
IgnorePkg   = go-openoffice

good night

Last edited by james (2010-03-05 23:18:41)

Pierre · 2010-02-28 20:46:15

xdelta is definitely something we want to add to our repos in future. But as Allan stated we need to add other improvements ot our repository/mirror structure first. (xdelta will increase load/disk space and traffic on our main server and between mirrors).

Any input and experience on this are welcome. Even more if you are trying to use "our" tools like repo-add (part of pacman) or devtools/dbscripts: http://projects.archlinux.org

cookiecaper · 2010-03-03 20:37:52

This is cool, great work. Can we get one for x86_64 now?

Runiq · 2010-03-03 20:53:16

sabooky wrote:

(…) bsdiff (…)

While looking up version control stuff and binary diffing for my thesis, I found out about bdiff which is supposedly faster to create diffs and easier on memory than bsdiff. That being said, I haven't tried it yet,and I didn't even know about xdelta at the time…

Last edited by Runiq (2010-03-03 20:55:12)

james · 2010-03-04 13:17:19

zsync is nice too. not as efficient but very nice to server space. very easy to handle. no deltas at all.
with some effort it might even get as nice as courgette or even nicer

http://zsync.moria.org.uk/
http://www.chromium.org/developers/desi … -courgette

james

pie86 · 2010-03-13 00:17:59

cookiecaper wrote:

This is cool, great work. Can we get one for x86_64 now?

I'd like it too... Can I lend a hand?

sabooky · 2010-03-13 03:17:43

@Allan
The unused deltas feature looks nice, once its in the stable release I'll take a look at it. I'm not sure the issue will come up in my case though, since the script that generates deltas also cleans up old ones.

@james
For the corrupt gzip packages try using the '-n' flag when compressing. gzip by default stores timestamps.. so if they change, it changes the md5 hash of the package.. weird that your cache files are uncompressed.. no idea why thats happening (maybe a pacman option?)

google-Courgette looks very interesting.. Hope they release it soon.

@Pierre
How are patches sent in? just use git format-patch and email it?

@cookiecaper and pie86
The only reason there isn't an x86_64 repo is because I don't have the hd space (locally) to do it.

As for helping out.. hmm.. let me explain how the scripts work on my end.
This way, anyone that wants to help out can have an idea of what it takes to host this.

Warning: this is messy and all over the place.. but it works, and thats all that matters :P

I have a script that drives all of this sync.sh, when its run it does the following:
1) uses reflector to get the top 10 fastest mirrors that synced within 2 hours.
2) wget the repo.db.tar.gz file
3) generates a metalinker file out of repo.db.tar.gz that uses aria2c to download
* this speeds up downloads by using 10 mirrors and takes care of md5sum validation on the downloaded packages.
* that reminds me.. vimpager package is broken, repo.db.tar.gz contains a different md5 and package size.
If there are changes, then the following steps are also done:
4) deltify.py is run, which creates deltas, removes old/oversized deltas, removes old packages.
5) cp repo.db.tar.gz from repo dir to deltas dir
6) repo-add $repo.db.tar.gz *.delta
* I've modified my local copy of repo-add to improve performance.
* it generates a cache file, so it doesn't have to get the md5, old_ver, new_ver for previously processed deltas.
7) lftp pushes changes to archdelta.net

Here's HD usage locally on my box:
# du -hcs repos/*/os/i686
7.3G repos/community/os/i686
270M repos/core/os/i686
11G repos/extra/os/i686
18G total

# du -hcs repos/*/os/i686/deltas
217M repos/community/os/i686/deltas
49M repos/core/os/i686/deltas
2.2G repos/extra/os/i686/deltas
2.5G total

Notes:
* feedback on whether there's a cleaner way to do this would be nice.

* I wrote a repo2ml.py which generates metalink file out of repository, but am not using it currently.
Need to test and use this rather than using bash to generate the metalink.
Need to post this, since this could be a pretty useful standalone tool for some.

* Where do I file a bug on vimpager, the community.db.tar.gz md5sum and size do not match whats in the repository.

pie86 · 2010-03-15 20:43:36

@sabooky
I don't know if you received my email, anyway I think I could have a machine that does the syncing job for 64 bit packets.
If you can host also 64 bit pkgs on archdelta.net I would be glad to help "completing" your repository

toofishes · 2010-03-24 03:16:04

If you wouldn't mind some time, could you send your "thoughts and experiences" type things to the pacman-dev mailing list? It would be great to hear how this stuff is working out in the wild and what we need to do to get it more widely used and live, especially before we commit to design decisions that you may be finding no-good as you roll this out.

schuay · 2010-03-24 08:20:31

sabooky, thanks for pushing this new feature

cool · 2010-03-24 19:40:05

Hi sabooky,
Thanks for this. delta would save lot of time and bandwidth for me.my internet connect is just 256kbps. so saving every mb of download translated in huge savings.

Thanks a Ton! you deserve a beer, I would buy you one :-)

Ramses de Norre · 2010-03-25 11:51:08

How often is your mirror being synced? I tried it but I got some 50 warnings about packages on my system being newer than core.. My usual mirror is mir.archlinux.fr

Thanks for putting effort into this anyhow, this might get something bigger started

sabooky · 2010-03-25 14:04:44

@ramses

oops.. I made a change to my dreamhost.net account, and didn't realize I manged to accidentally change my account login credentials.

The repository hasn't been synced since 17-Mar-2010 due to script failing to push updates because of a Login error.

I fixed the login information and forced a sync just now, the repository should be up to date.

The repository will continue to sync on the regularly scheduled time (3:30am est).

Sorry for the inconvenience.

Thanks for bringing this up to my attention.

Note: Lesson of the day... don't let scripts silently die, have them send emails... I'll make this change sometime this weekend.

lucke · 2010-03-25 21:45:54

Mmm, I like the idea of delta files very much. Thanks!

Would it be possible for you to support testing and community-testing repos? deltas seems to fit those especially nice, as they're often just some rebuilds or minor PKGBUILD changes.

sabooky · 2010-03-26 02:53:40

I added the testing and community-testing repos. However, I didn't use the ARM to get previous versions, so the repo will start out with 0 deltas and will grow as time goes on.

There's a chance I might add x86_64 support soon, I just need to move the storage off to my freebsd server machine (it has more storage space). If i do a x86_64 repo, I'll probably do the same thing that I did today, basically start with a repo with 0 deltas and let it grow naturally.

I'll post my "thoughts and experiences" to the pacman-dev. Though I'm not sure how much of what I've done here applies to the arch developers workflow.

There are two things I'd like to see implemented:
1. have delta files wrapped in a tar.gz file containing a metadata file, something like .PKGINFO (.DELTAINFO?). This makes it so the delta system isn't dependent on the xdelta3 format.
2. support multiple delta formats, even if this isn't done initially having #1 done will allow for this to be added later.

That said, I would be interested in contributing to the development of the delta support, I'm guessing db-update is where this would need to be added.

Is there a design doc already written for the planned implementation of this? What's the process for submitting patches for the arch linux projects (pacman, dbscripts, etc..)?

Thanks

Ramses de Norre · 2010-03-26 10:44:01

Ok I pulled the first deltas in today However, I get this:

:: Starting full system upgrade...
warning: gsfonts: local (8.11-5) is newer than extra (1.0.7pre44-1)
resolving dependencies...
looking for inter-conflicts...

Targets (4): libcups-1.4.2-5 [0.25 MB]  cups-1.4.2-5 [1.75 MB]  openssl-0.9.8n-1 [1.96 MB]  
             php-suhosin-0.9.30-1 [0.06 MB]  

Total Download Size:    0.00 MB
Total Installed Size:   21.78 MB

Proceed with installation? [Y/n] y
checking delta integrity...
applying deltas...
generating libcups-1.4.2-5-i686.pkg.tar.xz with libcups-1.4.2-4_to_1.4.2-5-i686.delta... success!
generating cups-1.4.2-5-i686.pkg.tar.xz with cups-1.4.2-4_to_1.4.2-5-i686.delta... xdelta3: seek to 6291456 failed: /var/cache/pacman/pkg/cups-1.4.2-4-i686.pkg.tar.xz: Illegal seek
success!
generating openssl-0.9.8n-1-i686.pkg.tar.xz with openssl-0.9.8m-2_to_0.9.8n-1-i686.delta... xdelta3: seek to 6291456 failed: /var/cache/pacman/pkg/openssl-0.9.8m-2-i686.pkg.tar.xz: Illegal seek
success!
generating php-suhosin-0.9.30-1-i686.pkg.tar.xz with php-suhosin-0.9.29-1_to_0.9.30-1-i686.delta... success!
checking package integrity...
(4/4) checking for file conflicts                   [---------------------------------------------------] 100%
(1/4) upgrading libcups                             [---------------------------------------------------] 100%
(2/4) upgrading cups                                [---------------------------------------------------] 100%
(3/4) upgrading openssl                             [---------------------------------------------------] 100%
(4/4) upgrading php-suhosin                         [---------------------------------------------------] 100%

Are the xdelta errors harmful?

lucke · 2010-03-26 12:50:54

Thanks for those repos, sabooky.

If we're talking about errors, I get (using xdelta3 3.0y)

generating wine-1.1.41-1-i686.pkg.tar.xz with wine-1.1.40-1_to_1.1.41-1-i686.delta... xdelta3: non-seekable source in decode: XD3_INTERNAL
failed.

Last edited by lucke (2010-03-26 12:52:01)

gohu · 2010-03-26 18:25:35

To sabooky:
I have sent you two emails asking if you would be ok with releasing some of the code you did for the repo. I didn't have any answer.
If you're not willing to disclose your work, that's ok but could you tell me so? I don't know if you ever received the mails I sent.
Thank you and sorry for disrupting the thread.

sabooky · 2010-03-26 20:28:24

@gohu

Sorry, the email account associated with my login here is my spam email account. It didn't used to be, but unfortunately someone decided to cite my email address in their script and since then it basically had to be retired to spam. So the easiest way to get in touch with me is through this thread.

I checked my email and saw your messages, I'll make an effort to post the scripts either on here, or maybe through github this weekend. (no promises, other than I will try)

I'm all for disclosing the work, hell.. it'd be great if someone can improve on it.

@lucke and @ramses
http://xdelta.org/ October 25th entry mentions some information on non-seekable sources, not sure why its happening though.

ramses, yours seems to be just a warning.

lucke, could you please post the md5sum on both wine-1.1.40-1_to_1.1.41-1-i686.delta and the wine-1.1.40*.pkg.tar.[xg]z file.
I would like to test generating the file using the delta manually.

You could try it yourself, the command is in the code snippit below. The -B flag might be helpful, if we can get the xdelta to behave properly, then it's just a matter of patching the pacman code.

Also.. could you give it a shot with xdelta3 3.0v? (if you need the tarball let me know and I'll post it for you).

pacman devs
Hmm.. I wonder if this move to xz is gonna break with the following logic in the pacman sourcecode
./lib/libalpm/sync.c

      if(endswith(to, ".gz")) {
        /* special handling for gzip : we disable timestamp with -n option */
        snprintf(command, PATH_MAX, "xdelta3 -d -q -R -c -s %s %s | gzip -n > %s", from, delta, to);
      } else {
        snprintf(command, PATH_MAX, "xdelta3 -d -q -s %s %s %s", from, delta, to);
      }

The following might need to be added.

else if (endswith(to,".xz")) {
snprintf(command, PATH_MAX, "xdelta3 -d -q -R -c -s %s %s | xz > %s", from, delta, to);
}

I'm at work right now so I can't look too much into it.. but this is definitely an interesting problem.. and I really hope that this xz change doesn't completely break archdelta.net for the time being :\.
I would really hate having to use a modified pacman to get archdelta to work.

Thanks for the feedback

lucke · 2010-03-26 22:34:53

3.0v seems to work.

That code seems to work - the only "problem" is that

xdelta3 -d -vv -R -c -s wine-1.1.40-1-i686.pkg.tar.gz wine-1.1.40-1_to_1.1.41-1-i686.delta | gzip -n > wine-1.1.41-1-i686.pkg.tar.gz

takes 10 seconds for me and

xdelta3 -d -vv -s wine-1.1.40-1-i686.pkg.tar.gz wine-1.1.40-1_to_1.1.41-1-i686.delta wine-1.1.41-1-i686.pkg.tar.xz

takes 91 seconds. Creating xz-compressed files using deltas (i.e. locally) doesn't seem to be a very good idea - unless someone really values their disk space (7 MB of difference between xz and gz for wine). I'm not sure if it's currently possible to control the resultant compression type of packages after applying xdelta3 - I don't quite grok that code - but such a choice should surely be presented to the user.

Allan · 2010-03-26 22:59:13

3.0v from upstream required patching to deal with xz faster. Is the time difference as great with 3.0y?

@sabooky: any chance you can send the query to the pacman-dev mailing list? If you do not want to subscribe, I can send it for you.

lucke · 2010-03-26 23:28:29

This time is generally the time it takes to compress a tar file with gz and xz, it seems. xdelta3 itself adds but a few seconds.

I see no performance-related patch in the svn - just a patch adding xz support to w; and I've indeed built v by just changing the version in w's PKGBUILD.

Arch Linux

#1 2010-02-27 01:47:32

http://archdelta.net/: An Archlinux delta repository (i686 only)

#2 2010-02-27 02:52:11

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#3 2010-02-27 02:59:36

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#4 2010-02-28 20:15:40

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#5 2010-02-28 20:46:15

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#6 2010-03-03 20:37:52

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#7 2010-03-03 20:53:16

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#8 2010-03-04 13:17:19

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#9 2010-03-13 00:17:59

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#10 2010-03-13 03:17:43

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#11 2010-03-15 20:43:36

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#12 2010-03-24 03:16:04

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#13 2010-03-24 08:20:31

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#14 2010-03-24 19:40:05

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#15 2010-03-25 11:51:08

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#16 2010-03-25 14:04:44

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#17 2010-03-25 21:45:54

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#18 2010-03-26 02:53:40

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#19 2010-03-26 10:44:01

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#20 2010-03-26 12:50:54

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#21 2010-03-26 18:25:35

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#22 2010-03-26 20:28:24

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#23 2010-03-26 22:34:53

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#24 2010-03-26 22:59:13

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

#25 2010-03-26 23:28:29

Re: http://archdelta.net/: An Archlinux delta repository (i686 only)

Board footer