You are not logged in.

#1 2006-10-01 22:57:10

apeiro
Daddy
From: Victoria, BC, Canada
Registered: 2002-08-12
Posts: 771
Website

Hack: downloading packages with metalink

I'm sure some of you have noticed the new .metalink files in the Download area on archlinux.org.  These things are basically just XML files that point to all the mirrors we have that host the ISOs.  When you download using one of these metalinks (using a metalink-capable downloader like aria2), you download chunks from each mirror, somewhat similar to how BitTorrent operates.  Metalinks are handy because you can use HTTP, FTP, or BitTorrent as mirror entries in them, and you can prioritize the mirrors with a "preference" metric.

Anyway, I was talking to the metalink creator about getting metalink support into pacman eventually.  For those of you who've noticed the glacial pace of pacman3 development recently, you probably know that "eventually" could mean "when I'm retired and have the time".

So instead of doing it properly, I wrote a little python script that basically does that.  It runs a 'pacman -Sup', grabs all the package filenames, and generate a .metalink file for each them.  It ignores files that are smaller than 5mb since those already download pretty quickly for me.

It's a quick little one-off hack, but after noticing the crazy download speeds I was getting with the ISO .metalink files, I had to try it out.

If you want to check it out, the script and metalink template are here:
http://www.archlinux.org/~judd/pacman-metalink/

Download both files, and use like so:

# pacman -Sy
# ./pacman-metalink.py
# pacman -Su

Quite untested, and the python code is ugly, cuz I'm no good at python.  Feel free to clean up and improve if it works for you.  Also, you can tweak the list of mirrors and their preferences in teh template.metalink file -- it may improve performance.  And you need to have aria2 installed.  It's in Community.

- J

Offline

#2 2006-10-01 23:00:57

cactus
Taco Eater
From: t͈̫̹ͨa͖͕͎̱͈ͨ͆ć̥̖̝o̫̫̼s͈̭̱̞͍̃!̰
Registered: 2004-05-25
Posts: 4,622
Website

Re: Hack: downloading packages with metalink

with the exception of your use of tabs instead of spaces..it looks fine to me.
wink


"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍

Offline

#3 2006-10-01 23:01:28

Romashka
Forum Fellow
Registered: 2005-12-07
Posts: 1,054

Re: Hack: downloading packages with metalink

Cool! I've never heard about Metalink standard before.

EDIT: Ah, you surpassed me, Cactus!  smile

BTW,

We have 7502 registered users

roll


to live is to die

Offline

#4 2006-10-01 23:03:09

apeiro
Daddy
From: Victoria, BC, Canada
Registered: 2002-08-12
Posts: 771
Website

Re: Hack: downloading packages with metalink

Sorry, forgot the metalinker URL.

http://www.metalinker.org/

Offline

#5 2006-10-01 23:05:12

Romashka
Forum Fellow
Registered: 2005-12-07
Posts: 1,054

Re: Hack: downloading packages with metalink

apeiro wrote:

Sorry, forgot the metalinker URL.

http://www.metalinker.org/

I've already extracted it from http://www.archlinux.org/~judd/pacman-m … e.metalink  big_smile


to live is to die

Offline

#6 2006-10-19 06:39:04

twanj
Member
From: Pompano Beach, FL
Registered: 2006-08-02
Posts: 47
Website

Re: Hack: downloading packages with metalink

This is really cool! I had no idea.

You could also generate the metalinks with Simba on the mirrors (automated, probably the easiest) or with metalink tools.


Simpler/Faster downloads with error recovery - http://www.metalinker.org/

Offline

#7 2006-10-19 17:57:46

phrakture
Arch Overlord
From: behind you
Registered: 2003-10-29
Posts: 7,879
Website

Re: Hack: downloading packages with metalink

Just for a tad more information.... at some point pacman will support package downloading via metalink.  This may be a decent amount of work, and I'm more concerned with other things right now 8)

Offline

#8 2006-10-19 21:23:21

twanj
Member
From: Pompano Beach, FL
Registered: 2006-08-02
Posts: 47
Website

Re: Hack: downloading packages with metalink

metalinks for all the packages could be generated with RoPkg::Metalink (perl) once on the server instead of each time on the client. When new packages are uploaded it would make new metalinks automatically. Then maybe aria downloading could be added right into pacman.


Simpler/Faster downloads with error recovery - http://www.metalinker.org/

Offline

#9 2006-11-08 20:09:17

twanj
Member
From: Pompano Beach, FL
Registered: 2006-08-02
Posts: 47
Website

Re: Hack: downloading packages with metalink

The new version of aria2 (0.9.0) allows downloading from metalinks that list multiple files, and downloading by location (--metalink-location=LOCATION, where LOCATION is your 2 letter country code. Since this may use less mirrors, it may be slower. On the other hand, if your country has more and faster mirrors, it may be better).

ps - new metalink article "Downloading bliss with Metalink" at http://www.linux.com/article.pl?sid=06/11/01/1641247


Simpler/Faster downloads with error recovery - http://www.metalinker.org/

Offline

#10 2007-01-29 20:52:08

twanj
Member
From: Pompano Beach, FL
Registered: 2006-08-02
Posts: 47
Website

Re: Hack: downloading packages with metalink

The newest version of aria2 (0.10.0+1) supports chunk checksums for ftp/http transfers:

"As of 0.10.0 aria2 supports chunk checksum in Metalink: it validates chunk checksums while downloading a file like BitTorrent. This gives safer and more reliable download experience."

When using these, downloads will automatically be corrected if there are errors in transfer. This means you don't have to wait until the file finishes to find out the checksum is bad, then re-download the whole thing. Should save time and bandwidth.

metalink tools can make these chunk checksums with the -d sha1pieces option.


Simpler/Faster downloads with error recovery - http://www.metalinker.org/

Offline

#11 2007-02-02 16:39:40

codemac
Member
From: Cliche Tech Place
Registered: 2005-05-13
Posts: 790
Website

Re: Hack: downloading packages with metalink

and aria2 has been updated in [extra]

Offline

#12 2007-02-02 17:31:34

twanj
Member
From: Pompano Beach, FL
Registered: 2006-08-02
Posts: 47
Website

Re: Hack: downloading packages with metalink

codemac wrote:

and aria2 has been updated in [extra]

great! I posted these updated ones with chunk checksums for the ISOs in another thread.

"As of 0.10.0 aria2 supports chunk checksum in Metalink: it validates chunk checksums while downloading a file like BitTorrent. This gives safer and more reliable download experience."

http://www.metalinker.org/test/arch-0.7 … o.metalink
http://www.metalinker.org/test/arch-0.7 … o.metalink
http://www.metalinker.org/test/arch-0.7.2.iso.metalink

http://www.metalinker.org/test/arch64-0 … o.metalink
http://www.metalinker.org/test/arch64-0 … o.metalink
http://www.metalinker.org/test/arch64-0 … o.metalink


Simpler/Faster downloads with error recovery - http://www.metalinker.org/

Offline

#13 2007-04-14 00:16:53

twanj
Member
From: Pompano Beach, FL
Registered: 2006-08-02
Posts: 47
Website

Re: Hack: downloading packages with metalink

phrakture wrote:

Just for a tad more information.... at some point pacman will support package downloading via metalink.  This may be a decent amount of work, and I'm more concerned with other things right now 8)

Is metalink support still a possibility for pacman 3.x?

aria2 continues to improve and have more features.

Arch 0.8 ISO .metalinks are at
http://www.metalinker.org/samples.html


Simpler/Faster downloads with error recovery - http://www.metalinker.org/

Offline

#14 2007-05-07 19:27:57

Romashka
Forum Fellow
Registered: 2005-12-07
Posts: 1,054

Re: Hack: downloading packages with metalink

twanj wrote:
phrakture wrote:

Just for a tad more information.... at some point pacman will support package downloading via metalink.  This may be a decent amount of work, and I'm more concerned with other things right now 8)

Is metalink support still a possibility for pacman 3.x?

Isn't it possible even with pacman 2.x using XfrerCommand = aria2 blah-blah-blah ?


to live is to die

Offline

#15 2007-05-07 21:16:38

phrakture
Arch Overlord
From: behind you
Registered: 2003-10-29
Posts: 7,879
Website

Re: Hack: downloading packages with metalink

Yes, it is possible with aria2.  The only "issue" with integrating metalink into pacman is that
a) there is no general "libmetalink" available, so everything would need to be written from scratch / cannibalized from other projects (aria2 is C++, pacman is C)
b) xml parsing would need to be integrated into pacman as well, which is a big undertaking that I'm not to sure I'd like to take at this point in time.

Offline

#16 2007-05-08 19:56:10

twanj
Member
From: Pompano Beach, FL
Registered: 2006-08-02
Posts: 47
Website

Re: Hack: downloading packages with metalink

phrakture wrote:

Yes, it is possible with aria2.  The only "issue" with integrating metalink into pacman is that
a) there is no general "libmetalink" available, so everything would need to be written from scratch / cannibalized from other projects (aria2 is C++, pacman is C)
b) xml parsing would need to be integrated into pacman as well, which is a big undertaking that I'm not to sure I'd like to take at this point in time.

Would it be possible to keep pacman ignorant of the XML (leave that to aria2?) and have .metalinks created when the package is uploaded? If people have opted to use aria2 for transfer, then instead of packagefilename it could request packagefilename.metalink

The next version of aria2 (not out yet) will be able to to download multiple files simultaneously. AFAIK other update tools like apt-get and yum don't support this. It could end up being a big improvement. I'm imagining a .metalink listing all updates getting everything really fast.


Simpler/Faster downloads with error recovery - http://www.metalinker.org/

Offline

#17 2007-05-08 23:56:01

phrakture
Arch Overlord
From: behind you
Registered: 2003-10-29
Posts: 7,879
Website

Re: Hack: downloading packages with metalink

Yeah I mean it's definitely possible to integrate metalinks into the *process* while requiring an external command to use them.  It might even be ideal, as pacman's XferCommand can help get rid of some of that complexity (this is how the Xdelta stuff is being done too).

Generating metalinks on the serverside for use with aria2 should be pretty easy, I'll look into it.

Offline

#18 2007-05-09 18:22:30

twanj
Member
From: Pompano Beach, FL
Registered: 2006-08-02
Posts: 47
Website

Re: Hack: downloading packages with metalink

phrakture wrote:

Yeah I mean it's definitely possible to integrate metalinks into the *process* while requiring an external command to use them.  It might even be ideal, as pacman's XferCommand can help get rid of some of that complexity (this is how the Xdelta stuff is being done too).

Generating metalinks on the serverside for use with aria2 should be pretty easy, I'll look into it.

Cool! That sounds like the simplest way to do it. aria2 is good at what it does, since it can be used, that's great. Xdelta is really nice too. With all this, Arch will have the fastest, most efficient updates.

This article talks about some programs for making metalinks. They may range from too complex to too simple, or not custom enough for you tho.


Simpler/Faster downloads with error recovery - http://www.metalinker.org/

Offline

#19 2007-06-14 04:49:18

twanj
Member
From: Pompano Beach, FL
Registered: 2006-08-02
Posts: 47
Website

Re: Hack: downloading packages with metalink

aria2 0.11.0+ now has the "ability to download multiple files  concurrently. * Now downloads multiple files concurrently. See -j option."

could this, along with the delta/diff packages could be the next evolution in package updates? it should at least be a nice improvement.


Simpler/Faster downloads with error recovery - http://www.metalinker.org/

Offline

#20 2007-07-18 18:10:12

twanj
Member
From: Pompano Beach, FL
Registered: 2006-08-02
Posts: 47
Website

Re: Hack: downloading packages with metalink

There's a new version of aria2 out. Would it be possible to update this script to use -j?


Simpler/Faster downloads with error recovery - http://www.metalinker.org/

Offline

#21 2007-09-12 04:56:41

twanj
Member
From: Pompano Beach, FL
Registered: 2006-08-02
Posts: 47
Website

Re: Hack: downloading packages with metalink

phrakture wrote:

Yeah I mean it's definitely possible to integrate metalinks into the *process* while requiring an external command to use them.  It might even be ideal, as pacman's XferCommand can help get rid of some of that complexity (this is how the Xdelta stuff is being done too).

Generating metalinks on the serverside for use with aria2 should be pretty easy, I'll look into it.

I'm putting up a $250 bounty for this feature request, if its something that fits in & makes sense.


Simpler/Faster downloads with error recovery - http://www.metalinker.org/

Offline

#22 2008-06-13 18:55:28

twanj
Member
From: Pompano Beach, FL
Registered: 2006-08-02
Posts: 47
Website

Re: Hack: downloading packages with metalink

phrakture wrote:

Yes, it is possible with aria2.  The only "issue" with integrating metalink into pacman is that
a) there is no general "libmetalink" available, so everything would need to be written from scratch / cannibalized from other projects (aria2 is C++, pacman is C)
b) xml parsing would need to be integrated into pacman as well, which is a big undertaking that I'm not to sure I'd like to take at this point in time.

there is now a libmetalink (C) available. among other things, they plan to use it for libzypp (opensuse):

After a lot of discussions about "how" to implement the goal of the project ("Libzypp Download Failover") via zypp-devel mailing list and #zypp channel at irc.freenode.net, we decided than:

* Libzypp will use "only" metalink files (and not Dinamic mirror lists).

* We can use libmetalink (a metalink parser library written in C by Tatsuhiro) to parse metalink files. (We think than this will be better than use external programs like aria2c).

* To download files, we can use, like right now, libcurl.

And… To devel metalink support within libzypp will be my goal in next weeks :-)


Simpler/Faster downloads with error recovery - http://www.metalinker.org/

Offline

#23 2008-06-16 03:20:48

sabooky
Member
Registered: 2006-11-02
Posts: 89

Re: Hack: downloading packages with metalink

What features does metalink offer over pacget? Does it handle broken downloads better? Is it a smoother process overall? Just wondering.. Also, what happened to that ISO metalink mirrors? sad

Offline

#24 2008-06-16 17:21:20

twanj
Member
From: Pompano Beach, FL
Registered: 2006-08-02
Posts: 47
Website

Re: Hack: downloading packages with metalink

sabooky wrote:

What features does metalink offer over pacget? Does it handle broken downloads better? Is it a smoother process overall? Just wondering.. Also, what happened to that ISO metalink mirrors? sad

it's probably similar to using aria2 as an external download program and using multiple mirrors.

metalinks can have chunk checksums which let downloads be repaired. not very useful for small files unless you're on a really slow connection, but could save some time if there are errors on a large download.

it may be worth it to wait & see what happens w/ the libzypp stuff.


Simpler/Faster downloads with error recovery - http://www.metalinker.org/

Offline

Board footer

Powered by FluxBB