pacget: Super fast pacman downloads

sabooky · 2006-11-02 04:19:45

I spent all day (yes I'm a slow horrible programmer) writing this scrip. It takes the download url from pacman, then looks up the mirror list in /etc/pacman.d/<repo> and adds them all as mirrors to aria2.

What ends up happening is aria2c connects to 10-20 servers downloading from all of them at the same time, which should give anyone on broadband a huge boost in download speed. This boosted my download speeds from 100-300KB/s to 2,000KB/s (my connections limit).

http://wiki.archlinux.org/index.php/Fas … ror_Script

Hope you guys like it.

ps: would making a package out of this be overkill?, like one that auto installs everything?

drakosha · 2006-11-02 07:06:43

Great idea, but i'm not sure it's a good idea - it's increasing load on mirrors. 100KB/sec = 30M per minute - it's a great speed already

sabooky · 2006-11-02 07:13:04

I was wondering about that myself while I was making it. The thing I'm wondering is, like for example:

before pacget: Downloading from first mirror for 2 min
post pacget : Downloading from 5-10 mirrors for 10-20 seconds.

^^^ thats sounds good on paper but in reality it might be bad on the servers. If any server admin complains about it I'll gladly take it off.

Mr Green · 2006-11-02 07:57:06

Metalink? maybe.....

The man to ask is Judd .... if he say yes then..... go for it

smoon · 2006-11-02 09:59:35

IIRC Judd already posted a similar script that creates metalink-files to use with aria2 either to the bbs or to one of the mailinglists.
Nice idea, though.

Aha, there it is: http://bbs.archlinux.org/viewtopic.php?t=25611

filoktetes · 2006-11-02 12:45:02

I can't understand why this should be bad for the servers. As I understand, this divides the load from one server to many. So instead of downloading at full speed for 2 minutes from one server, you download at full speed from several servers for 20 seconds.
As far as I see this can only be a good thing. Or is there something I don't see?

phrakture · 2006-11-02 15:25:09

For the record, at some point down the line (don't quote me on this) pacman3 will support segmented downloading and use the top X mirrors that are configured. 8)

Dusty · 2006-11-02 15:34:42

phrakture wrote:

(don't quote me on this)

phrakture never said that! :-D

Dusty

sabooky · 2006-11-02 18:03:10

phrakture wrote:

For the record, at some point down the line (don't quote me on this) pacman3 will support segmented downloading and use the top X mirrors that are configured. 8)

Any idea when pacman3 comes out? And yea segmented downloading would be cool to have by default.

smoon wrote:

IIRC Judd already posted a similar script that creates metalink-files to use with aria2 either to the bbs or to one of the mailinglists.
Nice idea, though.
Aha, there it is: http://bbs.archlinux.org/viewtopic.php?t=25611

Hmm that script kinda does the same thing, just a bit different. Kinda wish I had known about it before doing this. It seems that it only makes metalinks for all the packages and doesn't actually download them with pacman (tho I could be wrong). I really like how for files under 5mb it just downloads from one mirror. I'm thinking of doing something like it with my script. maybe 2 mirrors for under 5, 5-10 use 5 mirrors, 10+ don't limit.

Mr Green wrote:

Metalink? maybe.....
The man to ask is Judd .... if he say yes then..... go for it

Ask him what?

PS: Did anyone actually try my script. I'm not exactly 100% sure its bug free at this point, would be nice to get some feedback in that regard. Also I'm half the script I only learned yesterday so I'm about 99% sure that there's prolly better ways to do certain things in the script, feel free to point out modifications/improvements for me.

Xarturkhann · 2006-11-02 19:32:58

your script work fine..

mybe chance your log file directory root/ to var/log/?

also change log to pacget.log?

aria2c -t20 -m2 -l /var/log/pacget.log $mirrors -o $file.pacget

sabooky · 2006-11-03 14:03:26

Xarturkhann wrote:

your script work fine..
mybe chance your log file directory root/ to var/log/?
also change log to pacget.log?

Fixed. Thanks, that was actually there from when I was trying to debug a problem. So is the (dowloading from one mirror) thing, but I think I'll keep that there for now.

Hmm actually, are the echos annoying, Maybe i should take them out completly. I only had them there to kinda get an idea of whats going on with the program.

noriko · 2006-11-03 22:15:55

2000kb/s?? r u sure?

i used it, works perfectly btw but it's not enhancing anything lol.. my download is still still stuck at 240-5 kb/s ..

hmm.. might me an isp limit of some sorts....

phrakture · 2006-11-03 22:27:35

I get 1200K/s with a single mirror...

Simastrick · 2006-11-04 08:36:12

Why would this be bad on the servers? The load becomes shared between the mirrors, so... ?

Mr Green · 2006-11-04 16:45:14

Downloading: current.db.tar.gz
repo: current
(using one mirror)
41,992/72,174 Bytes 58% 04s 19.09 KB/s 1 connections                         
The download was complete. </current>

must be using it wrong ;-(

noriko · 2006-11-04 19:34:37

@mr.green what do you mean?

@everyone else...

is it safe to use multiple mirrors for the testing repo?
i just asked cos there's only one 'server' entry for it in pac~.conf and pac~.d list...

i simply copied the current repo, to ~.d/testing and replaced all occurrences of current with testing, and it seems to work fine...
still only get up to 250kb/s, but that's something else for me to look into...

Mr Green · 2006-11-04 19:59:02

XferCommand = /usr/bin/pacget %u %o

added this to pacman.conf as per wiki

one connection .... is that correct then?

just checking

noriko · 2006-11-04 21:11:06

yes, the script is written to use only one connection for database file downloads and resumes, and also the testing repo... which i changed.. check out the scrips source...

sabooky · 2006-11-05 02:23:29

noriko, I liked your idea with changing the testing repo. I made it so the script now tests if the /etc/pacman.d/repo file exists, if it does it'll just use it, otherwise it'll just do one mirror. So if you have an /etc/pacman.d/testing, it should now use it.

I mad a few revisions that I want to post up on here first to get some feedback on it before I change the wiki.

Whats changed is the repo check, and I made it so:
less then 5mb = 3 mirrors
less then 10mb = 7 mirrors
greater then 10mb = 30 mirrors
Also to clearify:
Any repo database file will only use one mirror (they are usually very small and not worth being split up).

Sorry for the messy script, I'll clean it up soon and add comments/allow easy modifications to it.

/usr/bin/pacget

#!/bin/bash
#pacget script

rm -f *.aria2
cachedir=/var/cache/pacman/pkg
file=$(awk -F '/' '{ print $NF }' <<<$1)
echo Downloading: $file|sed 's,/,,'
#exit 0
repo=$(awk -F '/' '{x=NF-3 } { print  $x }' <<<$1)
echo repo: $repo
#exit 0
function numir { 
size=$(sed -n '/^%CSIZE%$/{n;p}' /var/lib/pacman/$repo/$(sed 's/.pkg.tar.gz//'<<< $file)/desc)
if [[ $size -lt  5000000 ]]; then
      m=3
elif [[ $size -lt 10000000 ]]; then
      m=7
else
      m=30

fi
}
if [[ $2 = *.db.tar.gz.part || ! -f /etc/pacman.d/$repo  ]]; then
        echo "(using one mirror)"
        mirrors=$1
else
        if [ -f /etc/pacman.d/$repo ]; then
                numir
        mirrors=$(sed -n 's/^Server = //; s,$,/'"$file"', ; /^ftp/p; /^http/p' /etc/pacman.d/$repo|head -$m)
        fi
fi

aria2c -t20 -m2 -l /root/log $mirrors -o $file.pacget
if [ -f $cachedir/$file.pacget.aria2 ]; then
       rm $cachedir/$file.pacget $cachedir/$file.pacget.aria2
       exit 0
 fi

mv $file.pacget $2
exit 0

PS: Tell me if I should post this version up on the wiki.

PSS:

noriko wrote:

2000kb/s?? r u sure?
i used it, works perfectly btw Wink but it's not enhancing anything lol.. my download is still still stuck at 240-5 kb/s ..
hmm.. might me an isp limit of some sorts....

:: Retrieving packages from extra...
Downloading: openoffice-base-2.0.4-1.pkg.tar.gz
repo: extra
6,816,112/127,700,347 Bytes 5% 02m04s 2001.86 KB/s 24 connections

Mr Green · 2006-11-05 08:08:32

well does it work if you have user repo in pacman.d only I still get 1 mirror

LB06 · 2006-11-05 09:12:54

What if several mirrors are not yet in sync with the root ftp server? (which will happen a lot, since there are quite some mirrors who sync only once a day).

Anyway, I don't really need such a script, since my mirrors already gives me 100Mb/s, which is my theoretical max. The concept is great, though. And I do not believe it increases load on the mirrors. The the same amount of data (well, maybe there's a little bit of overhead) is being transferred, only in shorter times and at higher speeds.

sabooky · 2006-11-05 14:30:23

LB06 wrote:

What if several mirrors are not yet in sync with the root ftp server? (which will happen a lot, since there are quite some mirrors who sync only once a day).

If they aren't in sync then the file doesn't exist on that mirror yet, right? When that happens aria2c will prolly attempt to connect to that mirror then give up on it once it realizes the file doesn't exist. This issue will only cause problem in the files that are less then 10mb since the script is limiting the mirrors in that case, and if 2 out of 3 mirrors aren't synced well..then your only downloading from one mirror in that case. Quick fix would be to just use the old script which doesn't limit the number of mirrors, or you can try to make the top 7 servers or so ones that are usually updated frequently. Also make sure that your first mirror is ftp.archlinux.org, since thats what the script will use for your db files, thats one of the reasons why it only downloads repo files only from one mirror.

Mr Green wrote:

well does it work if you have user repo in pacman.d only I still get 1 mirror

Could u please paste your output, or private message me, I'll try my best to help u figure out the problem.

Mr Green · 2006-11-05 14:36:35

np seems to be working now ... it just did not have anything to upgrade when I first used it ...

sabooky · 2006-11-05 15:09:02

Mr Green wrote:

np seems to be working now ... it just did not have anything to upgrade when I first used it ...

Just wondering, did u notice any speed diff?

noriko · 2006-11-05 17:59:17

:: Retrieving packages from extra...
Downloading: openoffice-base-2.0.4-1.pkg.tar.gz
repo: extra
6,816,112/127,700,347 Bytes 5% 02m04s 2001.86 KB/s 24 connections

damn u [shakes fist @ u] lolol ... i'm so jealous.... x)

Arch Linux

#1 2006-11-02 04:19:45

pacget: Super fast pacman downloads

#2 2006-11-02 07:06:43

Re: pacget: Super fast pacman downloads

#3 2006-11-02 07:13:04

Re: pacget: Super fast pacman downloads

#4 2006-11-02 07:57:06

Re: pacget: Super fast pacman downloads

#5 2006-11-02 09:59:35

Re: pacget: Super fast pacman downloads

#6 2006-11-02 12:45:02

Re: pacget: Super fast pacman downloads

#7 2006-11-02 15:25:09

Re: pacget: Super fast pacman downloads

#8 2006-11-02 15:34:42

Re: pacget: Super fast pacman downloads

#9 2006-11-02 18:03:10

Re: pacget: Super fast pacman downloads

#10 2006-11-02 19:32:58

Re: pacget: Super fast pacman downloads

#11 2006-11-03 14:03:26

Re: pacget: Super fast pacman downloads

#12 2006-11-03 22:15:55

Re: pacget: Super fast pacman downloads

#13 2006-11-03 22:27:35

Re: pacget: Super fast pacman downloads

#14 2006-11-04 08:36:12

Re: pacget: Super fast pacman downloads

#15 2006-11-04 16:45:14

Re: pacget: Super fast pacman downloads

#16 2006-11-04 19:34:37

Re: pacget: Super fast pacman downloads

#17 2006-11-04 19:59:02

Re: pacget: Super fast pacman downloads

#18 2006-11-04 21:11:06

Re: pacget: Super fast pacman downloads

#19 2006-11-05 02:23:29

Re: pacget: Super fast pacman downloads

#20 2006-11-05 08:08:32

Re: pacget: Super fast pacman downloads

#21 2006-11-05 09:12:54

Re: pacget: Super fast pacman downloads

#22 2006-11-05 14:30:23

Re: pacget: Super fast pacman downloads

#23 2006-11-05 14:36:35

Re: pacget: Super fast pacman downloads

#24 2006-11-05 15:09:02

Re: pacget: Super fast pacman downloads

#25 2006-11-05 17:59:17

Re: pacget: Super fast pacman downloads

Board footer