Configure pacman to use several mirrors at the same time.

olive · 2014-03-22 16:35:14

All is in the title. My ISP provide me with a 2Megabyte/sec transfer rate. However I am constantly changing mirrors to find one that support that; once I have found one I have to change again a day or two after. I wonder if is it possible to configure pacman to use several minors at the same time. Some software downloader support that (axel I think) but I don't know how to integrate that with pacman and the mirrorlist. Is it possible? Has anyone tried it?

Slithery · 2014-03-22 16:49:06

Not a good idea. If the mirrors that you use aren't in sync with each other they will have conflicting package versions.

Why not use Reflector to manage your mirrorlist as it can sort your mirrors by speed?

I would imagine there is probably another cause for your issue though, my D/L is around 30Mb/s and I've yet to find a mirror that can't keep up (except ftp.archlinux.org obviously).

Last edited by Slithery (2014-03-22 16:49:50)

M177ER · 2014-03-22 16:51:41

Have a look at Powerpill

https://wiki.archlinux.org/index.php/Powerpill

Scimmia · 2014-03-22 17:08:24

slithery wrote:

Not a good idea. If the mirrors that you use aren't in sync with each other they will have conflicting package versions.

As long as all of the databases come from the same mirror, this is not an issue. If a mirror doesn't have the version needed, it will just return a 404.

karol · 2014-03-22 17:41:48

Do you desperately need to update (download packages) as fast as possible?

olive · 2014-03-22 18:04:22

@slithery. Correctly implemented, there would be no problem as scimma said. Also, are you sure that you do not mistake megabyte and megabit? 30 megabits/sec is about what I have.

@karol: Yes because I am a very impatient person. More seriously, faster is always better even if I am not desperate. I think it should be easy to implement in pacman. The Axel download manager support it, all you have to do is to pass it several URLs at the command line but I have not seen such an option in pacman.

karol · 2014-03-22 18:11:15

I'm not sure I understand. Have you treid XferCommand?
https://wiki.archlinux.org/index.php/Im … erformance

olive wrote:

30 megabits/sec is about what I have.

You have 16Mbitps, not 30.

Last edited by karol (2014-03-22 18:13:59)

olive · 2014-03-22 18:59:25

karol wrote:

I'm not sure I understand. Have you treid XferCommand?
https://wiki.archlinux.org/index.php/Im … erformance
But the XferCommand support only one URL. We should be able to pass "several %u" (one for all uncommented mirror).
olive wrote:
30 megabits/sec is about what I have.
You have 16Mbitps, not 30.

Euh? My modem sync at 25Mbitps.

karol · 2014-03-22 19:00:38

From your first post:

olive wrote:

All is in the title. My ISP provide me with a 2Megabyte/sec transfer rate.

olive · 2014-03-22 19:05:03

karol wrote:

From your first post:
olive wrote:
All is in the title. My ISP provide me with a 2Megabyte/sec transfer rate.

OK. I can get a little more 2.5 Megabyte/sec. But it is true that I never get the theoretical 25/(8*1020**2) megabyte/sec and by the way I never know how much a mega is 1000 or 1024.

karol · 2014-03-22 19:19:41

If you're using a standard consumer service, it doesn't really matter if mega is 10^3 or 2^10, as it's more like 'up to 2Mbps'.
Check their website or the contract you signed, there may be a part that outlines it.

Back on topic: have you treid any of the mentioned way of improving package download speed?

Xyne · 2014-03-22 19:21:42

M177ER wrote:

Have a look at Powerpill
https://wiki.archlinux.org/index.php/Powerpill

This. Powerpill was designed to use segmented and parallel downloads to speed up connections. It does exactly what OP wants.

Using the Xfer command will not achieve this because Pacman invokes the Xfer once for each package in the download queue, then waits for it to complete before downing the next command. Powerpill will start all downloads at once.

karol · 2014-03-22 19:29:37

For bigger packages, like many -data ones, the Xfer should be enough.

nomorewindows · 2014-03-22 20:41:46

I've never quite understood because it seems that powerpill doesn't get the entire packages there any quicker than pacman does. One thing about pacman is it relogins for every package even if in the same repo. I guess it's a possibility that the connection could drop (behind a proxy or something), but this seems slow(er). If your connection is X Mbps, you're just splitting up the broadband into several smaller downloads all going at the same time.

Last edited by nomorewindows (2014-03-22 20:42:41)

karol · 2014-03-22 20:45:39

I'm not an expert, but I think that powerpill opens many connections, while pacman just one.
This is not about your side, bt about how fast the server can send you data and it seems that OP gets his connections capped in some way.

progandy · 2014-03-22 20:47:07

nomorewindows wrote:

I've never quite understood because it seems that powerpill doesn't get the entire packages there any quicker than pacman does. [...] If your connection is X Mbps, you're just splitting up the broadband into several smaller downloads all going at the same time.

A mirror might limit the speed for each individual connection. So to fully utilize your broadband connection it might be beneficial to download multiple files at once. It doesn't help if you have already hit your speed limit.

nomorewindows · 2014-03-22 21:01:15

progandy wrote:

nomorewindows wrote:
I've never quite understood because it seems that powerpill doesn't get the entire packages there any quicker than pacman does. [...] If your connection is X Mbps, you're just splitting up the broadband into several smaller downloads all going at the same time.
A mirror might limit the speed for each individual connection. So to fully utilize your broadband connection it might be beneficial to download multiple files at once. It doesn't help if you have already hit your speed limit.

I'm hitting top end just with one package. No speed limits on the mirrors I'm using. Well, that clears it up.

Xyne · 2014-03-23 16:06:04

nomorewindows wrote:

I've never quite understood because it seems that powerpill doesn't get the entire packages there any quicker than pacman does. One thing about pacman is it relogins for every package even if in the same repo. I guess it's a possibility that the connection could drop (behind a proxy or something), but this seems slow(er). If your connection is X Mbps, you're just splitting up the broadband into several smaller downloads all going at the same time.

This is true during a file transfer if Pacman is using the full bandwidth of your connection. You will still notice an increase in speed due to it starting multiple connections in parallel, especially for small packages and the database files (e.g. powerpill -Sy is noticeably faster than pacman -Sy). It's also faster if you use Pacserve because it uses Pacserve's custom protocol to query packages instead of the standard HTTP requests. Furthermore it can use the Rsync protocol to download packages from Rsync-enabled mirrors, which is also faster than the HTTP protocol.

In general Powerpill should always be a little faster unless it is misconfigured. On a fiber optic network it will be much faster.

Incidentally, even if you use an XferCommand client that splits the connection for large downloads, Pacman will only pass it a single URI so all of those connections are made to the same server. Aside from possible server-side per-client speed caps that places a greater burden on the server. Powerpill circumvents both when multiple mirrors or internal Reflector support is enabled.

falconindy · 2014-03-23 19:16:25

nomorewindows wrote:

One thing about pacman is it relogins for every package even if in the same repo.

Hrmm, that's a bug in pacman. I can confirm the behavior you're talking about, but that isn't supposed to happen.

Arch Linux

#1 2014-03-22 16:35:14

Configure pacman to use several mirrors at the same time.

#2 2014-03-22 16:49:06

Re: Configure pacman to use several mirrors at the same time.

#3 2014-03-22 16:51:41

Re: Configure pacman to use several mirrors at the same time.

#4 2014-03-22 17:08:24

Re: Configure pacman to use several mirrors at the same time.

#5 2014-03-22 17:41:48

Re: Configure pacman to use several mirrors at the same time.

#6 2014-03-22 18:04:22

Re: Configure pacman to use several mirrors at the same time.

#7 2014-03-22 18:11:15

Re: Configure pacman to use several mirrors at the same time.

#8 2014-03-22 18:59:25

Re: Configure pacman to use several mirrors at the same time.

#9 2014-03-22 19:00:38

Re: Configure pacman to use several mirrors at the same time.

#10 2014-03-22 19:05:03

Re: Configure pacman to use several mirrors at the same time.

#11 2014-03-22 19:19:41

Re: Configure pacman to use several mirrors at the same time.

#12 2014-03-22 19:21:42

Re: Configure pacman to use several mirrors at the same time.

#13 2014-03-22 19:29:37

Re: Configure pacman to use several mirrors at the same time.

#14 2014-03-22 20:41:46

Re: Configure pacman to use several mirrors at the same time.

#15 2014-03-22 20:45:39

Re: Configure pacman to use several mirrors at the same time.

#16 2014-03-22 20:47:07

Re: Configure pacman to use several mirrors at the same time.

#17 2014-03-22 21:01:15

Re: Configure pacman to use several mirrors at the same time.

#18 2014-03-23 16:06:04

Re: Configure pacman to use several mirrors at the same time.

#19 2014-03-23 19:16:25

Re: Configure pacman to use several mirrors at the same time.

Board footer