You are not logged in.

#1 2014-03-22 16:35:14

olive
Member
From: Belgium
Registered: 2008-06-22
Posts: 1,490

Configure pacman to use several mirrors at the same time.

All is in the title. My ISP provide me with a 2Megabyte/sec transfer rate. However I am constantly changing mirrors to find one that support that; once I have found one I have to change again a day or two after. I wonder if is it possible to configure pacman to use several minors at the same time. Some software downloader support that (axel I think) but I don't know how to integrate that with pacman and the mirrorlist. Is it possible? Has anyone tried it?

Offline

#2 2014-03-22 16:49:06

Slithery
Administrator
From: Norfolk, UK
Registered: 2013-12-01
Posts: 5,776

Re: Configure pacman to use several mirrors at the same time.

Not a good idea. If the mirrors that you use aren't in sync with each other they will have conflicting package versions.

Why not use Reflector to manage your mirrorlist as it can sort your mirrors by speed?

I would imagine there is probably another cause for your issue though, my D/L is around 30Mb/s and I've yet to find a mirror that can't keep up (except ftp.archlinux.org obviously).

Last edited by Slithery (2014-03-22 16:49:50)


No, it didn't "fix" anything. It just shifted the brokeness one space to the right. - jasonwryan
Closing -- for deletion; Banning -- for muppetry. - jasonwryan

aur - dotfiles

Offline

#3 2014-03-22 16:51:41

M177ER
Member
Registered: 2008-06-15
Posts: 148

Re: Configure pacman to use several mirrors at the same time.

Offline

#4 2014-03-22 17:08:24

Scimmia
Fellow
Registered: 2012-09-01
Posts: 12,045

Re: Configure pacman to use several mirrors at the same time.

slithery wrote:

Not a good idea. If the mirrors that you use aren't in sync with each other they will have conflicting package versions.

As long as all of the databases come from the same mirror, this is not an issue. If a mirror doesn't have the version needed, it will just return a 404.

Online

#5 2014-03-22 17:41:48

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Configure pacman to use several mirrors at the same time.

Do you desperately need to update (download packages) as fast as possible?

Offline

#6 2014-03-22 18:04:22

olive
Member
From: Belgium
Registered: 2008-06-22
Posts: 1,490

Re: Configure pacman to use several mirrors at the same time.

@slithery. Correctly implemented, there would be no problem as scimma said. Also, are you sure that you do not mistake megabyte and megabit? 30 megabits/sec is about what I have.

@karol: Yes because I am a very impatient person. More seriously, faster is always better even if I am not desperate. I think it should be easy to implement in pacman. The Axel download manager support it, all you have to do is to pass it several URLs at the command line but I have not seen such an option in pacman.

Offline

#7 2014-03-22 18:11:15

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Configure pacman to use several mirrors at the same time.

I'm not sure I understand. Have you treid XferCommand?
https://wiki.archlinux.org/index.php/Im … erformance


olive wrote:

30 megabits/sec is about what I have.

You have 16Mbitps, not 30.

Last edited by karol (2014-03-22 18:13:59)

Offline

#8 2014-03-22 18:59:25

olive
Member
From: Belgium
Registered: 2008-06-22
Posts: 1,490

Re: Configure pacman to use several mirrors at the same time.

karol wrote:

I'm not sure I understand. Have you treid XferCommand?
https://wiki.archlinux.org/index.php/Im … erformance

But the XferCommand support only one URL. We should be able to pass "several %u" (one for all uncommented mirror).

olive wrote:

30 megabits/sec is about what I have.

You have 16Mbitps, not 30.

Euh? My modem sync at 25Mbitps.

Offline

#9 2014-03-22 19:00:38

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Configure pacman to use several mirrors at the same time.

From your first post:

olive wrote:

All is in the title. My ISP provide me with a 2Megabyte/sec transfer rate.

Offline

#10 2014-03-22 19:05:03

olive
Member
From: Belgium
Registered: 2008-06-22
Posts: 1,490

Re: Configure pacman to use several mirrors at the same time.

karol wrote:

From your first post:

olive wrote:

All is in the title. My ISP provide me with a 2Megabyte/sec transfer rate.

OK. I can get a little more 2.5 Megabyte/sec. But it is true that I never get the theoretical 25/(8*1020**2) megabyte/sec and by the way I never know how much a mega is 1000 or 1024.

Offline

#11 2014-03-22 19:19:41

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Configure pacman to use several mirrors at the same time.

If you're using a standard consumer service, it doesn't really matter if mega is 10^3 or 2^10, as it's more like 'up to 2Mbps'.
Check their website or the contract you signed, there may be a part that outlines it.


Back on topic: have you treid any of the mentioned way of improving package download speed?

Offline

#12 2014-03-22 19:21:42

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,965
Website

Re: Configure pacman to use several mirrors at the same time.

M177ER wrote:

This. Powerpill was designed to use segmented and parallel downloads to speed up connections. It does exactly what OP wants.

Using the Xfer command will not achieve this because Pacman invokes the Xfer once for each package in the download queue, then waits for it to complete before downing the next command. Powerpill will start all downloads at once.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#13 2014-03-22 19:29:37

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Configure pacman to use several mirrors at the same time.

For bigger packages, like many -data ones, the Xfer should be enough.

Offline

#14 2014-03-22 20:41:46

nomorewindows
Member
Registered: 2010-04-03
Posts: 3,399

Re: Configure pacman to use several mirrors at the same time.

I've never quite understood because it seems that powerpill doesn't get the entire packages there any quicker than pacman does.  One thing about pacman is it relogins for every package even if in the same repo.  I guess it's a possibility that the connection could drop (behind a proxy or something), but this seems slow(er).  If your connection is X Mbps, you're just splitting up the broadband into several smaller downloads all going at the same time.

Last edited by nomorewindows (2014-03-22 20:42:41)


I may have to CONSOLE you about your usage of ridiculously easy graphical interfaces...
Look ma, no mouse.

Offline

#15 2014-03-22 20:45:39

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Configure pacman to use several mirrors at the same time.

I'm not an expert, but I think that powerpill opens many connections, while pacman just one.
This is not about your side, bt about how fast the server can send you data and it seems that OP gets his connections capped in some way.

Offline

#16 2014-03-22 20:47:07

progandy
Member
Registered: 2012-05-17
Posts: 5,251

Re: Configure pacman to use several mirrors at the same time.

nomorewindows wrote:

I've never quite understood because it seems that powerpill doesn't get the entire packages there any quicker than pacman does. [...]  If your connection is X Mbps, you're just splitting up the broadband into several smaller downloads all going at the same time.

A mirror might limit the speed for each individual connection. So to fully utilize your broadband connection it might be beneficial to download multiple files at once. It doesn't help if you have already hit your speed limit.


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

#17 2014-03-22 21:01:15

nomorewindows
Member
Registered: 2010-04-03
Posts: 3,399

Re: Configure pacman to use several mirrors at the same time.

progandy wrote:
nomorewindows wrote:

I've never quite understood because it seems that powerpill doesn't get the entire packages there any quicker than pacman does. [...]  If your connection is X Mbps, you're just splitting up the broadband into several smaller downloads all going at the same time.

A mirror might limit the speed for each individual connection. So to fully utilize your broadband connection it might be beneficial to download multiple files at once. It doesn't help if you have already hit your speed limit.

I'm hitting top end just with one package.  No speed limits on the mirrors I'm using.  Well, that clears it up.


I may have to CONSOLE you about your usage of ridiculously easy graphical interfaces...
Look ma, no mouse.

Offline

#18 2014-03-23 16:06:04

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,965
Website

Re: Configure pacman to use several mirrors at the same time.

nomorewindows wrote:

I've never quite understood because it seems that powerpill doesn't get the entire packages there any quicker than pacman does.  One thing about pacman is it relogins for every package even if in the same repo.  I guess it's a possibility that the connection could drop (behind a proxy or something), but this seems slow(er).  If your connection is X Mbps, you're just splitting up the broadband into several smaller downloads all going at the same time.

This is true during a file transfer if Pacman is using the full bandwidth of your connection. You will still notice an increase in speed due to it starting multiple connections in parallel, especially for small packages and the database files (e.g. powerpill -Sy is noticeably faster than pacman -Sy). It's also faster if you use Pacserve because it uses Pacserve's custom protocol to query packages instead of the standard HTTP requests. Furthermore it can use the Rsync protocol to download packages from Rsync-enabled mirrors, which is also faster than the HTTP protocol.

In general Powerpill should always be a little faster unless it is  misconfigured. On a fiber optic network it will be much faster.

Incidentally, even if you use an XferCommand client that splits the connection for large downloads, Pacman will only pass it a single URI so all of those connections are made to the same server. Aside from possible server-side per-client speed caps that places a greater burden on the server. Powerpill circumvents both when multiple mirrors or internal Reflector support is enabled.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#19 2014-03-23 19:16:25

falconindy
Developer
From: New York, USA
Registered: 2009-10-22
Posts: 4,111
Website

Re: Configure pacman to use several mirrors at the same time.

nomorewindows wrote:

One thing about pacman is it relogins for every package even if in the same repo.

Hrmm, that's a bug in pacman. I can confirm the behavior you're talking about, but that isn't supposed to happen.

Offline

Board footer

Powered by FluxBB