Throttling ftp.archlinux.org

phrakture · 2007-03-21 16:45:36

Hey everyone.

This is a heads-up message regarding package mirrors. We've been having some problems with people using ftp.archlinux.org and not a proper mirror for pacman, and have started having bandwidth issues.

Due to this, downloads from ftp.archlinux.org have been throttled, and will be much slower - it is recommended that you sort your mirrors and find a faster one.

Mr Green · 2007-03-21 16:51:22

Is there still a sortmirrors script around ,,,,,?

Dusty · 2007-03-21 16:53:25

especially, a *working* sortmirror's script? Last I heard, the one distributed with pacman had a tendency to empty some people's pacman.d/ files. Tyler had written an alternative, but I don't know if the official one works... has this been fixed?

Dusty

hacosta · 2007-03-21 17:24:45

like i said in other post, i hope pacman3 runs rankmirror in the post install to solve this issues, i mean after all ftp.archlinux.org is the default

Acid7711 · 2007-03-21 17:48:12

And yet another suggestion goes unheeded.

These kinds of problems are to be expected when defaulting to a single server with a growing community.... Imo, they shoulda never happened to begin with. Importants up high knew the community was growing at this rate, It was bound to happen.

The flip side of the coin is for everyone to actually get a good, up to date, reliable mirrors that match and work. That's often a battle in itself.

Arch is great, but more and more often stupid simple fix problems continue to grow out of hand until the end result is something like this. Hell, even a simple selectable list at install with known good mirrors grouped geographically to choose would help solve this for everyone continuing to use an install disk. Wouldn't be that hard to implement, but in the end it's just another suggestion that will get disregarded.

In the end, this is going to result in a LOT of new comers to Arch not knowing wtf is going on, getting frustrated, and leaving.

Last edited by Acid7711 (2007-03-21 18:04:41)

Mr Green · 2007-03-21 18:21:50

found sortmirrors [in /usr/bin] read it & sort it out ;-)

Last edited by Mr Green (2007-03-21 18:53:33)

toofishes · 2007-03-21 18:32:08

Hey guys- grab this:
http://cvs.archlinux.org/cgi-bin/viewcv … oot=Pacman

It will soon be released with pacman3. It is much better than sortmirrors in its behavior.

I do like the idea of selecting a mirror on install, we should look into this.

EDIT: In addition, some people see the need to rsync with the Archlinux ftp servers. Unless you have like 20 computers that all need these files, you are probably being quite wasteful with bandwidth and that is why these throttling things need to happen.

An exercise:
pacman -Q | wc -l
pacman -Sl | wc -l

Divide the two numbers, and see how small the percentage of packages you actually use is.

Last edited by toofishes (2007-03-21 18:34:02)

iBertus · 2007-03-21 20:21:22

I've been using the Virgina Tech mirror on the east coast and it is much faster for me than Arch Linux FTP ever was. So, if you live on the east coast of the US give it a try.

oli · 2007-03-21 20:46:36

The problem is I tried both of the German mirrors, both of them are very fast but not synced. Some of the US mirrors aren't synced too, but I keep trying - it the moment I have some bad mixture in update. But I do understand your problem heavy traffic is very expensive.

XazZ · 2007-03-21 21:19:18

oli wrote:

The problem is I tried both of the German mirrors, both of them are very fast but not synced...

I'm using the hosteurope mirror for a while now...
It's getting updated once a day... (usually at ~4 pm)
And you are right: this mirror is very fast...

Hmmm my idea for everyone: wait for pacman 3
Somewhere it was said that it allows you to download packages from multiple mirrors - that would split the traffic between a few servers
I'm not sure if this feature will be in pacman 3.0 because I can't find that point on the todo list - but I hope it'll be there soon

XazZ

codemac · 2007-03-21 21:41:48

iBertus wrote:

I've been using the Virgina Tech mirror on the east coast and it is much faster for me than Arch Linux FTP ever was. So, if you live on the east coast of the US give it a try.

\o/ We are also running rsync on the virginia tech server. Email me if you'd like more details.

phrakture · 2007-03-22 00:50:19

Running rankmirrors in a post install is not a solution. The problem is that people will see on the main page "oh hey a new PACKAGE-X is out! pacman -Syu... it's not there" - eventually, due to timing, people will switch their mirrors to ftp.archlinux.org to get "omgupdates".

oli · 2007-03-22 01:19:41

>people will switch their mirrors to ftp.archlinux.org to get "omgupdates".

The "problem" is there is no need to switch, because it's default after the installation and I do think most of the people will not change this

Dusty · 2007-03-22 01:51:44

phrakture wrote:

Running rankmirrors in a post install is not a solution. The problem is that people will see on the main page "oh hey a new PACKAGE-X is out! pacman -Syu... it's not there" - eventually, due to timing, people will switch their mirrors to ftp.archlinux.org to get "omgupdates".

Sync the main page to the slowest mirror instead of ftp.archlinux.org.

Dusty

hacosta · 2007-03-22 02:00:01

@phrakture
but you are forgetting persons who don't actually visit www.archlinux.org, throtling down the server is not a solution imho; people who use archlinux and don't visit the forums or the web page won't even know what's happening.. i think more people use www.archlinux.org because its the default than because of omgupdates.
i mean how are users supposed to know that they shouldn't use archlinux.org. besides, users who use testing will also suffer from this as there are no mirrors for testing.

phrakture · 2007-03-22 02:29:08

Dusty wrote:

phrakture wrote:
Running rankmirrors in a post install is not a solution. The problem is that people will see on the main page "oh hey a new PACKAGE-X is out! pacman -Syu... it's not there" - eventually, due to timing, people will switch their mirrors to ftp.archlinux.org to get "omgupdates".
Sync the main page to the slowest mirror instead of ftp.archlinux.org.

It comes from a DB that has all the data input into it when we update the central repos.

Dusty · 2007-03-22 05:18:07

phrakture wrote:

Dusty wrote:
phrakture wrote:
Running rankmirrors in a post install is not a solution. The problem is that people will see on the main page "oh hey a new PACKAGE-X is out! pacman -Syu... it's not there" - eventually, due to timing, people will switch their mirrors to ftp.archlinux.org to get "omgupdates".
Sync the main page to the slowest mirror instead of ftp.archlinux.org.
It comes from a DB that has all the data input into it when we update the central repos.

Surely that's the only way it can be done then? ;-)

Dusty

lessthanjake · 2007-03-22 06:14:36

toofishes wrote:

Hey guys- grab this:
http://cvs.archlinux.org/cgi-bin/viewcv … oot=Pacman

This works great! At least almost. I was able to find some mirrors working 12 times faster than the one I used before. The only ploblem was that the script put some mirrors not workin on the top my list. Deleting those and everything was good.

vilefridge · 2007-03-22 06:33:09

The rankmirrors script worked well, however for the past 2 days I've wondered what the hoof has been going on with these super slow pacman updates. It would hit 60k, then drop to 0. 80k, then 0. over and over and over. The average would always be 10.9kb/s and I finally decided there's no way I'm the only one experiencing this problem and checked the forum today. I would have appreciated knowing about this before it went down. In fact it _should_ have been reported to Arch users in a different manner than a forum posting. It almost makes me grumpy and wanting a cup of hot chocolate, maybe someone to rub my back and tell me everything is ok. And a soft pillow. One that's brand new. On the other hand, it was good to get out from under this rock, and find out exactly where pacman's server list was and learn how to sort it for speed.

dolby · 2007-03-22 06:38:08

vilefridge wrote:

The rankmirrors script worked well, however for the past 2 days I've wondered what the hoof has been going on with these super slow pacman updates. It would hit 60k, then drop to 0. 80k, then 0. over and over and over. The average would always be 10.9kb/s and I finally decided there's no way I'm the only one experiencing this problem and checked the forum today. I would have appreciated knowing about this before it went down. In fact it _should_ have been reported to Arch users in a different manner than a forum posting. It almost makes me grumpy and wanting a cup of hot chocolate, maybe someone to rub my back and tell me everything is ok. And a soft pillow. One that's brand new. On the other hand, it was good to get out from under this rock, and find out exactly where pacman's server list was and learn how to sort it for speed.

have u boosted pacman?

vilefridge · 2007-03-22 06:56:46

dolby, thanks for that suggestion. I optimized & configured with aria2 Can't wait for the next set of updates to see it in action!

neotuli · 2007-03-22 07:14:08

Dusty wrote:

phrakture wrote:
Dusty wrote:
Sync the main page to the slowest mirror instead of ftp.archlinux.org.
It comes from a DB that has all the data input into it when we update the central repos.
Surely that's the only way it can be done then? ;-)

Actually, it is. We don't know how many mirrors we have... how could we possibly know when the last of them has synced?

Bob Day · 2007-03-22 10:59:38

neotuli wrote:

Actually, it is. We don't know how many mirrors we have... how could we possibly know when the last of them has synced?

I do not know how this syncing actually works. Is there a chain reaction started that the mirror syncs the next mirror, or does ftp.archlinux.org sync them all?

Perhaps one should make ftp.archlinux.org a private server for developers and TU only (except for [testing]). This will safe a lot of bandwidth on the server and that can be used to sync a few selected mirrors all over the world.

Another idea could be to us an ip-detection on http://www.archlinux.org saying: "Your IP suggesting you are from coming from Germany, click here for a list of recommended servers near you". If people click on it you can redirect them to a list of good mirrors in Germany together with instructions on how to select there mirror.

japetto · 2007-03-22 13:18:09

iBertus wrote:

I've been using the Virgina Tech mirror on the east coast and it is much faster for me than Arch Linux FTP ever was. So, if you live on the east coast of the US give it a try.

+1. Works great for me as well.

sh__ · 2007-03-22 14:46:14

Acid7711 wrote:

Hell, even a simple selectable list at install with known good mirrors grouped geographically to choose would help solve this for everyone continuing to use an install disk. Wouldn't be that hard to implement, but in the end it's just another suggestion that will get disregarded.

Hmm... I did one ftp install with 0.8-beta1 and if I recall correctly, the installer indeed had a list of mirrors to choose from. I don't remember if the mirror I chose was set as the default in /etc/pacman.d/ after the installation.

Arch Linux

#1 2007-03-21 16:45:36

Throttling ftp.archlinux.org

#2 2007-03-21 16:51:22

Re: Throttling ftp.archlinux.org

#3 2007-03-21 16:53:25

Re: Throttling ftp.archlinux.org

#4 2007-03-21 17:24:45

Re: Throttling ftp.archlinux.org

#5 2007-03-21 17:48:12

Re: Throttling ftp.archlinux.org

#6 2007-03-21 18:21:50

Re: Throttling ftp.archlinux.org

#7 2007-03-21 18:32:08

Re: Throttling ftp.archlinux.org

#8 2007-03-21 20:21:22

Re: Throttling ftp.archlinux.org

#9 2007-03-21 20:46:36

Re: Throttling ftp.archlinux.org

#10 2007-03-21 21:19:18

Re: Throttling ftp.archlinux.org

#11 2007-03-21 21:41:48

Re: Throttling ftp.archlinux.org

#12 2007-03-22 00:50:19

Re: Throttling ftp.archlinux.org

#13 2007-03-22 01:19:41

Re: Throttling ftp.archlinux.org

#14 2007-03-22 01:51:44

Re: Throttling ftp.archlinux.org

#15 2007-03-22 02:00:01

Re: Throttling ftp.archlinux.org

#16 2007-03-22 02:29:08

Re: Throttling ftp.archlinux.org

#17 2007-03-22 05:18:07

Re: Throttling ftp.archlinux.org

#18 2007-03-22 06:14:36

Re: Throttling ftp.archlinux.org

#19 2007-03-22 06:33:09

Re: Throttling ftp.archlinux.org

#20 2007-03-22 06:38:08

Re: Throttling ftp.archlinux.org

#21 2007-03-22 06:56:46

Re: Throttling ftp.archlinux.org

#22 2007-03-22 07:14:08

Re: Throttling ftp.archlinux.org

#23 2007-03-22 10:59:38

Re: Throttling ftp.archlinux.org

#24 2007-03-22 13:18:09

Re: Throttling ftp.archlinux.org

#25 2007-03-22 14:46:14

Re: Throttling ftp.archlinux.org

Board footer