Throttling ftp.archlinux.org

dtw · 2007-03-22 15:09:51

Erm, as a pkg maintainer it really helps me to have access to the very latest pkgs. What if I rebuild a bunch of pkgs only to find my fastest mirror is 12 hours behind and need to rebuild all my pkgs again?! That'd suck!

Cerebral · 2007-03-22 15:20:47

Well, afaik, ftp.archlinux.org is never guaranteed to be perfectly uptodate either - it syncs from the master server like all the rest of the mirrors, does it not?

dolby · 2007-03-22 16:08:14

Cerebral wrote:

Well, afaik, ftp.archlinux.org is never guaranteed to be perfectly uptodate either - it syncs from the master server like all the rest of the mirrors, does it not?

when i updated from the local mirror here after the big migration of packages a few days ago the mpd package hadnt yet been updated on ftp.archlinux.org so thats probably correct

Bob Day wrote:

Perhaps one should make ftp.archlinux.org a private server for developers and TU only (except for [testing]). This will safe a lot of bandwidth on the server and that can be used to sync a few selected mirrors all over the world.

imho thats a good idea

Last edited by dolby (2007-03-22 16:10:12)

phrakture · 2007-03-22 16:10:23

Yeah ftp.archlinux.org rsyncs the same way all other mirrors do, which means it can get bottlenecked too.

dtw · 2007-03-22 16:18:34

Oh, well, I'm fscked either way then

yankees26 · 2007-03-23 13:27:00

lessthanjake wrote:

toofishes wrote:
Hey guys- grab this:
http://cvs.archlinux.org/cgi-bin/viewcv … oot=Pacman
This works great! At least almost. I was able to find some mirrors working 12 times faster than the one I used before. The only ploblem was that the script put some mirrors not workin on the top my list. Deleting those and everything was good.

Would you mind either posting on the mailing list (pacman-dev) with the output or report it as a bug on flyspray, 'cause thats something that probably should be fixed.

slackhack · 2007-03-24 00:45:51

hacosta wrote:

@phrakture
but you are forgetting persons who don't actually visit www.archlinux.org, throtling down the server is not a solution imho; people who use archlinux and don't visit the forums or the web page won't even know what's happening.. i think more people use www.archlinux.org because its the default than because of omgupdates.
i mean how are users supposed to know that they shouldn't use archlinux.org. besides, users who use testing will also suffer from this as there are no mirrors for testing.

that's an excellent point. i didn't even know there was a problem with using ftp.archlinux.org until these recent problems. archlinux.org was the default server in the pacman files, it always worked fine, so i just left it alone. i only upgraded a single computer about once or twice a week, so i doubt that would have been a huge problem in the scheme of things. but it really doesn't make much sense imo to make that the default server if you don't want most of the people to use it.

Snarkout · 2007-03-24 03:02:01

Count me among those who only started using a mirror after being gently flamed a while back. If using ftp.archlinux is discouraged it should absolutely not be the default. Like most others in my situation I never even looked at my mirrors list since it was working just fine. I had no idea I was being a selfish dickhead, and I do understand what mirrors are for, but I really just never even gave it much thought since I expected sane defaults.

soloport · 2007-03-24 03:18:07

Please correct me if my understanding of things is off, but one problem that can crop up is ABS sync'ing against <some_mirror>. If one runs gensync a lot, this can be frustrating -- some ABS package versions can be newer than those on <some_mirror>. So, one suggestion would be: Make an abs-1.0.0-1-tar.gz package and keep it in lock-step with mirrored packages. Can this be done?

Another related issue is that packages are deposited at www.archlinux.org at random times. Therefore, some mirrors seem partially up-to-date. So, another suggestion: Update the server (where mirrors get their packages) on a more periodic, say, once-a-day-only basis. Can this be done?
.

neotuli · 2007-03-24 03:44:02

once a day would make the problem worse. Mirrors are partially synced because it takes them so long to pull big updates from the server that the files only appear one by one on the mirror and very slowly at that. Forcing mirrors to pull a large load at once instead of an evenly distributed load would not help.

lessthanjake · 2007-03-24 04:35:47

OK, now I'm trying to install Gnome from testing, 310.2 MB including eclipse and jre. It's unbelievable slow, would it be possible not to throttle [testing] ? It's not any fun in testing if it should be like this. I might just drop it, leave it to others and wait for it to go stable.

karmapolice · 2007-03-24 06:46:59

Yeah it's not funny to wait almost 3 minutes to get 1.5MB. I'm also trying to get gnome from testing.

lessthanjake · 2007-03-24 07:01:02

You can do this:

su -
wget ftp://ftp.archlinux.org/testing/os/i686
cd /var/cache/pacman/pkg/
for i in `cat ~/index.html |cut -d"\"" -f2 |grep pkg |grep -v eclipse| grep -v jre`; do wget $i & done
pacman -Syu

... but it might be against the throttling idea. But then again I don't think the throttling should apply to [testing]

neotuli · 2007-03-24 07:29:56

@ lessthanjake

What exactly do you think that bash script does? In fact it does nothing, and the throttling would apply just the same, even if the script worked like you intended it to...

lessthanjake · 2007-03-24 07:35:50

Did you try it?

I got all my packages in 15min, using packman at 10 K/s would take me (168000/10/3600) = 4h40m

Edit: I see I miss a / after 686, except from that it works.

Last edited by lessthanjake (2007-03-24 07:39:07)

barebones · 2007-03-24 15:50:35

While I'll agree with phrak that this was might have been a necessary measure, until there is a mirror for [Testing] I don't think it was a very good solution. I just spent an hour and half this morning trying to get my downloads to work (and probably wasting bandwidth). Pretty much the first thing that came to mind was rankmirrors, so I did that to all my repos, but it still didn't affect anything since all the gnome updates from testing made it look like nothing had changed. I'll still use testing, since I can just run a Pacman -Syuw and let the downloads go while I do other stuff, but this is probably the most annoying thing I've ever had to deal with on arch. I can imagine that for people who don't check these forums (and unless I missed it, there wasn't even anything about this on the arch mailing list) something like this could be a cause of major frustrations.

Pierre · 2007-03-24 15:59:28

Why don`t you use a mirror for testing, too? All mirrors I know have a testing repository, too.

barebones · 2007-03-24 16:51:48

Really? Can you point me to a list of them? The testing file in /etc/pacman.d only has the official repo for testing.

Last edited by barebones (2007-03-24 16:52:00)

tlaloc · 2007-03-24 16:59:21

http://ftp.hosteurope.de/mirror/ftp.arc … g/os/i686/

dtw · 2007-03-24 17:01:16

They're the same as all the other mirrors! As root/sudo:

cd /etc/pacman.d
cp current testing
sed -i "s|current|testing|g" testing

Simple enough?!

Zer0 · 2007-03-24 17:24:16

This needs to be announced on the homepage!!!

Hell it was just by chance I found this because a few days before I started hitting up the forums (see my low post count). Being that I'm a fairly new arch user I thought the ftp.archlinux.org mirror was slow due to recent move of packages from testing to current etc.. and everyone downloading from the mirrors.

I had no idea, I never would have thought to look in the forums because I thought this issue would pass. I've been using ftp.arhclinux.org since I starting using arch sometime last year. I didn't think to use a different mirror and why would I? It worked fine "out of the box". Also the main mirror is in my "neck of the woods" so I thought not to change it at all.

Had I known it was an issue I would have changed it because I'm not a prick. Hopefully it's at least announced in the next newsletter before we abandon lots of arch newcomers who will think arch updates are unbarebly slow.

A quick suggestion would be to change the /arch/setup script on the new about to be release arch 0.8 (voodoo) so that the selection of mirrors is more clear for the end user. I was thinking maybe a submenu with Continent --> Country --> Province/State etc.. --> City/Town. Maybe that's to extensive
but at least make some way of knowing where the server's lie so the user can select the one nearest to him/her. Last time I did an install I was still unsure as to which server to select, and this could be a problem as any new user doing an FTP install off a slow mirror will probably just give up, hit reset and go with another distro.

barebones · 2007-03-24 17:59:43

Wow, I feel silly now, thanks guys.

neotuli · 2007-03-24 19:17:51

@lessthanjake:

oh hey I missed the &, you missed the /, we're even methinks

Now I see what it does, it just does all the downloads in parallel.

mutlu_inek · 2007-03-24 23:01:46

dolby wrote:

have u boosted pacman?

Funnily, the aria2 part of that wiki page refers people to ftp.archlinux.org. It should be changed. I dod not do it since I am not knowledgeable about aria2 at all.

miraman · 2007-03-25 00:43:56

I am another selfish pig using a 200kb/s band (real download speed) in my company to download packages and install then at home...
And I am another one that never looked at the mirrors list (I didn't even know where was the list)...
Since there is no real close mirror to where I live (Brasil), I took a easy to solve solution, put archlinux mirror in the end of the list.
And send a suggestion, this could be the default for new installs. Do not make archlinux mirror private, but make it the "if everything fails, use this" mirror.

And a few doubts:
1 - There is not a way to make the archlinux mirror redirect the downloads by itself? I think I have seen this before...
2 - Can pacman (version 2) receive a quick implementation to use a mirror "randomically", as long as the selected mirror have the requested file? This could be used if the mirrors file have never been changed (or if the archlinux mirror is the first of the list). Of course this would be a temporary chance until pacman3 arrives (or the problem solves by itself )
3 - How much of "pacman -Sy" depends on the mirrors?

And of course, have a official list of mirrors, and the time when the sync is made would help a little.

MiraMan

Last edited by miraman (2007-03-25 00:47:09)

Arch Linux

#26 2007-03-22 15:09:51

Re: Throttling ftp.archlinux.org

#27 2007-03-22 15:20:47

Re: Throttling ftp.archlinux.org

#28 2007-03-22 16:08:14

Re: Throttling ftp.archlinux.org

#29 2007-03-22 16:10:23

Re: Throttling ftp.archlinux.org

#30 2007-03-22 16:18:34

Re: Throttling ftp.archlinux.org

#31 2007-03-23 13:27:00

Re: Throttling ftp.archlinux.org

#32 2007-03-24 00:45:51

Re: Throttling ftp.archlinux.org

#33 2007-03-24 03:02:01

Re: Throttling ftp.archlinux.org

#34 2007-03-24 03:18:07

Re: Throttling ftp.archlinux.org

#35 2007-03-24 03:44:02

Re: Throttling ftp.archlinux.org

#36 2007-03-24 04:35:47

Re: Throttling ftp.archlinux.org

#37 2007-03-24 06:46:59

Re: Throttling ftp.archlinux.org

#38 2007-03-24 07:01:02

Re: Throttling ftp.archlinux.org

#39 2007-03-24 07:29:56

Re: Throttling ftp.archlinux.org

#40 2007-03-24 07:35:50

Re: Throttling ftp.archlinux.org

#41 2007-03-24 15:50:35

Re: Throttling ftp.archlinux.org

#42 2007-03-24 15:59:28

Re: Throttling ftp.archlinux.org

#43 2007-03-24 16:51:48

Re: Throttling ftp.archlinux.org

#44 2007-03-24 16:59:21

Re: Throttling ftp.archlinux.org

#45 2007-03-24 17:01:16

Re: Throttling ftp.archlinux.org

#46 2007-03-24 17:24:16

Re: Throttling ftp.archlinux.org

#47 2007-03-24 17:59:43

Re: Throttling ftp.archlinux.org

#48 2007-03-24 19:17:51

Re: Throttling ftp.archlinux.org

#49 2007-03-24 23:01:46

Re: Throttling ftp.archlinux.org

#50 2007-03-25 00:43:56

Re: Throttling ftp.archlinux.org

Board footer