You are not logged in.

#51 2007-03-25 13:43:20

mcover
Member
From: Germany
Registered: 2007-01-25
Posts: 134

Re: Throttling ftp.archlinux.org

mutlu_inek wrote:
dolby wrote:

Funnily, the aria2 part of that wiki page refers people to ftp.archlinux.org. It should be changed. I dod not do it since I am not knowledgeable about aria2 at all.

About that, parallel downloading and speeding up the download:
When I was new to Arch, I also wanted to get my upgrades faster and looked at the Boost Pacman wiki site. But I dropped the whole aria2 thing because I think it is selfish to use multiple connections on one server.

However couple days ago, I thought about aria2 again when I had to download approx 300mb of upgrades (testing-move); plus i currently live in a location (asia) where there is no local mirror (no good up-to-date one) of the arch-repos to be found.

Well I came up with a script that makes use of aria2 again (probably a _bad_ 10mins hack cuz I was too lazy to think about something better + i was tired - but it works, pretty well actually.. ). There is no extra stress on the individual mirrors cuz I still use one connection (the script lets you choose, though) per download. Aria2 just uses multiple mirrors to download the same file from (if one mirror doesn't have the file yet, aria2 just skips the not-up-to-date-mirror).

Here is the code:
/usr/bin/pacxfer

#!/bin/bash
# by Marco E aka mcover
#use as XferCommand in /etc/pacman.conf ... XferCommand = /usr/bin/pacxfer %o %u [wget|aria2c] [only aria2c: splits per server]

if [ $# -eq 0 ]
then
  echo -e "Usage: pacxfer destination-file source-url [wget|aria2c] [only aria2c: splits per server]\n"
  exit 0
fi

BASE_NAME=`echo "$2" | awk -F"/" '{print $NF}'`
REPO=`echo "$2" | awk -F"/" '{print $(NF-3)}'`
ARCH=`echo "$2" | awk -F"/" '{print $(NF-1)}'`

#URL 1 is obviously the original URL
URL_2="ftp://ftp.belnet.be/packages/archlinux/$REPO/os/$ARCH/$BASE_NAME"
URL_3="http://ftp.parrswood.manchester.sch.uk/Mirrors/ftp.archlinux.org/$REPO/os/$ARCH/$BASE_NAME"
URL_4="ftp://ftp.hosteurope.de/mirror/ftp.archlinux.org/$REPO/os/$ARCH/$BASE_NAME"
URL_5="ftp://ftp.ibiblio.org/pub/linux/distributions/archlinux/$REPO/os/$ARCH/$BASE_NAME"


echo -e "\n===== Downloading to \"$1\" from \"$2\". Basename = \"$BASE_NAME\". Repo = \"$REPO\". Arch = \"$ARCH\". =====\n"

if [ "$3" = "wget" ]
then
  /usr/bin/wget --passive-ftp --retry-connrefused -c -O $1 $2
  EXIT_CODE=$?
fi

if [ "$3" = "aria2c" ]
then
#  /usr/bin/aria2c -s $4 -m 10 -o $1 $2

  echo "$2" | grep "$REPO.db" > /dev/null
  if [ $? -eq 0 ]
  then
#    /usr/bin/aria2c -s $4 -m 10 -o $1 $2
    /usr/bin/wget --passive-ftp --retry-connrefused -c -O $1 $2
  else
    /usr/bin/aria2c -s $4 -m 10 -o $1 $2 $URL_2 $URL_3 $URL_4 $URL_5
  fi
  EXIT_CODE=$?

  if [ $EXIT_CODE -eq 0 ]
  then
    chmod go+r $1
  fi
fi

exit $EXIT_CODE

As you can see I use wget to get the db, because I think it would be a terrible terrible mistake to get the db from more than one server simultaneously since the db doesn't have a versionized name.

My first mirror (I put it in the pacman.conf) is the nethat one. I selected the mirrors 2-5 because I think they were pretty up-to-date when I checked them (i avoid using the ftp.archlinux.org mirror - even though archlinux.org is the fastest mirror where i am at). If you want to use different extra mirrors you have to edit the lines with URL_*. Sorry about the hack - I wanted something simple -> KISS. tongue

My XferCommand looks like that:

XferCommand = /usr/bin/pacxfer %o %u aria2c 1

The result is that I get about 40-50kbs instead of 5-10kbs.

Hope it helps (and hope it isn't posted in the wrong place - even though i think its a bit off-topic).

€: One thing i forgot to mention: Yes i did see one of the existing scripts that do a similar thing, however I did not like the idea of ALL mirrors being added to the list. I wanted more control over the mirror-list.

Last edited by mcover (2007-03-25 13:51:03)

Offline

#52 2007-03-25 13:59:11

cromo
Member
From: Czestochowa, Poland
Registered: 2006-09-20
Posts: 87

Re: Throttling ftp.archlinux.org

I think we should try a different approach. Thing is people usually don't use mirrors because ftp.archlinux.org one seems to be most up to date. What I think we should do is to use the aria2 approach proposed on the wiki - the one which uses all the mirrors to boost the download speeds. Pacman should download the packages lists from the ftp.archlinux.org (that would be warranted to be always up to date) and then it would try all the mirrors to download the package. At least some of the mirrors would probably be already synced and had the package we wanted to get, so the user would always be happy with the up to date package AND the bandwidth usage would be shared among the mirrors. Additionally the speed boost would be significant, too. Finally there would no more be the need for the user to manually set up the mirrors. This is something that could be done for pacman3 before release.

Offline

#53 2007-03-25 16:38:47

ozar
Member
From: USA
Registered: 2005-02-18
Posts: 1,686

Re: Throttling ftp.archlinux.org

cromo wrote:

What I think we should do is to use the aria2 approach proposed on the wiki - the one which uses all the mirrors to boost the download speeds. Pacman should download the packages lists from the ftp.archlinux.org (that would be warranted to be always up to date) and then it would try all the mirrors to download the package.

If that would work without too many issues, I really like the idea!  cool


oz

Offline

#54 2007-03-25 17:38:22

Mandor
Member
Registered: 2006-06-06
Posts: 154

Re: Throttling ftp.archlinux.org

Simply, +1 "ftp.archlinux.org should not be default if its usage is discouraged". The "omgupdate" think seem quite overstated, especially as a reason for high load.


If everything else fails, read the manual.

Offline

#55 2007-03-25 17:48:50

_alexmyself
Member
From: france
Registered: 2005-09-18
Posts: 89

Re: Throttling ftp.archlinux.org

maybe you could make a new pacman version just to tell "please use mirror....." that everyone could see when upgrading because i personnally just find this post and see the homepage and understand why my kbps are so slow, i'm sure there's still a lot of people who don't know and, like me, use default ftp since arch has been installed..

Offline

#56 2007-03-25 17:58:51

entvex
Member
Registered: 2006-04-02
Posts: 52

Re: Throttling ftp.archlinux.org

cromo wrote:

I think we should try a different approach. Thing is people usually don't use mirrors because ftp.archlinux.org one seems to be most up to date. What I think we should do is to use the aria2 approach proposed on the wiki - the one which uses all the mirrors to boost the download speeds. Pacman should download the packages lists from the ftp.archlinux.org (that would be warranted to be always up to date) and then it would try all the mirrors to download the package. At least some of the mirrors would probably be already synced and had the package we wanted to get, so the user would always be happy with the up to date package AND the bandwidth usage would be shared among the mirrors. Additionally the speed boost would be significant, too. Finally there would no more be the need for the user to manually set up the mirrors. This is something that could be done for pacman3 before release.

I like that idea!


Linux can run on any thing...
look at me
[img]http://img53.imageshack.us/img53/8413/indexol8.png[/img]

Offline

#57 2007-03-25 18:09:10

iBertus
Member
From: Greenville, NC
Registered: 2004-11-04
Posts: 2,228

Re: Throttling ftp.archlinux.org

cromo wrote:

I think we should try a different approach. Thing is people usually don't use mirrors because ftp.archlinux.org one seems to be most up to date. What I think we should do is to use the aria2 approach proposed on the wiki - the one which uses all the mirrors to boost the download speeds. Pacman should download the packages lists from the ftp.archlinux.org (that would be warranted to be always up to date) and then it would try all the mirrors to download the package. At least some of the mirrors would probably be already synced and had the package we wanted to get, so the user would always be happy with the up to date package AND the bandwidth usage would be shared among the mirrors. Additionally the speed boost would be significant, too. Finally there would no more be the need for the user to manually set up the mirrors. This is something that could be done for pacman3 before release.

This is a sound idea in theory. It should distribute the load over a higher number of mirrors, and since Arch Linux isn't an uber-large community it could solve bandwidth issues for awhile. I'm not sure how simple this is in practice though. It could directly defy the KISS strategy that Arch has tried to maintain.

I think it would work better for all if:
1. ftp.archlinux.org was no longer the default.
2. all mirrors are automatically sorted when pacman is installed.
3. this parallel downloading, multi-mirror approach is left to those with the desire to de-KISS the system.

Perhaps a dev (phrakture?) could comment about this sort of idea and if it's truly in the vison of the pacman system.

Offline

#58 2007-03-25 18:19:58

hacosta
Member
From: Mexico
Registered: 2006-10-22
Posts: 423

Re: Throttling ftp.archlinux.org

can anyone tell me of a good mirror in north america, as im just testing the mirrors i can't seem to find one that is up to date (i just updated yesterday so they are still not synched) i dont mind if updates ar 2 or 3 days behind..

Offline

#59 2007-03-25 18:39:21

Echo
Member
From: Ohio, United States
Registered: 2006-05-16
Posts: 239

Re: Throttling ftp.archlinux.org

I'm on the east coast and I have the two mirrors below at the top:

Server = ftp://ftp-linux.cc.gatech.edu/pub/linux … ty/os/i686
Server = ftp://mirror.cs.vt.edu/pub/ArchLinux/community/os/i686

Seem to do just fine for me.

Edit: I noticed the first link was automatically "condensed" when I posted, make sure you get the complete address.

Last edited by Echo (2007-03-25 18:41:51)

Offline

#60 2007-03-25 19:21:00

mutlu_inek
Member
From: all over the place
Registered: 2006-11-18
Posts: 683

Re: Throttling ftp.archlinux.org

hacosta wrote:

can anyone tell me of a good mirror in north america, as im just testing the mirrors i can't seem to find one that is up to date (i just updated yesterday so they are still not synched) i dont mind if updates ar 2 or 3 days behind..

I use ibiblio.org. I have no idea whether it is really fast, but it surely delivers more than my DSL at home can handle. And it is more or less up to date. smile

Server = ftp://ibiblio.org/pub/linux/distributions/archlinux/current/os/i686
Server = ftp://ibiblio.org/pub/linux/distributions/archlinux/extra/os/i686
Server = ftp://ibiblio.org/pub/linux/distributions/archlinux/unstable/os/i686
Server = ftp://ibiblio.org/pub/linux/distributions/archlinux/community/os/i686

Last edited by mutlu_inek (2007-03-25 19:21:48)

Offline

#61 2007-03-26 16:55:54

djclue917
Member
Registered: 2006-12-03
Posts: 121

Re: Throttling ftp.archlinux.org

I have updated http://wiki.archlinux.org/index.php/Imp … erformance as well as the aria2 script. I've cleaned it up, put fancy comments and simple configuration scheme which would most likely be very useful. I've also changed the behavior so that pacman doesn't complain about corrupted files when you forcibly "kill" aria2 or pacget (even via Ctrl+C).

mcover wrote:
mutlu_inek wrote:
dolby wrote:

Funnily, the aria2 part of that wiki page refers people to ftp.archlinux.org. It should be changed. I dod not do it since I am not knowledgeable about aria2 at all.

About that, parallel downloading and speeding up the download:
When I was new to Arch, I also wanted to get my upgrades faster and looked at the Boost Pacman wiki site. But I dropped the whole aria2 thing because I think it is selfish to use multiple connections on one server.

However couple days ago, I thought about aria2 again when I had to download approx 300mb of upgrades (testing-move); plus i currently live in a location (asia) where there is no local mirror (no good up-to-date one) of the arch-repos to be found.

Well I came up with a script that makes use of aria2 again (probably a _bad_ 10mins hack cuz I was too lazy to think about something better + i was tired - but it works, pretty well actually.. ). There is no extra stress on the individual mirrors cuz I still use one connection (the script lets you choose, though) per download. Aria2 just uses multiple mirrors to download the same file from (if one mirror doesn't have the file yet, aria2 just skips the not-up-to-date-mirror).

<snipped>

Hope it helps (and hope it isn't posted in the wrong place - even though i think its a bit off-topic).

€: One thing i forgot to mention: Yes i did see one of the existing scripts that do a similar thing, however I did not like the idea of ALL mirrors being added to the list. I wanted more control over the mirror-list.

Offline

#62 2007-03-26 16:57:32

crouse
Arch Linux f@h Team Member
From: Iowa - USA
Registered: 2006-08-19
Posts: 907
Website

Re: Throttling ftp.archlinux.org

Is there a way to limit the archlinux.org servers to allow rsync only ? and changing the default repos of new installations to one of the mirrors ?

I rsync my own mirror to archlinux.org .... simply to keep it updated.  My mirror serves over 50 computers, so it reduces overall load on other mirrors and archlinux.org somewhat I suppose.  If there is a better way for me to rsync my mirror , I would love to know about it.

Offline

#63 2007-03-26 18:29:13

dtw
Forum Fellow
From: UK
Registered: 2004-08-03
Posts: 4,439
Website

Re: Throttling ftp.archlinux.org

crouse wrote:

Is there a way to limit the archlinux.org servers to allow rsync only ? and changing the default repos of new installations to one of the mirrors ?

I rsync my own mirror to archlinux.org .... simply to keep it updated.  My mirror serves over 50 computers, so it reduces overall load on other mirrors and archlinux.org somewhat I suppose.  If there is a better way for me to rsync my mirror , I would love to know about it.

Well, it would be better if you rsync'ed from one of the other mirrors for a start, eh?

Offline

#64 2007-03-26 18:45:48

Zer0
Member
From: Windsor, ON, Canada
Registered: 2006-08-25
Posts: 299

Re: Throttling ftp.archlinux.org

djclue917 wrote:

I have updated http://wiki.archlinux.org/index.php/Imp … erformance as well as the aria2 script. I've cleaned it up, put fancy comments and simple configuration scheme which would most likely be very useful. I've also changed the behavior so that pacman doesn't complain about corrupted files when you forcibly "kill" aria2 or pacget (even via Ctrl+C).

Thanks for the cleanup !!  I wonder if this aria2 method will cause issues with pacman 3.
When pacman 3 goes to current should I return to the default method and remove this pacget script?

Offline

#65 2007-03-26 19:04:40

crouse
Arch Linux f@h Team Member
From: Iowa - USA
Registered: 2006-08-19
Posts: 907
Website

Re: Throttling ftp.archlinux.org

dtw wrote:
crouse wrote:

Is there a way to limit the archlinux.org servers to allow rsync only ? and changing the default repos of new installations to one of the mirrors ?

I rsync my own mirror to archlinux.org .... simply to keep it updated.  My mirror serves over 50 computers, so it reduces overall load on other mirrors and archlinux.org somewhat I suppose.  If there is a better way for me to rsync my mirror , I would love to know about it.

Well, it would be better if you rsync'ed from one of the other mirrors for a start, eh?

yes that's great...... but i was told in irc ...... to rsync to archlinux.org as it was the most current, now if i change my rsync mirror and everything is "out of phase"..... what happens then ?

Wouldn't it be prudent to have one main mirror....... that accepts rsync only...... Master > Mirrors > Users ?  The Master "could" be bandwidth throttled.... to discourage people from using it, but I would "think" that you would want mirrors to mirror the Master server .... not each other.  So, wouldn't it be  better for the master server to restrict mirrors by IP address..... forcing users to USE a mirror vs archlinux.org's master server ?  Only mirrors would be allowed to sync. Perhaps a bit painful for some .... but not as painful as having a crawling download speed that takes forever.

I had asked before about the bandwidth a normal mirror used...... it was apparently over 2,500 gb month.... if we were able to limit that to about 2,000 per mirror... I could add one to the mix.  As it stands, I can't go over 2,500 gb without incurring huge fee's. I'm currently just using http to serve up the mirror, as I don't really want to run an ftp server, but it would probably work better than http for more users. I currently mirror only the "current" and "extra" repos.

Last edited by crouse (2007-03-26 19:05:37)

Offline

#66 2007-03-26 22:38:24

hacosta
Member
From: Mexico
Registered: 2006-10-22
Posts: 423

Re: Throttling ftp.archlinux.org

yes that's great...... but i was told in irc ...... to rsync to archlinux.org as it was the most current, now if i change my rsync mirror and everything is "out of phase"..... what happens then ?

then, you wait 2 or 3 days for your mirror to synch, honestly it's not such a big deal

Offline

#67 2007-03-27 01:16:22

Cerebral
Forum Fellow
From: Waterloo, ON, CA
Registered: 2005-04-08
Posts: 3,108
Website

Re: Throttling ftp.archlinux.org

I rsync my own mirror to archlinux.org .... simply to keep it updated.  My mirror serves over 50 computers, so it reduces overall load on other mirrors and archlinux.org somewhat I suppose.

How many packages out of the entirety of [current] and [extra] do your 50+ computers actually use, and how often do they update? You might want to consider something more intelligent than a full mirror-copy - like a caching http proxy - for your network. 

Wouldn't it be prudent to have one main mirror....... that accepts rsync only...... Master > Mirrors > Users ?  The Master "could" be bandwidth throttled.... to discourage people from using it, but I would "think" that you would want mirrors to mirror the Master server .... not each other.  So, wouldn't it be  better for the master server to restrict mirrors by IP address..... forcing users to USE a mirror vs archlinux.org's master server ?  Only mirrors would be allowed to sync. Perhaps a bit painful for some .... but not as painful as having a crawling download speed that takes forever.

That's what we already have.  ftp.archlinux.org is on a different server than rsync.archlinux.org (this is why it needs to sync like any other mirror out there) however, the reason official mirrors take forever to update is because, with only ~50 official mirrors, we somehow have over 500 unique machines rsync'ing from our rsync server.  With a big testing move, it takes far, far too long for the official mirrors to update, simply because everyone's rsync'ing from the master.

A better structure, which I think we're slowly working towards, is the master server feeds out packages to the official mirrors through rsync, and blocks anyone who's not recognized as an official mirror.  Then other people rsync from the geographically closest 'secondary' mirror that supports rsync'ing.

-=edit=-
Relevant newspost: http://www.archlinux.org/news/302/
Apparently we're not working towards that solution as slowly as I thought - it's already implemented.
-=/edit=-

Last edited by Cerebral (2007-03-27 01:27:18)

Offline

#68 2007-03-27 10:04:11

shining
Pacman Developer
Registered: 2006-05-10
Posts: 2,043

Re: Throttling ftp.archlinux.org

Was archlinux.org actually fast for some people for being throttled down?
It's so slow for me that it's totally unusable, and the first thing I do after installing arch is actually change the mirror to a much much faster one.
And that's the way it is for a long time.
The time needed for downloading an update, all mirrors would have already synced up, so what's the point? smile

Otherwise, I prefer the debian mirror structure much more, and the way it's configured make arch look really poor imo :
http://www.debian.org/doc/manuals/apt-h … urces.list

What's really annoying in the current config, is you can't tell it once to use the same mirror for several repo. I personally use the same mirror for every repo : current, extra, testing, unstable and community
and I need 5 nearly identical files for that.
Being able to just specify a line like that :
ftp://mir1.archlinuxfr.org current extra testing unstable community

but I guess that goes against the KISS philosophy? Unless you read it as Keep It Simple but Smart

And that's obviously just a minor annoyance since it only needs to be set up once, but still smile
maybe that would make it easier for users to switch from archlinux.org to another mirror.


pacman roulette : pacman -S $(pacman -Slq | LANG=C sort -R | head -n $((RANDOM % 10)))

Offline

#69 2007-03-27 10:20:18

.:B:.
Forum Fellow
Registered: 2006-11-26
Posts: 5,819
Website

Re: Throttling ftp.archlinux.org

As others said: an option to set the default mirror during installation might be a good option. Even better: force users to pick a mirror other than the default one.


Got Leenucks? :: Arch: Power in simplicity :: Get Counted! Registered Linux User #392717 :: Blog thingy

Offline

#70 2007-03-28 02:32:38

scottro
Member
From: NYC
Registered: 2002-10-11
Posts: 466
Website

Re: Throttling ftp.archlinux.org

One comment made in this thread was the impression that it might give the newcomer to Arch.  A friend who has heard me rave about how great Arch is gave it a shot and emailed me that it was much too slow.  (At the time, I hadn't been spending much time with Arch, playing with the BSDs, so wasn't aware of the site being throttled.  The other night, I went to put it on a laptop and couldn't believe how slow pacman had become.  So, I checked the forums, saw the suggestion to comment out ftp.archlinux.org from the various pacman.d files.

I think, with the overabundance of newcomer friendly distributions, Arch is often like a breath of fresh air--especially to folks coming from, say BSD background who need a Linux around for some hardware or program that the BSD's don't do without a lot of effort.   Is it possible, assuming that various suggested changes in these threads, aren't implemented for awhile, that might be some sort of message--perhaps the first time one runs pacman, with a note that one might be best off commenting out the ftp.archlinux.org line from the various /etc/pacman.d files?

Offline

#71 2007-03-28 04:39:14

dolby
Member
From: 1992
Registered: 2006-08-08
Posts: 1,581

Re: Throttling ftp.archlinux.org

[offtopic] @scottro: u are saying arch is slow & also couldn't believe how slow pacman had become while refering actually to the speed of a mirror. neither of them as long as im using them were ever slow. a mirror maybe but that hasnt got much to do with arch. if u think a mirror is slow u just switch to another one [/offtopic]


There shouldn't be any reason to learn more editor types than emacs or vi -- mg (1)
[You learn that sarcasm does not often work well in international forums.  That is why we avoid it. -- ewaller (arch linux forum moderator)

Offline

#72 2007-03-28 08:31:08

dtw
Forum Fellow
From: UK
Registered: 2004-08-03
Posts: 4,439
Website

Re: Throttling ftp.archlinux.org

dolby wrote:

[offtopic] @scottro: u are saying arch is slow & also couldn't believe how slow pacman had become while refering actually to the speed of a mirror. neither of them as long as im using them were ever slow. a mirror maybe but that hasnt got much to do with arch. if u think a mirror is slow u just switch to another one [/offtopic]

He was actually quoting a friend, so he's not saying anything.  He is suggesting what many have suggested that some sort of interim warning/fix would be useful.

Offline

#73 2007-03-28 22:00:15

crouse
Arch Linux f@h Team Member
From: Iowa - USA
Registered: 2006-08-19
Posts: 907
Website

Re: Throttling ftp.archlinux.org

Cerebral wrote:

I rsync my own mirror to archlinux.org .... simply to keep it updated.  My mirror serves over 50 computers, so it reduces overall load on other mirrors and archlinux.org somewhat I suppose.

How many packages out of the entirety of [current] and [extra] do your 50+ computers actually use, and how often do they update? You might want to consider something more intelligent than a full mirror-copy - like a caching http proxy - for your network.

More intelligent.. uhhhh ok.... whatever...... roll

In actuality it might be well over 50 and into the 100+ mark by now..... i'm not sure as I don't track the usage of it by others .... currently I have almost 20+ machines i take care of myself using my mirror, and alot of people that use it that I have given the url have more than one machine themselves too.  The mirror is on a fast server and serves not just my local network, but others from all over the world as well. So mirroring makes the most sense.

Updates to my entire system happen daily, others also update daily from the mirror.  My creating an rsync to archlinux.org to mirror the repo's saves all the other machines from updating from archlinux.org ....which is the way pacman was set to do by default....   I was TRYING to take some of the load AWAY from archlinux.org ... not increase it.  So when I rsync, and pull down 300 MB's of updates, it's not 50-100+ machines pulling down 300MB's from archlinux.org

So I thought it was pretty darn intelligent myself.... to create my own mirror so when I installed arch for people I could direct their pacman updates to pull from the mirror vs archlinux.org which was the default...... wink

Offline

#74 2007-03-28 23:23:32

Cerebral
Forum Fellow
From: Waterloo, ON, CA
Registered: 2005-04-08
Posts: 3,108
Website

Re: Throttling ftp.archlinux.org

Hey, I was just trying to be helpful and make a suggestion based on the limited information I had (hence the wording "you might want to consider") - no need to respond with sarcasm in such a defensive tone.  If making a full mirror makes the most sense, then you might now have to do so from one of arch's official mirrors that support rsync, as access to rsync.archlinux.org is now limited to the official mirrors only to ensure updates are propagated as quickly as possible.

-edit- Upon re-reading, it appears you may have taken offense to my choice of words.  When I spoke of a "more intelligent" solution, I was speaking in the terms of computer-science - a full mirror-copy is considered simple, or "un-intelligent", because it grabs every package regardless of that package's importance.  A "more intelligent" solution would, for example, only cache or mirror the packages that other people needed - I was not calling you personally unintelligent, and never meant to imply such.

Last edited by Cerebral (2007-03-28 23:32:27)

Offline

#75 2007-03-29 00:30:40

crouse
Arch Linux f@h Team Member
From: Iowa - USA
Registered: 2006-08-19
Posts: 907
Website

Re: Throttling ftp.archlinux.org

I'm sorry, I didn't mean to sound defensive, I was just trying to point out that "I was TRYING to help" , I didn't realize I was putting a burden on the server, I thought i was taking some of it OFF of the server .... wink 

I will rewrite my rsync script soon.  Currently using: rsync -avz --delete rsync://rsync.archlinux.org/current/ any idea's which mirrors would be best to rsync from ?? (My server is in Texas).   I went looking through my logs and did find this wink

Rsync Started on:
Wed Mar 28 12:00:01 CDT 2007
- Hide quoted text -

ARCHLINUX RSYNC Information
--------------------------------------

~~~~~~~ Current Repo Sync ~~~~~~~
******************************************************
*                                                    *
* This rsync server has recently been restricted to  *
* official public mirrors only.  If your host is not *
* an official public mirror, please rsync from one   *
* of the official public mirrors who offer a rsync   *
* service.                                           *
*                                                    *
* If you believe you have reached this in error,     *
* Please contact dale [AT] archlinux (DOT) org       *
*                                                    *
* Thank You,                                         *
*                                                    *
* The Arch Linux Team.                               *
*                                                    *
******************************************************


~~~~~~~ Extra Repo Sync ~~~~~~~
******************************************************
*                                                    *
* This rsync server has recently been restricted to  *
* official public mirrors only.  If your host is not *
* an official public mirror, please rsync from one   *
* of the official public mirrors who offer a rsync   *
* service.                                           *
*                                                    *
* If you believe you have reached this in error,     *
* Please contact dale [AT] archlinux (DOT) org       *
*                                                    *
* Thank You,                                         *
*                                                    *
* The Arch Linux Team.                               *
*                                                    *
******************************************************


Rsync Finished on:
Wed Mar 28 12:00:02 CDT 2007

Offline

Board footer

Powered by FluxBB