How is MirrorStatus done?

gri6507 · 2010-02-23 14:34:43

Hello,

I am actually a developer for a different distribution. Recently, I stumbled across https://www.archlinux.de/?page=MirrorStatus and realized that we would like to implement something similar. So, instead of trying to recreate the proverbial wheel, I would like to first ask if the source code for making this page work is Open Source and is available for our uses (after whatever necessary modifications)?

I'm sorry if this is an incorrect forum for posting this question. If it is, please direct me to the correct area to ask this question.
Thank you!

Allan · 2010-02-23 14:37:24

Send Pierre (http://bbs.archlinux.org/profile.php?id=1760) a PM. He would be the one to know how this is done.

archlinuxsagi · 2010-02-23 14:48:41

gri6507..
You nick looks familiar. Are you from unity linux?

3]) · 2010-02-23 14:59:11

archlinuxsagi wrote:

gri6507..
You nick looks familiar. Are you from unity linux?

He is from Unity Linux, that distro sounds kind of similar to Arch!

Never noticed Arch had this MirrorStatus page till now, nicely done!

gri6507 · 2010-02-23 16:20:53

Allan wrote:

Send Pierre (http://bbs.archlinux.org/profile.php?id=1760) a PM. He would be the one to know how this is done.

Thanks for the contact point. I'll be sending Pierre a PM.

P.S. I am indeed from Unity Linux - good memory! Our group is still rather young and we are always looking for great ways to improve our both our development process and our product. That's why we try to refer to other distros, including Arch to see how certain aspects have been addressed there. That's exactly why I was interested in the MirrorStatus implementation (which, by the way, i think it top notch!)

gri6507 · 2010-02-23 18:49:14

gri6507 wrote:

Allan wrote:
Send Pierre (http://bbs.archlinux.org/profile.php?id=1760) a PM. He would be the one to know how this is done.
Thanks for the contact point. I'll be sending Pierre a PM.

I actually do not see a way to contact Pierre from his profile page. The Jabber contact (is it like an email address?) bounced back as undeliverable. Is there another means?

tomk · 2010-02-23 20:19:23

pierre at archlinux dot de

gri6507 · 2010-02-23 21:07:02

Thanks. Email sent.

sand_man · 2010-02-23 21:35:54

Kind of off topic but I really like the design of http://unity-linux.org
Just thought I'd throw that out there

gri6507 · 2010-02-23 21:43:15

sand_man wrote:

Kind of off topic but I really like the design of http://unity-linux.org
Just thought I'd throw that out there

Thanks for the praise. It took us a few redesigns to get to this point, but in the end, our web admins really nailed it! I will forward your comment to them.

fukawi2 · 2010-02-23 22:12:58

There's also mirmon which is open source, but not exactly the same as the AL mirror status page
Example:
http://www.archserver.org/mirmon
PKGBUILD:
http://git.archserver.org/?p=hydrogen-a … 7f0133fc19

devnet · 2010-02-24 19:09:08

Heya everyone...

I wanted to show you the fruits of our labor and make sure everyone know how grateful we are to Arch and the community. Announcement is here: http://unity-linux.org/mirror-mirror-on-the-wall/

Mirror Listing is here: http://unity-linux.org/mm/mirrorstatus.html

Thanks again everyone

Pierre · 2010-02-25 07:33:06

The funny thing is that this MirrorStatus page started as a proof of concept (and sadly didn't evolve since then a lot). The code is availalbe under GPL. But it was never meant to be used standalone and is more or less tied to the Arch website and Arch Linux in general. But one could get some inspiration from it.

It contains of several scripts:
* At first there is this cronjob which runs hourly and collects all data. It also updated the mirror list from our main database: https://git.archlinux.de/www.archlinux. … irrors.php All data and possible errors are logged into a database. This way we wont just have an overview of the current mirror state but could also generate statistics over a longer period. ATM I don't use all its potential. The MirrorStatus page just shows the average sync delay and the average time it takes to download files.
* The actual output is generated at https://git.archlinux.de/www.archlinux. … Status.php (it might look a little weired due to the caching)
* There is also a special page for reflector to generate a list of fast and up2date mirrors: https://git.archlinux.de/www.archlinux. … lector.php
* Another proof of concept is a simple Download redirector: https://git.archlinux.de/www.archlinux. … Mirror.php

fukawi2 · 2010-02-25 09:08:45

Since we're in the mood for sharing things for mirrors... I've written this script to redirect requests to a geographically closest mirror based on the client IP address:
http://git.archserver.org/?p=sysopscrip … 269b1186e2

The mapping is a static mapping based on IP -> Country -> Mirror but it's a simple way to off-load some bandwidth usage to other mirrors, hopefully providing a faster response for the end-user

EDIT: oh, and it's implemented using Apache mod_rewrite:

HostnameLookups On
RewriteMap      geomap  prg:/usr/local/sysopscripts/geo-mapper.pl
RewriteCond     %{REMOTE_ADDR}  ^(.*)$
RewriteRule     ^(.*)$          ${geomap:%1|http://repo.archserver.org}$1  [redirect=temporary,last,nocase,env=mirror:${geomap:%1|http://repo.archserver.org}]

The 'env' part of the redirect is used for logging:

LogFormat "%{%s}t %b %v %t %{mirror}e" traffic

Last edited by fukawi2 (2010-02-25 09:13:31)

Pierre · 2010-02-25 09:40:30

You should have a look at mirrorbrain for this usage.

gri6507 · 2010-02-25 13:02:38

Pierre wrote:

The funny thing is that this MirrorStatus page started as a proof of concept (and sadly didn't evolve since then a lot). The code is availalbe under GPL. But it was never meant to be used standalone and is more or less tied to the Arch website and Arch Linux in general. But one could get some inspiration from it.
It contains of several scripts:
* At first there is this cronjob which runs hourly and collects all data. It also updated the mirror list from our main database: https://git.archlinux.de/www.archlinux. … irrors.php All data and possible errors are logged into a database. This way we wont just have an overview of the current mirror state but could also generate statistics over a longer period. ATM I don't use all its potential. The MirrorStatus page just shows the average sync delay and the average time it takes to download files.
* The actual output is generated at https://git.archlinux.de/www.archlinux. … Status.php (it might look a little weired due to the caching)
* There is also a special page for reflector to generate a list of fast and up2date mirrors: https://git.archlinux.de/www.archlinux. … lector.php
* Another proof of concept is a simple Download redirector: https://git.archlinux.de/www.archlinux. … Mirror.php

Thank you Pierre,

I will certainly take a look at the details of Arch's implementation. For now, based on the input of fukawi2, we have a working solution using mirmon (see earlier posts in this thread). I don't know if either one solution is better than the other, but that's something I'll be looking into.

fukawi2 · 2010-02-25 21:41:03

Pierre wrote:

You should have a look at mirrorbrain for this usage.

Oooohhh, pretty

Didn't know about that (hence I reinvented the wheel)... I'll look into it more, thanks

Arch Linux

#1 2010-02-23 14:34:43

How is MirrorStatus done?

#2 2010-02-23 14:37:24

Re: How is MirrorStatus done?

#3 2010-02-23 14:48:41

Re: How is MirrorStatus done?

#4 2010-02-23 14:59:11

Re: How is MirrorStatus done?

#5 2010-02-23 16:20:53

Re: How is MirrorStatus done?

#6 2010-02-23 18:49:14

Re: How is MirrorStatus done?

#7 2010-02-23 20:19:23

Re: How is MirrorStatus done?

#8 2010-02-23 21:07:02

Re: How is MirrorStatus done?

#9 2010-02-23 21:35:54

Re: How is MirrorStatus done?

#10 2010-02-23 21:43:15

Re: How is MirrorStatus done?

#11 2010-02-23 22:12:58

Re: How is MirrorStatus done?

#12 2010-02-24 19:09:08

Re: How is MirrorStatus done?

#13 2010-02-25 07:33:06

Re: How is MirrorStatus done?

#14 2010-02-25 09:08:45

Re: How is MirrorStatus done?

#15 2010-02-25 09:40:30

Re: How is MirrorStatus done?

#16 2010-02-25 13:02:38

Re: How is MirrorStatus done?

#17 2010-02-25 21:41:03

Re: How is MirrorStatus done?

Board footer