Bash script to update mirrorlist. Uses curl and rankmirrors.

mountaintrek · 2025-03-29 05:36:17

There is a thread in the Pacman index regarding reflector and updating the mirrorlist.

I wrote the below some time ago to update the mirror list. I made a few updates for general usage. This may be helpful to others and educational.

#!/bin/bash
# get-new-mirrorlist.sh
#
# Update arch mirrorlist
#
declare -- country='US'
declare -- protocol='https'
declare -- use_mirror_status='on'
declare -- number_of_servers=5
declare -- timestamp="$( date '+%Y%m%dT%H%M' )"
declare -- mirror_file='mirrorlist'
declare -- mirror_dir='/etc/pacman.d'
declare -- mirror_list_url="https://archlinux.org/mirrorlist/"
declare -- mirror_list_options="?country=${country}&protocol=${protocol}&use_mirror_status=${use_mirror_status}"

if (( EUID == 0 )); then
  cd  "${mirror_dir}" || exit 0
else
  printf -- 'Must be root\n'
  exit
fi

# Save previous list
cp --verbose --archive "${mirror_file}" "${mirror_file}.${timestamp}"

# Get new list, prepare file, and rank
curl --silent \
     --show-error \
     --write-out '%{stderr}
%header{date}
%{time_total} seconds %{size_download} bytes %{speed_download} bytes/second
%{exitcode} exit %{response_code} response\n' \
     "${mirror_list_url}${mirror_list_options}" \
  | sed -e 's/^#Server/Server/' -e '/^#/d' \
  | rankmirrors -n "${number_of_servers}" - >"${mirror_file}"

# Add timestamp
sed -i '1 i\# Created on '"${timestamp}"'\n#' "${mirror_file}"

echo
cat --number "${mirror_file}"

NuSkool · 2025-03-29 18:08:47

What is the world coming to these days?
First, dictators (and wannabe dictators) being placed as leaders in world power nations along with the threat of nuclear war being discussed,
and now reflector is being removed from the official Arch repos with the developer MIA.

@mountaintrek Interesting... and nice script in my scripting language of choice.

That said, I played around with a script to update pacman mirrors maybe a decade ago. Reflector was and has been my tool of choice though, with my effort being more an exercise in scripting. I've not really used my script since initially writing and testing it. IIRC, it only prints the results to screen for manual copy, paste. I may as well post it here though as a hard coded, simplified example, along with a link to the wiki for additional info.

Something similar to the wget and sed lines combined, set as a shell alias could possibly be an optional way of dealing with our loss of reflector.

#!/bin/bash
## Update mirrorlist

echo '##            #################################################################
##            ###################### Generated by udml ########################
##            #################################################################
##
##            Source: https://archlinux.org/mirrorlist/?country=US&protocol=https&ip_version=4&use_mirror_status=on'

	rm /tmp/mirrorlist &>/dev/null
	
	wget -q --output-document=/tmp/mirrorlist "https://archlinux.org/mirrorlist/?country=US&protocol=https&ip_version=4&use_mirror_status=on"

	sed '/## United States/d' /tmp/mirrorlist | sed '/Server/s/^#//g'

	rm /tmp/mirrorlist

https://wiki.archlinux.org/title/Mirrors

frostschutz · 2025-03-29 20:50:43

I just pick them at random.

curl https://archlinux.org/mirrorlist/all/ |
sed -n -e 's/^#//' -e '/# Germany/,/#/p' |
grep -F 'Server = https://' |
shuf -n 3

There is also …mirrorlist/?use_mirror_status=on but I'm not sure what that does, other than randomizing the order every time. If this order reflects mirror quality in any way, could use it above, and replace 'shuf' with 'head'.

seth · 2025-03-29 23:01:37

https://wiki.archlinux.org/title/Mirror … irror_list
https://archlinux.org/mirrors/status/ - it's probably the mirror score, but that doesn't say much about how good the mirror will work for you… except (likely) in germany.

twelveeighty · 2025-03-29 23:08:06

Everyone is free to script whatever they wish, so I'm not talking down to these works, but I fail to see how these scripts are any different from the one liner that's in the Wiki:

$ curl -s "https://archlinux.org/mirrorlist/?country=FR&country=GB&protocol=https&use_mirror_status=on" | sed -e 's/^#Server/Server/' -e '/^#/d' | rankmirrors -n 5 -

Edit: 'not' was an unfortunate omission in the first sentence

Last edited by twelveeighty (2025-03-29 23:08:46)

seth · 2025-03-30 06:11:05

It was not? Or was it not? Not it was? What?

If the mirror status isn't useless, you can probably skip client-side ranking if you're sufficiently close to the hetznerverse (and I suspect also the shuffle because the altering results indicate some sort of load-balancing is performed) and save yourself some time and the mirrors some load.
Or maybe head -n+1 down to the first n mirrors before ranking them.

If there're not many (good) mirrors between the two rivers, filtering for country can cause terrible results.

(These concerns however apply to reflector as well)

twelveeighty · 2025-03-30 14:44:24

seth wrote:

It was not? Or was it not? Not it was? What?
(These concerns however apply to reflector as well)

I did not have not in there before I added the not edit

I guess that's my point: all the tools and scripts have the exact same upside and downside. In fact, the mirror that Reflector selected during my recent install wasn't in my region (because it doesn't filter by region on the ISO?) and for 3 months it was the fastest `pacman -Syu` I have ever experienced. After /etc/pacman.d/mirrorlist got 'pacnew'-ed recently I foolishly used `rankmirrors` to overwrite the mirrorlist, this time filtered by region, without backing up the old one. Now I'm back to the 'usual', much slower downloads. I know how to fix it, that's not the point. But you are 100% right that filtering by region isn't always the right choice.

Trilby · 2025-03-30 15:22:56

The OP's script is excellent. Clean, efficient, and easily configurable. It's functionally similar to the "one-liner" in the wiki, but it has variables that would be easy to edit rather than having everything smashed on one line. It also appears to do a bit of interesting formatting to the curl output which I've not tested.

While the OP's is a great contribution, I'll also throw my hat in the ring. While I do think python is often over-used and it's not typically one of the first languages I look to, it does have some features that make it well suited to this task IMHO:

#!/bin/python

from urllib.request import urlopen
from json import load
from time import time

with urlopen('https://www.archlinux.org/mirrors/status/json/') as f:
    mirrors = [ m for m in load(f)['urls'] if
        m['country'] == 'United States' and
        m['protocol'] == 'https' and
        m['completion_pct'] == 1.0 and
        m['delay'] <= 3600 and
        m['duration_avg'] + 2 * m['duration_stddev'] <= 1.25
    ]

for m in sorted(mirrors, key=lambda k: k['score']):
    print(f'Server = {m["url"]}$repo/os/$arch')

Last edited by Trilby (2025-03-30 15:23:20)

Fixxer · 2025-03-30 19:11:10

Probably I've discovered shortcut for multiple countries LOL

curl -s "https://archlinux.org/mirrorlist/?country={FI,SE,NO}&protocol=https&use_mirror_status=on" | sed -e 's/^#Server/Server/' -e '/^#/d' | rankmirrors -n 3 -

Who knows the trick - "the keyword", if any exists, to include Worldwide servers into ranking / query?

Last edited by Fixxer (2025-03-30 19:13:50)

twelveeighty · 2025-03-30 23:58:10

I assume the Web service's code is in an Arch GitLab project, you can check there if there is a "Worldwide" country code. But its front-end page doesn't have "Worldwide" in the Country dropdown, so it may not be possible to filter on those.

seth · 2025-03-31 06:45:50

Just omit the country specifier?

curl -s "https://archlinux.org/mirrorlist/?protocol=https&use_mirror_status=on"

I don't think there're mirrors on cloudflare/akamai (and even if, those would be slow because they first need to check that you're a human … and pacman is not)

twelveeighty · 2025-03-31 14:35:47

Fixxer meant the ##Worldwide servers listed in /etc/pacman.d/mirrorlist. Omitting the country specifier lists all BUT those "worldwide" servers, it seems.

seth · 2025-03-31 15:51:17

Using the website generator spits out https://archlinux.org/mirrorlist/?count … _status=on and that has the worldwide™ mirrors
(I haven's seen the pacman-mirrorlist…list in ages)

Fixxer · 2025-03-31 17:06:47

Generally I don't care about "worldwide" mirrors. I'm asking it cause these servers seems to be quite fast and usable, referring to Reflector Almighty™ results (example for 3 and 5 mirrors):

reflector -c 'fi,se,no,' -p https -l 5 -f 5 --score 5 --sort rate --threads 6           
[2025-03-31 18:50:32] WARNING: failed to rate http(s) download (https://mirror.bahnhof.net/pub/archlinux/extra/os/x86_64/extra.db): Download timed out after 5 second(s).
[2025-03-31 18:50:33] WARNING: failed to rate http(s) download (https://mirror.5i.fi/archlinux/extra/os/x86_64/extra.db): Download timed out after 5 second(s).
################################################################################
################# Arch Linux mirrorlist generated by Reflector #################
################################################################################

# With:       reflector -c fi,se,no, -p https -l 5 -f 5 --score 5 --sort rate --threads 6
# When:       2025-03-31 16:50:33 UTC
# From:       https://archlinux.org/mirrors/status/json/
# Retrieved:  2025-03-31 16:49:44 UTC
# Last Check: 2025-03-31 15:50:43 UTC

Server = https://mirror.osbeck.com/archlinux/$repo/os/$arch
Server = https://mirror.accum.se/mirror/archlinux/$repo/os/$arch
Server = https://geo.mirror.pkgbuild.com/$repo/os/$arch
Server = https://mirror.bahnhof.net/pub/archlinux/$repo/os/$arch
Server = https://mirror.5i.fi/archlinux/$repo/os/$arch

reflector -c 'fi,se,no,' -p https -l 3 -f 3 --score 3 --sort rate --threads 6
################################################################################
################# Arch Linux mirrorlist generated by Reflector #################
################################################################################

# With:       reflector -c fi,se,no, -p https -l 3 -f 3 --score 3 --sort rate --threads 6
# When:       2025-03-31 16:51:14 UTC
# From:       https://archlinux.org/mirrors/status/json/
# Retrieved:  2025-03-31 16:49:44 UTC
# Last Check: 2025-03-31 15:50:43 UTC

Server = https://mirror.osbeck.com/archlinux/$repo/os/$arch
Server = https://mirror.accum.se/mirror/archlinux/$repo/os/$arch
Server = https://geo.mirror.pkgbuild.com/$repo/os/$arch

If reflector has uncertain future, so - as I see, probably the best way is not to use any external helpers, at all. Shortened example:

curl -s "https://archlinux.org/mirrorlist/?country={FI,SE,NO}&protocol=https&use_mirror_status=on" | sed -e 's/^#Server/Server/;/^#/d' | rankmirrors -n 3

(I haven's seen the pacman-mirrorlist…list in ages)

So do I, till I've seen these two threads.

Last edited by Fixxer (2025-03-31 18:21:30)

adventurer · 2025-03-31 17:42:51

Trilby wrote:

The OP's script is excellent. Clean, efficient, and easily configurable. It's functionally similar to the "one-liner" in the wiki, but it has variables that would be easy to edit rather than having everything smashed on one line. It also appears to do a bit of interesting formatting to the curl output which I've not tested.
While the OP's is a great contribution, I'll also throw my hat in the ring. While I do think python is often over-used and it's not typically one of the first languages I look to, it does have some features that make it well suited to this task IMHO:
#!/bin/python

from urllib.request import urlopen
from json import load
from time import time

with urlopen('https://www.archlinux.org/mirrors/status/json/') as f:
    mirrors = [ m for m in load(f)['urls'] if
        m['country'] == 'United States' and
        m['protocol'] == 'https' and
        m['completion_pct'] == 1.0 and
        m['delay'] <= 3600 and
        m['duration_avg'] + 2 * m['duration_stddev'] <= 1.25
    ]

for m in sorted(mirrors, key=lambda k: k['score']):
    print(f'Server = {m["url"]}$repo/os/$arch')

Thanks, Trilby, that looks great! How could that be applied to multiple countries? Something like

m['country'] == 'United States' or 'Canada' and

or

m['country'] == 'United States' or
m['country'] == 'Canada' and

produces errors:

   for m in sorted(mirrors, key=lambda k: k['score']):
             ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<' not supported between instances of 'NoneType' and 'float'

twelveeighty · 2025-03-31 17:56:54

Generally I don't care about "worldwide" mirrors

For whatever reason, in my region they are the fastest mirrors compared to all USA and Canada mirrors.

seth · 2025-03-31 22:02:06

country and country_code in the json status are just empty strings for the "worldwide" mirrors.

curl -sL 'https://www.archlinux.org/mirrors/status/json/' | jq '.urls[] | "Server = " + select(.country_code == "" or .country_code == "CA").url + "$repo/os/$arch"'

The US mirrors are probably slow because there's a 25% punishment tariff on all bits

Trilby · 2025-04-01 00:23:08

adventurer wrote:

Thanks, Trilby, that looks great! How could that be applied to multiple countries?

Your second example should work, though a double check of logical operator precedence may be needed and parenthesis to enforce grouping.

But a more pythonic approach which would be far better if the list of countries goes beyond two could be to replace the current country line with the following:

   m['country'] in [ 'United States', 'Canada' ] and

But bare in mind that with those options python may throw an error like "TypeError: lists not supported containing instances of 'FascistDictatorship' and 'FreeDemocracy'"

Last edited by Trilby (2025-04-01 00:23:28)

adventurer · 2025-04-01 15:18:57

Trilby wrote:

But a more pythonic approach which would be far better if the list of countries goes beyond two could be to replace the current country line with the following:
   m['country'] in [ 'United States', 'Canada' ] and

Ah - this works excellently! Thanks a lot!

But bare in mind that with those options python may throw an error like "TypeError: lists not supported containing instances of 'FascistDictatorship' and 'FreeDemocracy'"

ROFL ...

Arch Linux

#1 2025-03-29 05:36:17

Bash script to update mirrorlist. Uses curl and rankmirrors.

#2 2025-03-29 18:08:47

Re: Bash script to update mirrorlist. Uses curl and rankmirrors.

#3 2025-03-29 20:50:43

Re: Bash script to update mirrorlist. Uses curl and rankmirrors.

#4 2025-03-29 23:01:37

Re: Bash script to update mirrorlist. Uses curl and rankmirrors.

#5 2025-03-29 23:08:06

Re: Bash script to update mirrorlist. Uses curl and rankmirrors.

#6 2025-03-30 06:11:05

Re: Bash script to update mirrorlist. Uses curl and rankmirrors.

#7 2025-03-30 14:44:24

Re: Bash script to update mirrorlist. Uses curl and rankmirrors.

#8 2025-03-30 15:22:56

Re: Bash script to update mirrorlist. Uses curl and rankmirrors.

#9 2025-03-30 19:11:10

Re: Bash script to update mirrorlist. Uses curl and rankmirrors.

#10 2025-03-30 23:58:10

Re: Bash script to update mirrorlist. Uses curl and rankmirrors.

#11 2025-03-31 06:45:50

Re: Bash script to update mirrorlist. Uses curl and rankmirrors.

#12 2025-03-31 14:35:47

Re: Bash script to update mirrorlist. Uses curl and rankmirrors.

#13 2025-03-31 15:51:17

Re: Bash script to update mirrorlist. Uses curl and rankmirrors.

#14 2025-03-31 17:06:47

Re: Bash script to update mirrorlist. Uses curl and rankmirrors.

#15 2025-03-31 17:42:51

Re: Bash script to update mirrorlist. Uses curl and rankmirrors.

#16 2025-03-31 17:56:54

Re: Bash script to update mirrorlist. Uses curl and rankmirrors.

#17 2025-03-31 22:02:06

Re: Bash script to update mirrorlist. Uses curl and rankmirrors.

#18 2025-04-01 00:23:08

Re: Bash script to update mirrorlist. Uses curl and rankmirrors.

#19 2025-04-01 15:18:57

Re: Bash script to update mirrorlist. Uses curl and rankmirrors.

Board footer