You are not logged in.

#1 2020-03-14 22:14:25

jbreizh
Member
Registered: 2012-10-20
Posts: 37

How pacman manage mirrolist?

hello,
it's the second time in one month that i have to edit my mirrorlist to access new upgrade. First time i had all french server activated and no new upgrade for 1 week. So i add worldwide server and i got 300+ upgrade. But since one week same problem and after switching to us server : 300 more upgrade. So pacman seems to rely on only one server on the list (the fastest to respond ???) and if this server is out of synchronisation you have no more upgrade and no notification of the problem.
Is this a limitation of pacman or there is configuration to do?
thanks

Offline

#2 2020-03-14 22:33:17

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 22,777
Website

Re: How pacman manage mirrolist?

jbreizh wrote:

So pacman seems to rely on only one server on the list (the fastest to respond ???)

It does only use one, but there is no testing to see which is fastest - that's on you to do.  Pacman simply uses the first mirror in the list.  Only if the first one fails to respond (or responds with a 404 or equivalent) will further mirrors even be attempted.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#3 2020-03-14 22:43:53

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 28,455
Website

Re: How pacman manage mirrolist?


Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

Offline

#4 2020-03-14 22:52:35

CarbonChauvinist
Member
Registered: 2012-06-16
Posts: 221

Re: How pacman manage mirrolist?

$ grep reflectme .bashrc
alias reflectme="sudo reflector --country 'United States' --latest 1000 --protocol https -f 15 --save /etc/pacman.d/mirrorlist"

"the wind-blown way, wanna win? don't play"

Offline

#5 2020-03-14 22:54:03

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 22,777
Website

Re: How pacman manage mirrolist?

Or just roll your own - if you are going to statically define a bunch a parameters, no need to use a tool with a good bit of code to allow for all different parameters:

#!/bin/python

from urllib.request import urlopen
from json import loads
from time import time
from sys import stderr

with urlopen('https://www.archlinux.org/mirrors/status/json/') as f:
    mirrors = [ m for m in loads(f.read().decode())['urls'] if
        m['country'] == 'United States' and
        m['protocol'] == 'https' and
        m['completion_pct'] == 1.0 and
        m['delay'] <= 3600 and
        m['duration_avg'] + m['duration_stddev'] <= 1.5
    ]

for m in mirrors:
    t0 = time()
    with urlopen(m['url'] + 'core/os/x86_64/core.db') as f:
        m['rate'] = len(f.read()) / (time() - t0)

for m in sorted(mirrors, key=lambda k: k['rate'], reverse=True):
   print(f'Server = {m["url"]}$repo/os/$arch')

Edit the first block according to your region and preferences.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#6 2020-03-15 00:44:21

CarbonChauvinist
Member
Registered: 2012-06-16
Posts: 221

Re: How pacman manage mirrolist?

@trilby, nice...

$ time ./reflection.py
Server = https://iad.mirrors.misaka.one/archlinux/$repo/os/$arch                                                                                                
Server = https://mirror.wdc1.us.leaseweb.net/archlinux/$repo/os/$arch                                                                                           
Server = https://arch.mirror.constant.com/$repo/os/$arch                                                                                                        
Server = https://mirror.kaminski.io/archlinux/$repo/os/$arch                                                                                                    
Server = https://mirror.mia11.us.leaseweb.net/archlinux/$repo/os/$arch                                                                                          
Server = https://mirrors.rit.edu/archlinux/$repo/os/$arch                                                                                                       
Server = https://mirror.dal10.us.leaseweb.net/archlinux/$repo/os/$arch                                                                                          
Server = https://repo.ialab.dsu.edu/archlinux/$repo/os/$arch                                                                                                    
Server = https://archmirror1.octyl.net/$repo/os/$arch                                                                                                           
Server = https://mirror.sfo12.us.leaseweb.net/archlinux/$repo/os/$arch                                                                                          
Server = https://arch.rrig.gs/$repo/os/$arch                                                                                                                    
Server = https://mirrors.xtom.com/archlinux/$repo/os/$arch  
...
real    0m6.854s                                                                                                                                                
user    0m0.361s                                                                                                                                                
sys     0m0.021s

$ time reflector --country 'United States' --latest 1000 --protocol https -f 15
...
Server = https://iad.mirrors.misaka.one/archlinux/$repo/os/$arch                                                                                                
Server = https://mirror.wdc1.us.leaseweb.net/archlinux/$repo/os/$arch                                                                                           
Server = https://mirrors.kernel.org/archlinux/$repo/os/$arch                                                                                                    
Server = https://mirror.pit.teraswitch.com/archlinux/$repo/os/$arch                                                                                             
Server = https://arch.mirror.constant.com/$repo/os/$arch                                                                                                        
Server = https://mirror.stephen304.com/archlinux/$repo/os/$arch
Server = https://plug-mirror.rcac.purdue.edu/archlinux/$repo/os/$arch
Server = https://mirror.dal10.us.leaseweb.net/archlinux/$repo/os/$arch
Server = https://mirror.mia11.us.leaseweb.net/archlinux/$repo/os/$arch
Server = https://mirrors.rit.edu/archlinux/$repo/os/$arch
Server = https://iad.mirror.rackspace.com/archlinux/$repo/os/$arch
Server = https://mirrors.xtom.com/archlinux/$repo/os/$arch
Server = https://mirror.kaminski.io/archlinux/$repo/os/$arch
Server = https://mirror.lty.me/archlinux/$repo/os/$arch
Server = https://mirrors.rutgers.edu/archlinux/$repo/os/$arch
...
real    0m4.530s
user    0m0.634s                                                                                                                                                
sys     0m0.055s

I'll play around with this some more, thanks for sharing.


"the wind-blown way, wanna win? don't play"

Offline

#7 2020-03-15 02:04:11

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 22,777
Website

Re: How pacman manage mirrolist?

I'm surprised there's not a much bigger difference in speed.  Mine is not designed to be fast.  Reflector runs the speed tests in parallel, I believe.  This should give a massive speed boost.  But I'm not sure I'd trust the speed tests with parallel processes as the local bandwidth could be a limiting factor.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#8 2020-03-15 10:00:20

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 9,925
Website

Re: How pacman manage mirrolist?

+1

If you prefer not to need sudo.  Edit /etc/pacman.conf to have all your repo lines use a new file to be owned by your user:

[core]
Include = /etc/pacman.d/mirrorlist.reflector

[extra]
Include = /etc/pacman.d/mirrorlist.reflector

[community]
Include = /etc/pacman.d/mirrorlist.reflector

Make the file and use the little function in your shellrc (rename it if you want):

# touch /etc/pacman.d/mirrorlist.reflector
# chown youruser:yourgroup /etc/pacman.d/mirrorlist.reflector

Function to include:

upp () {
 for i in 1 3 10
  do
   if reflector -c US -a $i -f 5 -p https --sort rate --save /etc/pacman.d/mirrorlist.reflector
    then
     return 0
    else
   echo "something is fucked up"
  fi
 done
}

CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

Board footer

Powered by FluxBB