You are not logged in.

#1 2020-12-06 16:28:58

zpg443
Member
Registered: 2016-12-03
Posts: 271

How frequently do you update your mirror list?

Seems like it is a waste of time to do it before every system update. The only reference found on this subject was "once a week" on some random thread.

Assuming you update your system daily, what is an appropriate frequency and why?

Last edited by zpg443 (2020-12-06 16:31:21)

Offline

#2 2020-12-06 16:35:24

Scimmia
Fellow
Registered: 2012-09-01
Posts: 11,466

Re: How frequently do you update your mirror list?

whenever I notice an issue with the top mirror on the list. No need to change it if it's updating frequently and fast.

Offline

#3 2020-12-06 16:44:17

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,447
Website

Re: How frequently do you update your mirror list?

zpg443 wrote:

Seems like it is a waste of time to do it before every system update.

It certainly is - especially if you have reflector sort the mirrors by actual rated speed.  If you update daily (or more) there may be nothing at all to do for a give update - in many cases there will not even be new db files to download, so all that happens is a query of the timestamps from the server and absolutely nothing else is downloaded at all.  But if you update your mirrorlist, you'd download one database file from every mirror that gets rated for speed (half a dozen to a dozen in many cases).  And as of a recent change, reflector would instead download of a chunk of the iso to rate speeds with is quite a bit larger than the core.db.

Not only is this drastically slower in a large portion of cases of an attempted update (I'd bet it'd also be slower on average for daily updates), but it'd also be a waste of mirror resources.

If in contrast you don't rate the mirrors for speed, but just get a fresh list of recently synced mirrors, then the above concern doesn't really apply, but you'd still be downloading the mirrorlist json information every time.  And all this would serve almost no purpose whatsoever.

I follow the same pattern as Scimmia - I pretty much only update my mirrorlist when I see unusual lag or problems downloading updates.  The only time I'd pre-emptively update my mirrorlist was if I was traveling a good distance (to different regions) - in such a case it's reasonably safe to assume that the mirrors that were good when I was at home probably will no longer be particularly good choices and I'd likely do a pre-emptive update of the mirrorlist.

Last edited by Trilby (2020-12-06 16:45:22)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#4 2020-12-06 17:17:33

porcelain1
Member
Registered: 2020-01-18
Posts: 97

Re: How frequently do you update your mirror list?

Scimmia wrote:

whenever I notice an issue with the top mirror on the list. No need to change it if it's updating frequently and fast.

That's what I've been doing too.


Behemoth, wake up!

Offline

#5 2020-12-06 17:40:31

zpg443
Member
Registered: 2016-12-03
Posts: 271

Re: How frequently do you update your mirror list?

Trilby wrote:
zpg443 wrote:

Seems like it is a waste of time to do it before every system update.

It certainly is - especially if you have reflector sort the mirrors by actual rated speed.  If you update daily (or more) there may be nothing at all to do for a give update - in many cases there will not even be new db files to download, so all that happens is a query of the timestamps from the server and absolutely nothing else is downloaded at all.  But if you update your mirrorlist, you'd download one database file from every mirror that gets rated for speed (half a dozen to a dozen in many cases).  And as of a recent change, reflector would instead download of a chunk of the iso to rate speeds with is quite a bit larger than the core.db.

Not only is this drastically slower in a large portion of cases of an attempted update (I'd bet it'd also be slower on average for daily updates), but it'd also be a waste of mirror resources.

If in contrast you don't rate the mirrors for speed, but just get a fresh list of recently synced mirrors, then the above concern doesn't really apply, but you'd still be downloading the mirrorlist json information every time.  And all this would serve almost no purpose whatsoever.

I follow the same pattern as Scimmia - I pretty much only update my mirrorlist when I see unusual lag or problems downloading updates.  The only time I'd pre-emptively update my mirrorlist was if I was traveling a good distance (to different regions) - in such a case it's reasonably safe to assume that the mirrors that were good when I was at home probably will no longer be particularly good choices and I'd likely do a pre-emptive update of the mirrorlist.

For a time, I used your helpful script, which did not rely on reflector:

#!/bin/python

from urllib.request import urlopen
from json import load
from time import time

with urlopen('https://www.archlinux.org/mirrors/status/json/') as f:
    mirrors = [ m for m in load(f)['urls'] if
        m['country'] == 'United States' and
        m['protocol'] == 'https' and
        m['completion_pct'] == 1.0 and
        m['delay'] <= 4600 and
        m['duration_avg'] + m['duration_stddev'] <= 1.0
    ]

for m in mirrors:
    t0 = time()
    with urlopen(m['url'] + 'core/os/x86_64/core.db') as f:
        m['rate'] = len(f.read()) / (time() - t0)

for m in sorted(mirrors, key=lambda k: k['rate'], reverse=True):
   print(f'Server = {m["url"]}$repo/os/$arch')
exit()

It stopped working after the python update (3.9). I have not figured out why it broke, so I reluctantly went back to reflector.

Here is the console error output:

Traceback (most recent call last):
  File "~/scripts/python/mirrors.py", line 18, in <module>
    with urlopen(m['url'] + 'core/os/x86_64/core.db') as f:
  File "/usr/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

Last edited by zpg443 (2020-12-06 18:04:44)

Offline

#6 2020-12-06 18:23:22

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,447
Website

Re: How frequently do you update your mirror list?

My script is functionally equivalent to reflector for the purposes of this conversation.  As for the error - when posted it was (or should have been) noted that it doesn't have any error checking.  But the same script still works fine for me, so it is not simply due to python 3.9 - and from that error, I suspect the problem is specifically with one of the mirrors that it's attempting to test.  The mirror is returning a 403 which the script simply doesn't handle.  FWIW, I've revised my script to have the following in the middle sorting block:

for m in mirrors:
    t0 = time()
    try:
        with urlopen(m['url'] + 'iso/latest/arch/x86_64/airootfs.sfs', timeout=2) as f:
            m['rate'] = len(f.read(1024*1024)) / (time() - t0)
    except:
        m['rate'] = 0

This would handle this (and other) errors from mirrors not behaving as expected and would effectively just disqualify them (or really just move them to the very end of the output list).

Also note I switched from checking the core.db to downloading a specified number of bytes (1MB in this case) from airootfs.sys.  This was precipitated by the same discussions that lead Xyne to make a related change in reflector (I think reflector is now downloading a chunk of the iso).

Last edited by Trilby (2020-12-06 18:25:20)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#7 2020-12-07 13:09:34

zpg443
Member
Registered: 2016-12-03
Posts: 271

Re: How frequently do you update your mirror list?

Your error checking section resolved my script issue. Thanks.

Usually, there is nothing at all to do for a given system update between mirror db updates, but over time I have regularly been getting lag/speed issues between db updates.
So, I prepared a script (rather than a timer) to test the mirrorlist file age and regenerate the mirrorlist file once a week:

#!/bin/bash
function fileAge
{
    local fileMod
    if fileMod=$(stat -c %Y -- "$1")
    then
        echo $(( $(date +%s) - $fileMod ))
    else
        return $?
    fi
}

ageinsecs=$(fileAge /etc/pacman.d/mirrorlist)
if [[ $ageinsecs -gt 604800 ]]
then
    printf "Mirrors have not been updated within the last 7 days. Updating the list now using existing db.\nThe most recently synchronized U.S. mirrors, sorted by fastest, are:\n"
    python /home/zpg/scripts/python/mirrors.py | sudo tee /etc/pacman.d/mirrorlist
else
    printf "Mirrors were updated within the last 7 days. Will update Arch using the current list."
fi

I doubt hitting the servers again every 7 days is going to be much strain on the load. If 7 days resolves the issue, will increase the interval.

Last edited by zpg443 (2020-12-07 13:17:11)

Offline

Board footer

Powered by FluxBB