Reflector reborn

skanky · 2015-10-05 08:15:39

karol wrote:

Which version are you using? 2014.11-2 ?
https://projects.archlinux.org/svntogit … a04c649232

64 bit here too and yes was still at 2014.11.2 and using Xyne's x86_64 repo.
That's still at 2014.11-2 and reports an invalid signature.

However I've just discovered that I had that repo above community in the pacman.conf file. Switching them round, I now get the Community version of Reflector, which has been updated and works.

Xyne · 2015-10-06 19:26:25

Sorry, I haven't had time to update my binary repo in a while. I plan to rebuild everything within an hour or two.

skanky · 2015-10-06 20:28:14

Speaking for myself at least, no problem, but thanks for the updates.

Mikaela · 2015-11-21 18:45:22

I am getting this error,

error: failed to retrieve mirror data: (<urlopen error [Errno -2] Name or service not known>)

. I am not sure when it started, but I noticed it today.

Reflector 2014.11-2, DNS works with everything else (DNSSEC validating Unbound).

nameserver ::1
nameserver 127.0.0.1
#nameserver 2001:4860:4860::8888
#nameserver 8.8.4.4
options edns0
search mikaela.info

EDIT: Fixed formatting.

EDIT2: The command is

/usr/bin/reflector --protocol https --latest 30 --number 20 --sort rate --save /etc/pacman.d/mirrorlist

EDIT3: After testing without Unbound (those Google DNS addresses) it worked and now it also works with Unbound.

Last edited by Mikaela (2015-11-21 19:00:16)

Red54 · 2016-12-01 10:04:06

Please remove 'completion_pct' check!

/usr/lib/python3.5/site-packages/Reflector.py:
189: # Filter incomplete mirrors and mirrors that haven't synced.
190: mirrors = list(m for m in mirrors if m['last_sync'] and m['completion_pct'] == 1.0)

According to https://www.archlinux.org/mirrors/status/

The 'completion_pct' is :
The number of mirror checks that have successfully connected and disconnected from the given URL. If this is below 100%, the mirror may be unreliable.

If 'completion_pct' != 1.0, it just shows the mirror is not 100% that the archlinux mirror check server can connect to it, and has nothing business to show whether it is incomplete or has synced, and also has nothing business to show whether the mirror is 100% that the users in the country of the mirror can connected to it because the archlinux mirror check server it not in that country!

Xyne · 2016-12-03 19:26:37

I have added a new filter option: --completion-percent
Any mirrors with a completion percent below the given value will be filtered. The default value is 100.

Xyne · 2019-03-02 12:09:36

I've refactored most of the code while maintaining backwards compatibility (for now) following an unexpected reaction to the code in another thread. I'll push the update to [community] after some more testing. It should be available in my repo in a few minutes.

There are three new filter options as well: --isos, --ipv4 and --ipv6

Kudlaty · 2019-03-07 18:22:44

after update:

Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.7/site-packages/Reflector.py", line 915, in <module>
    run_main(configure_logging=True)
  File "/usr/lib/python3.7/site-packages/Reflector.py", line 909, in run_main
    main(args, **kwargs)
  File "/usr/lib/python3.7/site-packages/Reflector.py", line 881, in main
    mirrors = list(mirrors)
  File "/usr/lib/python3.7/site-packages/Reflector.py", line 573, in sort
    yield from sort(mirrors, n_threads=self.threads, **kwargs)
  File "/usr/lib/python3.7/site-packages/Reflector.py", line 176, in sort
    rates = rate(mirrors, n_threads=n_threads)
  File "/usr/lib/python3.7/site-packages/Reflector.py", line 308, in rate
    logger.info(fmt.format(url, kibps, dt))
TypeError: unsupported format string passed to NoneType.__format__

Kudlaty · 2019-03-07 22:23:46

i think the problem is in rate_http function in except, where you return dt as None

Xyne · 2019-03-08 02:09:41

Kudlaty wrote:

i think the problem is in rate_http function in except, where you return dt as None

Yep. Thanks for reporting the issue. A fix has been pushed to the repos.

Neurognostic · 2020-01-22 22:43:32

When trying to get any mirrorlist with reflector. I keep getting

error: no mirrors found

with all the options I have tried. It worked a few days ago.

The source JSON is reachable. https://www.archlinux.org/mirrors/status/json/

# reflector --info
error: no mirrors found

Neurognostic · 2020-01-22 22:45:58

Looks like there is an issue with the completion percent of all the mirrors today, none of them are 100 and reflector defaults to 100.

using:

reflector --completion-percent=98

worked for me.

Related thread: https://bbs.archlinux.org/viewtopic.php?id=252334

Last edited by Neurognostic (2020-01-22 22:56:29)

sammiev · 2020-01-22 22:47:42

Neurognostic wrote:

When trying to get any mirrorlist with reflector. I keep getting
error: no mirrors found
with all the options I have tried. It worked a few days ago.
The source JSON is reachable. https://www.archlinux.org/mirrors/status/json/
# reflector --info
error: no mirrors found

I read about that today and the work around here.

https://bbs.archlinux.org/viewtopic.php?id=252334

Xyne · 2020-01-25 00:55:38

Not sure what happened with the mirrors but everything seems to be working normally again.

matteoipri · 2020-03-17 14:41:44

Hi Xyne,
First of all thank you for Reflector! :-)

I usually run the following command to update the mirrors list on my systems running archlinux:

sudo tee /etc/pacman.d/mirrorlist <<< "$(reflector --verbose --sort rate -l 5 -p https)"

Today I noticed that Reflector prints an extra line that looks like a debug print to me.
It prints:

/home/matteo/.cache/mirrorstatus.json

and I think this comes from line 111:

    print(cache_path)

This extra print breaks my one-liner above I use to avoid running the script as root (because I read somewhere on the wiki that it was discouraged to run the script as root).
The breakage consist in pacman printing the following during updates:

warning: config file /etc/pacman.d/mirrorlist, line 1: directive '/home/matteo/.cache/mirrorstatus.json' in section 'core' not recognized.
warning: config file /etc/pacman.d/mirrorlist, line 1: directive '/home/matteo/.cache/mirrorstatus.json' in section 'extra' not recognized.
warning: config file /etc/pacman.d/mirrorlist, line 1: directive '/home/matteo/.cache/mirrorstatus.json' in section 'community' not recognized.
warning: config file /etc/pacman.d/mirrorlist, line 1: directive '/home/matteo/.cache/mirrorstatus.json' in section 'multilib' not recognized.

Here is a list of possible solutions I see:
- make that print go to stderr instead of stdout;
- remove that print;
- add a "DEBUG" logging level and command line option and put that print call in an if statement.

What do you think of my proposals?
What do you think of the one-liner above?

Thank you for your attention,
Matteo

Fixxer · 2020-03-17 19:08:41

What do you think of the one-liner above?

Always try to make things as simple as possible:

sudo bash -c "reflector --verbose --sort rate -l 5 -p https --save /etc/pacman.d/mirrorlist"

If somoeone runs reflector as a service can temporarily avoid dropping one line $HOME/.cache/mirrorstatus.json to journal using Exec= like this:

bash -c 'reflector --ipv4 -p https -l 5 -f 5 --sort rate --save /etc/pacman.d/mirrorlist >/dev/null'

Last edited by Fixxer (2020-03-17 19:43:35)

Xyne · 2020-03-18 13:05:09

Sorry, that print statement should not be there. I'll push the fix in a few minutes.

YCH · 2020-03-18 18:06:07

Xyne wrote:

Sorry, that print statement should not be there. I'll push the fix in a few minutes.

Hello, Xyne .

I just wanted to write to say thank for reflector, from long time reflector user.

ugga · 2020-03-19 18:33:24

I tried using reflector today as usual, but i got some Error.

sudo reflector --verbose --latest 10 --country Germany --protocol https --sort rate --save /etc/pacman.d/mirrorlist

resulting in this output:
/root/.cache/mirrorstatus.json
error: The read operation timed out

I got rid of this Error by doing this:
cp /home/ugga/.cache/mirrorstatus.json /root/.cache/mirrorstatus.json

After that, reflector updated /etc/pacman.d/mirrorlist succesfully

Xyne · 2020-03-21 11:07:50

ugga wrote:

/root/.cache/mirrorstatus.json
error: The read operation timed out

I suspect that the error was a transient network or server error unrelated to the cache file (the cache filepath in the output was due to an erroneous print statement and has been corrected in the latest release). The command would likely have worked when you ran it again even if you hadn't copied the file. I wasn't able to reproduce the error by deleting the file, but please let me know if you are.

I'll update the code to be more explicit about which error was raised.

Drew · 2020-04-29 23:39:23

The latest, score, and fastest filters have unintuitive behavior. Each filter has a difference precedence which is totally unclear from the UI. I see this as a problem because only the filter with the highest precedence is fully respected.

Normally, I would expect all filters to act on the source list in a boolean AND fashion. I would not expect to see any result unless it matches every filter. This appears to be how the other filters work.

Instead, these three filters acts in sequence on the result of each other:

    if options.latest and options.latest > 0:
        mirrors = mirrorstatus.sort(mirrors, by='age')
        mirrors = itertools.islice(mirrors, options.latest)

    if options.score and options.score > 0:
        mirrors = mirrorstatus.sort(mirrors, by='score')
        mirrors = itertools.islice(mirrors, options.score)

    if options.fastest and options.fastest > 0:
        mirrors = mirrorstatus.sort(mirrors, by='rate')
        mirrors = itertools.islice(mirrors, options.fastest)

Let's say I use this command:

reflector --score 10 --latest 10

My intuition and the documentation both tell me this should only return results that are in the top 10 scores and the top 10 ages.

  -l n, --latest n      Limit the list to the n most recently synchronized servers.
  --score n             Limit the list to the n servers with the highest score.

But that's not how this works. First it gets the 10 latest servers. Then it picks the 10 highest scoring out of those 10 latest servers. Obviously the second filter would have no effect. The user is forced to figure out this precedence and then write their filters accordingly. Something like this next command might approximate the desired result of the above command by honoring both filters to some degree. Still, there is not enough information to know which filter plays a bigger role.

reflector --score 10 --latest 50

------------------------------------------------------------

For a fix, I would determine absolute numbers for each ranking. Let's say the #10 latest mirror was updated 5 minutes ago. That means you can filter ages over 5 minutes from any list using mirrorstatus.filter (assuming this has the desired behavior). You would have to find the min of that and the --age filter to make sure they are both respected.

Also, why did you use iterators? I'm probably missing something, but it smells of premature optimization. Compared to lists, it makes the code harder to read and impossible to debug. I doubt there is a noticeable performance difference, especially since you iterate through the whole thing repeatedly to sort it regardless.

------------------------------------------------------------

Why don't you put this on github or some analogous service? These features would make it easier for the community to help you:

issue reporting
pull requests
source code view
version control

Last edited by Drew (2020-04-30 00:17:06)

eschwartz · 2020-04-30 15:03:02

@Drew,

From https://bbs.archlinux.org/viewtopic.php?id=121663

Xyne wrote:

Uploading the Git repos is still on my "I should probably do that but meh, not a priority" list. I have a few concerns about uploading them.

This is a 4.5 year old reply to me asking for an update to a previous discussion from *9* years ago.

Trilby · 2020-04-30 15:40:30

Drew, FYI - I hope Xyne doesn't mind me posting this here - I loved reflector, but for similar uncertainty as yours about exactly how the filters were working I looked into making my own tool. Reflector is very flexible, but I never needed that flexibility - I knew what I wanted and just needed a quick way to get it. So I came up with the following code which is very specific to my locale and my preferences ... but it is very easy to edit:

#!/bin/python

from urllib.request import urlopen
from json import load
from time import time

with urlopen('https://www.archlinux.org/mirrors/status/json/') as f:
    mirrors = [ m for m in load(f)['urls'] if
        m['country'] == 'United States' and
        m['protocol'] == 'https' and
        m['completion_pct'] == 1.0 and
        m['delay'] <= 3600 and
        m['duration_avg'] + m['duration_stddev'] <= 1.0
    ]

for m in mirrors:
    t0 = time()
    with urlopen(m['url'] + 'core/os/x86_64/core.db') as f:
        m['rate'] = len(f.read()) / (time() - t0)

for m in sorted(mirrors, key=lambda k: k['rate'], reverse=True):
   print(f'Server = {m["url"]}$repo/os/$arch')

This filters by criteria in the top block, then sorts by (actual) speed.

Note that this speed sorting must be last. There'd be no sane way to rank *every* possible server by actual speed unless you test every one on every run of reflector. This would take a very long time.

EDIT: Patched as per below. Thanks - I don't know the JSON lib very well.

Last edited by Trilby (2020-04-30 17:11:03)

schard · 2020-04-30 16:53:27

Just a nitpick:

< from json import loads
< loads(f.read().decode())
> from json import load
> load(f)

Xyne · 2020-05-01 03:13:09

@Drew
--fastest, --latest and --score are different from the other filters because they are based on relative rather than absolute criteria. I'm not sure that it would make sense to intersect them with the other filters. For small arguments (e.g. --score 10 --latest 10), there is a good chance that the intersection will be empty, Even more so when you add other criteria. They are not intended to filter the whole mirrorlist but rather the subset that meets the other criteria, e.g. the latest 10 https servers in country xx or the fastest 5 servers synced in the last 6 hours.

I'm considering changing the behavior for --latest and --score but I'm not convinced it would make sense. It definitely doesn't for --fastest. There are currently 577 mirrors. Ranking all of them to determine the x fastest would be unacceptably slow and a waste of server bandwidth.

In what situation would you want the intersection the top x from each category? Give me some examples and I will think about how to improve reflector.

As for generators and iterators, it has nothing to do with optimization. I find it much cleaner to work with generators and list comprehensions when modifying list elements. Why bother with intermediate lists if you're just going to discard them?

@eschwartz
Life gets in the way and I procrastinate. If it's any consolation, I'm hoping to release a new project soon so I'm thinking about how I want to handle uploading my Git repos again.

@Trilby
I don't mind at all.

Arch Linux

#76 2015-10-05 08:15:39

Re: Reflector reborn

#77 2015-10-06 19:26:25

Re: Reflector reborn

#78 2015-10-06 20:28:14

Re: Reflector reborn

#79 2015-11-21 18:45:22

Re: Reflector reborn

#80 2016-12-01 10:04:06

Re: Reflector reborn

#81 2016-12-03 19:26:37

Re: Reflector reborn

#82 2019-03-02 12:09:36

Re: Reflector reborn

#83 2019-03-07 18:22:44

Re: Reflector reborn

#84 2019-03-07 22:23:46

Re: Reflector reborn

#85 2019-03-08 02:09:41

Re: Reflector reborn

#86 2020-01-22 22:43:32

Re: Reflector reborn

#87 2020-01-22 22:45:58

Re: Reflector reborn

#88 2020-01-22 22:47:42

Re: Reflector reborn

#89 2020-01-25 00:55:38

Re: Reflector reborn

#90 2020-03-17 14:41:44

Re: Reflector reborn

#91 2020-03-17 19:08:41

Re: Reflector reborn

#92 2020-03-18 13:05:09

Re: Reflector reborn

#93 2020-03-18 18:06:07

Re: Reflector reborn

#94 2020-03-19 18:33:24

Re: Reflector reborn

#95 2020-03-21 11:07:50

Re: Reflector reborn

#96 2020-04-29 23:39:23

Re: Reflector reborn

#97 2020-04-30 15:03:02

Re: Reflector reborn

#98 2020-04-30 15:40:30

Re: Reflector reborn

#99 2020-04-30 16:53:27

Re: Reflector reborn

#100 2020-05-01 03:13:09

Re: Reflector reborn

Board footer