Reflector reborn

contra · 2020-12-19 22:22:50

@Xune

Thanks for the feedback.

The reason why someone may not want to filter is that some countries don't have many mirrors. I live in such place and looking at the mirror status page I think my situation is somewhat common. If you want to stick to the mirrors supporting rsync or https, it gets worse.

Of course one could always add the United States (or some other country with plenty of mirrors) at end of the filter list, which would also solve the issue, so it's more of a matter of preference than something of any real importance.

Last edited by contra (2020-12-19 22:41:07)

Xyne · 2020-12-20 22:32:59

@contra
I've come up with a solution that should handle all use cases without an extra command-line option. The --country option will now recognize the string "*" in addition to country names and country codes. When "*" is given, it will no longer filter countries but the order will still be used when sorting by country. Here are some examples:

# Restrict list to mirrors in Sweden and Denmark without sorting
$ reflector --country se,dk

# Restrict list to mirrors in Sweden and Denmark, sorting Sweden first and Denmark last.
$ reflector --country se,dk --sort country

# Select the 20 most recently synchronized mirrors. If any are in Sweden, place
# them at the top of the list. If any are in Denmark, place them at the end of
# the list. Servers in any other country are placed in between and sorted by
# country alphabetically. There is no guarantee that any of the latest 20
# mirrors will be in either Sweden or Denmark.
$ reflector --latest 20 --country "se,*,dk" --sort country

I'll push the update later today.

stef_204 · 2021-01-05 16:33:35

reflector 2020.12.20.1-1:

Since the last few updates, reflector has become incredibly slow.

reflector --latest 200 --protocol http,https --sort rate

took 13+ minutes.

8.13user 4.42system 13:32.24elapsed 1%CPU (0avgtext+0avgdata 30936maxresident)k
0inputs+792outputs (0major+5592minor)pagefaults 0swaps

See full message here: <https://paste.xinu.at/ONGaq/>

Previous versions took a fraction of that.
I tried to change protocol to `rsync` but same.
Is this "expected" or something wrong on my side?
(My internet connection is plenty fast.)

Last edited by stef_204 (2021-01-05 16:59:22)

Alad · 2021-01-07 12:29:11

I guess it's related to the "*" for --country explained in the post above...

sekret · 2021-01-07 12:58:52

I use

reflector --sort score > /etc/pacman.d/mirrorlist

as the first command in my update script. This produces a mirrorlist file with a problematic first line:

sorting by score <function sort.<locals>.key at 0x7f6311a61940>

Of course pacman complains afterwards with warning messages.

warning: config file /etc/pacman.d/mirrorlist, line 1: directive 'sorting by score <function sort.<locals>.key at 0x7f6311a61940>' in section 'core' not recognized.
warning: config file /etc/pacman.d/mirrorlist, line 1: directive 'sorting by score <function sort.<locals>.key at 0x7f6311a61940>' in section 'extra' not recognized.
warning: config file /etc/pacman.d/mirrorlist, line 1: directive 'sorting by score <function sort.<locals>.key at 0x7f6311a61940>' in section 'community' not recognized.
warning: config file /etc/pacman.d/mirrorlist, line 1: directive 'sorting by score <function sort.<locals>.key at 0x7f6311a61940>' in section 'multilib' not recognized.

Last edited by sekret (2021-01-07 13:00:58)

schard · 2021-01-07 13:08:06

I didn't review the latest code, but this seems like there is a call to a function wrapper missing, so the function object is returned instead of the result from a function call.
You can try speculum from the AUR if you want an alternative or interim solution.

Last edited by schard (2021-01-07 13:09:51)

stef_204 · 2021-01-07 14:20:33

Alad wrote:

I guess it's related to the "*" for --country explained in the post above...

@Alad
Is this a reply to my post or someone else's? Confused here as, as far as I'm concerned, I am not even using the --country flag or option. The latest changes have sort of "killed or broken" reflector for me.....
I can still do this

sudo reflector --verbose --country xx --age 6 --protocol https --sort rate --score 10 --save /etc/pacman.d/mirrorlist

but I fail to understand why the new logic and what seems to be a bottleneck in speed where the previous worked just great. Used it for years and it got the job done.

Last edited by stef_204 (2021-01-07 14:50:50)

Trilby · 2021-01-07 15:23:35

stef_204, you were asking reflector to get actual (empirically determined) speed metrics for 200 servers. That will take a long time as is really a huge bandwidth waste. You should prefilter a bit before sorting by rate. But until some relatively recent changes, it may have been quite a bit faster to do the same command because assessing mirror rate was done in parallel based on how many cpu cores you have. But you still retrieved data from 200 servers every time you ran reflector, and arguably that was not only excess, but a total waste as I've argued that parallel rate determination is unreliable - particularly when the number of threads is based just on $(nproc) - in many cases to the point of meaninglessness.

So if you want all 200 mirrors with unlreliable rate sorting, just don't sort by rate at all - and it will be even much faster than it used to be (as nothing is retrieved from the mirrors at all). If you do want to sort mirrors by rate, you should do it reliably - and this is only practical with a more limited set of mirrors.

Of course I think the --threads option still exists (not 100% sure) and you could still do multithreaded sorting. But I'd advise against --threads=$(nproc) (which used to be the default), but rather get some estimation of how many times greater your ISP bandwidth is relative to the better mirrors that you may test and use that as an upper-bound for --threads. But really a far better and more reliable approach would just be to prefilter.

Also it's a bit odd to include http and https when sorting for rate as quite a few servers provide both protocols. So you'll end up testing the same server twice for no reason; and https will always be slower, though likely not sufficiently so to surpass the variance in test-retest consistency - in which case the "better" rate is noise rather than signal anyways.

(edit: lots of typos found; lots more likely missed)

Last edited by Trilby (2021-01-07 15:45:33)

stef_204 · 2021-01-08 17:46:10

@Trilby
Thanks for the info. Without wanting to get into the finer points, from a practical standpoint; I've found that the following seems to suit my objectives (and I'm sure it can be further improved):

sudo reflector --verbose --country xx --age 6 --protocol https --sort rate --score 10 --save /etc/pacman.d/mirrorlist

Xyne · 2021-01-10 00:17:08

@sekret
The extraneous print statement has been removed. Sorry about that.

@stef_204
The previous rating method was seriously flawed. Downloading multiple files in parallel gives inaccurate results if the bandwidth is saturated at any point during the test, and using a small file doesn't give reliable results for sustained download rates. Switching to a larger file (~1 MB -> ~5 MB) and using a single thread is slower but much more reliable. As already stated above, the rating should be limited to a handful of filtered servers. If you really want faster download speeds, try powerpill for parallel downloads from multiple servers.

Although I recommend not bothering with rating at all, I have nevertheless re-added the --threads option to enable parallel rating, with a warning in the help text.

stef_204 · 2021-01-10 16:47:46

Xyne wrote:

Although I recommend not bothering with rating at all, I have nevertheless re-added the --threads option to enable parallel rating, with a warning in the help text.

Can you explain what you mean by the above? If you don't recommend rating, why use Reflector at all, then? I must be missing something. What is your recommended use?

schard · 2021-01-10 18:46:08

AFAIK, the mirror list backend already rates the mirrors. This is where the respective rating data in the JSON objects comes from.
So there's no need for each and every client to rate the mirrors again.
I think that this was a feature in reflector to allow users to fine tune the mirror list for their respective location and internet connection.
But whether such functionality is really worth the effort is indeed doubtful.
When I implemented speculum as an alternative to reflector, I refrained from implementing such a thing in the first place.

Xyne · 2021-01-11 18:54:39

stef_204 wrote:

Xyne wrote:
Although I recommend not bothering with rating at all, I have nevertheless re-added the --threads option to enable parallel rating, with a warning in the help text.
Can you explain what you mean by the above? If you don't recommend rating, why use Reflector at all, then? I must be missing something. What is your recommended use?

I mean that I don't think rating by download speed is an important metric when selecting a main server. I consider it far more important to select the most up-to-date mirrors with the desired protocol, possibly with a country filter to limit results to servers near you or countries that have better data protection policies, if that's a concern for you. But I use powerpill so my downloads are fast regardless of the speed of my main mirror. The only thing that concerns me about my main mirror is that I get the latest databases reliably.

Btw, powerpill uses reflector internally to build a list of mirrors for parallel downloads, which is why the speed of the mirrors in the system mirrorlist is basically irrelevant when using powerpill. The versions and checksums are checked against the downloaded databases, and the packages are all signed, so the only thing that really matters is the accuracy of the databases.

So yeah, spending time downloading files to assess download rates of different servers seems like a waste of time and server resources just to gain a little speed when better solutions exist. Reflector can be used to find a fast server, but unless someone is running a mirror on their toaster I don't really expect the speed between active servers to vary enough for it to matter in practice. Anyone who's concerned about the difference between a 3 minute download and a 5 minute download should use powerpill.

schard wrote:

AFAIK, the mirror list backend already rates the mirrors. This is where the respective rating data in the JSON objects comes from.
So there's no need for each and every client to rate the mirrors again.
I think that this was a feature in reflector to allow users to fine tune the mirror list for their respective location and internet connection.
But whether such functionality is really worth the effort is indeed doubtful.
When I implemented speculum as an alternative to reflector, I refrained from implementing such a thing in the first place.

The rating in the official mirrorlist data only measures the speed between the Arch servers and the mirrors. The speed of a connection between a mirror and a user will depend on the location and connection of the user. The fastest mirror from the Arch servers may be in the same datacenter, whereas the faster mirror for a user would probably be one hosted by the same service provider near them. The only way to tell is to measure the download speed.

DoXer · 2021-02-10 19:29:23

I get the following error after starting Reflector as a service:

# systemctl status reflector.service
● reflector.service - Refresh Pacman mirrorlist with Reflector.
     Loaded: loaded (/usr/lib/systemd/system/reflector.service; enabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Wed 2021-02-10 20:25:52 CET; 10s ago
TriggeredBy: ● reflector.timer
       Docs: https://wiki.archlinux.org/index.php/Reflector
    Process: 3014 ExecStart=/usr/bin/reflector @/etc/xdg/reflector/reflector.conf (code=exited, status=226/NAMESPACE)
   Main PID: 3014 (code=exited, status=226/NAMESPACE)

Feb 10 20:25:52 TST systemd[1]: Starting Refresh Pacman mirrorlist with Reflector....
Feb 10 20:25:52 TST systemd[3014]: reflector.service: Failed to set up mount namespacing: /run/systemd/unit-root/: Input/output error
Feb 10 20:25:52 TST systemd[3014]: reflector.service: Failed at step NAMESPACE spawning /usr/bin/reflector: Input/output error
Feb 10 20:25:52 TST systemd[1]: reflector.service: Main process exited, code=exited, status=226/NAMESPACE
Feb 10 20:25:52 TST systemd[1]: reflector.service: Failed with result 'exit-code'.
Feb 10 20:25:52 TST systemd[1]: Failed to start Refresh Pacman mirrorlist with Reflector..

The command

/usr/bin/reflector @/etc/xdg/reflector/reflector.conf

runs without error.

Any hints to solve the issue are more then welcome

Last edited by DoXer (2021-02-10 19:29:38)

Xyne · 2021-02-11 12:49:14

Are you trying to run reflector in a container? I searched for the error message "failed to set up mount namespacing" and all results mentioned permission settings in containers, e.g.

https://unix.stackexchange.com/question … -container
https://forum.proxmox.com/threads/apach … ied.56871/
https://stackoverflow.com/questions/599 … ied-failed

Do any of those solutions work?

LaserEyess · 2021-04-03 19:52:14

I have a suggestion for the service provided for reflector. Currently the line

RestrictAddressFamilies=~AF_AX25 AF_IPX AF_APPLETALK AF_X25 AF_DECnet AF_KEY AF_NETLINK AF_PACKET AF_RDS AF_PPPOX AF_LLC AF_IB AF_MPLS AF_CAN AF_TIPC AF_BLUETOOTH AF_ALG AF_VSOCK AF_KCM AF_UNIX AF_XDP

Is quite restrictive but also a blacklist. The problem with it being so restrictive is that it breaks DNS when you use systmed-resolved without a stub resolver. What is happening is that calls to getent for nss_resolve.so require dbus which requires AF_UNIX, which is explicitly blacklisted. Because I don't have any other nameservers specifed in /etc/resolv.conf getent will return that DNS is unavailable, despite it being available. The second problem, IMO, is that a blacklist is inherently less secure than a whitelist. It is clear this directive was chosen to severely limit the damage it can do as root, but it can be done better with using the same directive but as a whitelist. Currently I have

[Service]
RestrictAddressFamilies=
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

In my override file for the service, which both fixes the issue of /etc/resolv.conf not existing and makes the address families an explicit whitelist which is more secure.

Xyne · 2021-04-03 20:10:50

I agree that a whitelist is better for security. I honestly don't know if there was a specific reason for using a blacklist. The service file was provided with the request to add one to the package by the archiso devs and I didn't really review the various security settings in detail because I was unfamiliar with most of them. I am in favor of replacing the blacklist with the whitelist but I'll wait a few days to see if there are any objections in this thread before changing it and pushing it to the community repo. Feel free to ping me in a few days if I don't follow up.

nl6720 · 2021-04-05 16:52:05

Xyne, could you please reduce the generated mirrorlist boilerplate so that the first mirror is visible within the first 10 lines?

This would allow to view the first "Server" line by simply running:

$ head /etc/pacman.d/mirrorlist

Currently this requires passing the -n 11 option to head, since it defaults to 10 lines.

LaserEyess · 2021-04-07 00:09:10

Xyne wrote:

Feel free to ping me in a few days if I don't follow up.

I guess it can be considered "a few days" now. I would recommend

RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

Over the current line. That is to say, only allow ipv4, ipv6, and unix sockets. The last one should allow systemd-resolved, as well as tor I think, to work properly with reflector.

Xyne · 2021-04-09 22:57:37

Changed to whitelist, will release shortly.

@nl6720
What's the significance of 10 lines? I could reformat the metadata header but all that's going to change is the number after -n in your command, no? I could add an option to skip the header (e.g. "--no-header") but it's already possible to achieve the same thing with a pipe (e.g. | grep '^Server =').

I'm not opposed to either. I'm just trying to understand the problem.

nl6720 · 2021-04-10 06:55:37

I wouldn't really call it a problem.

The significance of 10 lines is that head defaults to 10 lines, so no -n option would be required.
It would just work with:

head /etc/pacman.d/mirrorlist

no-cheating · 2021-04-14 11:39:36

LaserEyess wrote:

I have a suggestion for the service provided for reflector. Currently the line
RestrictAddressFamilies=~AF_AX25 AF_IPX AF_APPLETALK AF_X25 AF_DECnet AF_KEY AF_NETLINK AF_PACKET AF_RDS AF_PPPOX AF_LLC AF_IB AF_MPLS AF_CAN AF_TIPC AF_BLUETOOTH AF_ALG AF_VSOCK AF_KCM AF_UNIX AF_XDP
Is quite restrictive but also a blacklist. The problem with it being so restrictive is that it breaks DNS when you use systmed-resolved without a stub resolver. What is happening is that calls to getent for nss_resolve.so require dbus which requires AF_UNIX, which is explicitly blacklisted. Because I don't have any other nameservers specifed in /etc/resolv.conf getent will return that DNS is unavailable, despite it being available. The second problem, IMO, is that a blacklist is inherently less secure than a whitelist. It is clear this directive was chosen to severely limit the damage it can do as root, but it can be done better with using the same directive but as a whitelist. Currently I have
[Service]
RestrictAddressFamilies=
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
In my override file for the service, which both fixes the issue of /etc/resolv.conf not existing and makes the address families an explicit whitelist which is more secure.

I have the same problem with RestrictAddressFamilies as LaserEyess. Whenever I run reflector.service I'm getting an error:

error: failed to retrieve mirrorstatus data: URLError: <urlopen error [Errno -3] Temporary failure in name resolution>

After some debugging I found out that if I remove RestrictAddressFamilies line in reflector.servcie everything works fine. As LaserEyess I'm using systemd-resolved.

Xyne wrote:

I agree that a whitelist is better for security. I honestly don't know if there was a specific reason for using a blacklist. The service file was provided with the request to add one to the package by the archiso devs and I didn't really review the various security settings in detail because I was unfamiliar with most of them. I am in favor of replacing the blacklist with the whitelist but I'll wait a few days to see if there are any objections in this thread before changing it and pushing it to the community repo. Feel free to ping me in a few days if I don't follow up.

Great to hear it's getting resolved.

Last edited by no-cheating (2021-04-14 11:41:18)

JustSomeGeek · 2021-05-02 12:48:15

DoXer wrote:

I get the following error after starting Reflector as a service:

# systemctl status reflector.service
● reflector.service - Refresh Pacman mirrorlist with Reflector.
     Loaded: loaded (/usr/lib/systemd/system/reflector.service; enabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Wed 2021-02-10 20:25:52 CET; 10s ago
TriggeredBy: ● reflector.timer
       Docs: https://wiki.archlinux.org/index.php/Reflector
    Process: 3014 ExecStart=/usr/bin/reflector @/etc/xdg/reflector/reflector.conf (code=exited, status=226/NAMESPACE)
   Main PID: 3014 (code=exited, status=226/NAMESPACE)

Feb 10 20:25:52 TST systemd[1]: Starting Refresh Pacman mirrorlist with Reflector....
Feb 10 20:25:52 TST systemd[3014]: reflector.service: Failed to set up mount namespacing: /run/systemd/unit-root/: Input/output error
Feb 10 20:25:52 TST systemd[3014]: reflector.service: Failed at step NAMESPACE spawning /usr/bin/reflector: Input/output error
Feb 10 20:25:52 TST systemd[1]: reflector.service: Main process exited, code=exited, status=226/NAMESPACE
Feb 10 20:25:52 TST systemd[1]: reflector.service: Failed with result 'exit-code'.
Feb 10 20:25:52 TST systemd[1]: Failed to start Refresh Pacman mirrorlist with Reflector..

The command

/usr/bin/reflector @/etc/xdg/reflector/reflector.conf

runs without error.

Any hints to solve the issue are more then welcome

Just adding a "me too" to this, sorry!

running "sudo reflector --save /etc/pacman.d/mirrorlist --protocol https --country GB --latest 5 --sort age" OK
Using those same options in reflector.conf and running "sudo /usr/bin/reflector @/etc/xdg/reflector/reflector.conf" also OK
"sudo systemctl enable reflector.service" OK
"sudo systemctl start reflector.service" fails
"sudo systemctl status reflector.service" shows "reflector[88709]: error: failed to retrieve mirrorstatus data: URLError: <urlopen error [Errno -3] Temporary failure in name resolution>"

Using DHCP. Everything else so far seems to be OK.

thiagowfx · 2021-05-09 23:39:57

@JustSomeGeek Could you post:

stat /etc/resolv.conf
cat /etc/resolv.conf

I happen to have experienced the same issue and in my case the culprit was an empty (sans comments) /etc/resolv.conf file.

Assuming you're using systemd-resolved, you'll need to symlink /etc/resolv.conf to /run/systemd/resolve/stub-resolv.conf.

This is covered in the wiki: https://wiki.archlinux.org/title/Systemd-resolved#DNS.

I can't troubleshoot the root cause, but for some reason, when running under systemd (as of the current package), proper config of /etc/resolv.conf is needed for reflector. When running it stand-alone, /etc/resolv.conf is not needed.

Last edited by thiagowfx (2021-05-09 23:40:18)

thiagowfx · 2021-05-09 23:48:05

Sorry for the double post, but this is completely unrelated to my previous comment:

Feature Request: What do you think of adding a "Worldwide" option to the list of countries supported by reflector? Rationale:

pacman-mirrorlist comes with a Worldwide section. Currently, it looks like this:

## Worldwide
#Server = http://mirrors.evowise.com/archlinux/$repo/os/$arch
#Server = http://mirror.rackspace.com/archlinux/$repo/os/$arch
#Server = https://mirror.rackspace.com/archlinux/$repo/os/$arch

My use case would be to append the Worldwide list to the list of mirrors of a specific country, as a escape hatch in case of all mirrors of that country are down for some reason. For example:

Instead of

reflector --protocol https --country 'Canada'

One would use:

reflector --protocol https --country 'Canada,Worldwide'

The difference is that the latter would generate a traditional config of reflector with all Canadian mirrors, immediately followed by the Worldwide mirrors (consistently with pacman-mirrorlist). Note that, in the context above, "*" is not the same as "Worldwide". The latter is not meant to match any country, it is meant to be country agnostic.

Arch Linux

#126 2020-12-19 22:22:50

Re: Reflector reborn

#127 2020-12-20 22:32:59

Re: Reflector reborn

#128 2021-01-05 16:33:35

Re: Reflector reborn

#129 2021-01-07 12:29:11

Re: Reflector reborn

#130 2021-01-07 12:58:52

Re: Reflector reborn

#131 2021-01-07 13:08:06

Re: Reflector reborn

#132 2021-01-07 14:20:33

Re: Reflector reborn

#133 2021-01-07 15:23:35

Re: Reflector reborn

#134 2021-01-08 17:46:10

Re: Reflector reborn

#135 2021-01-10 00:17:08

Re: Reflector reborn

#136 2021-01-10 16:47:46

Re: Reflector reborn

#137 2021-01-10 18:46:08

Re: Reflector reborn

#138 2021-01-11 18:54:39

Re: Reflector reborn

#139 2021-02-10 19:29:23

Re: Reflector reborn

#140 2021-02-11 12:49:14

Re: Reflector reborn

#141 2021-04-03 19:52:14

Re: Reflector reborn

#142 2021-04-03 20:10:50

Re: Reflector reborn

#143 2021-04-05 16:52:05

Re: Reflector reborn

#144 2021-04-07 00:09:10

Re: Reflector reborn

#145 2021-04-09 22:57:37

Re: Reflector reborn

#146 2021-04-10 06:55:37

Re: Reflector reborn

#147 2021-04-14 11:39:36

Re: Reflector reborn

#148 2021-05-02 12:48:15

Re: Reflector reborn

#149 2021-05-09 23:39:57

Re: Reflector reborn

#150 2021-05-09 23:48:05

Re: Reflector reborn

Board footer