Reflector reborn

JustSomeGeek · 2021-05-11 12:07:40

thiagowfx wrote:

@JustSomeGeek Could you post:
stat /etc/resolv.conf
cat /etc/resolv.conf
I happen to have experienced the same issue and in my case the culprit was an empty (sans comments) /etc/resolv.conf file.
Assuming you're using systemd-resolved, you'll need to symlink /etc/resolv.conf to /run/systemd/resolve/stub-resolv.conf.
This is covered in the wiki: https://wiki.archlinux.org/title/Systemd-resolved#DNS.
I can't troubleshoot the root cause, but for some reason, when running under systemd (as of the current package), proper config of /etc/resolv.conf is needed for reflector. When running it stand-alone, /etc/resolv.conf is not needed.

stat /etc/resolv.conf
File: /etc/resolv.conf
Size: 65 Blocks: 8 IO Block: 4096 regular file
Device: 1ah/26d Inode: 479 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2021-05-10 20:04:39.527080221 +0100
Modify: 2021-01-19 01:32:10.000000000 +0000
Change: 2021-05-09 13:44:07.680347807 +0100
Birth: 2021-05-09 13:44:07.680347807 +0100

cat /etc/resolv.conf
# Resolver configuration file.
# See resolv.conf(5) for details.

I use DHCPD as client, and not had any apparent DNS issues, so never thought to look at this, as the direct command line invocation works fine. I'll have a read, and give your solution a try. I'm guessing it's the same cause, as my resolv.conf is empty. Thanks!

EDIT: That seems to have done it. Thanks muchly! :-D

Last edited by JustSomeGeek (2021-05-11 12:20:12)

Xyne · 2021-05-31 02:51:11

thiagowfx wrote:

Feature Request: What do you think of adding a "Worldwide" option to the list of countries supported by reflector? Rationale:
pacman-mirrorlist comes with a Worldwide section. Currently, it looks like this:
## Worldwide
#Server = http://mirrors.evowise.com/archlinux/$repo/os/$arch
#Server = http://mirror.rackspace.com/archlinux/$repo/os/$arch
#Server = https://mirror.rackspace.com/archlinux/$repo/os/$arch

Sorry, I missed your post (the one after it started a new page). The Mirror Status server's JSON response sets an empty string as the country and country code for the "worldwide" servers. You can therefore include them by passing an empty country code to reflector. To use your example, replace "Worldwide" with the empty string (but keep the comma):

reflector --protocol https --country 'Canada,'

I could modify reflector to replace "Worldwide" with the empty string but I prefer to avoid hard-coded keywords that may change in the future.

thiagowfx · 2021-06-06 02:10:16

Thanks Xyne, that makes sense. There's no need to modify reflector for that, it's just a matter of documentation. I've gone ahead and added it as a Tip to the wiki (https://wiki.archlinux.org/title/Reflector#Examples). Dunno if it's worth to add that to the man page as well, I'll leave that to your judgment.

Last edited by thiagowfx (2021-06-06 02:27:23)

pyfisch · 2021-07-08 10:36:38

Hi Xyne,

I'm learning about the systemd unit hardening settings right now. Reflector has a comprehensive list of settings but I wonder if it can be made somewhat simpler:

CapabilityBoundingSet=~CAP_SETUID CAP_SETGID CAP_SETPCAP CAP_SYS_ADMIN CAP_SYS_PTRACE CAP_CHOWN CAP_FSETID CAP_SETFCAP CAP_DAC_OVERRIDE CAP_DAC_READ_SEARCH CAP_FOWNER CAP_IPC_OWNER CAP_NET_ADMIN CAP_SYS_TIME CAP_AUDIT_CONTROL CAP_AUDIT_READ CAP_AUDIT_WRITE CAP_KILL CAP_NET_BIND_SERVICE CAP_NET_BROADCAST CAP_NET_RAW CAP_SYS_NICE CAP_SYS_RESOURCE CAP_MAC_ADMIN CAP_MAC_OVERRIDE CAP_SYS_BOOT CAP_LINUX_IMMUTABLE CAP_IPC_LOCK CAP_SYS_CHROOT CAP_BLOCK_SUSPEND CAP_LEASE CAP_SYS_PACCT CAP_SYS_TTY_CONFIG CAP_WAKE_ALARM

Instead of enumerating all capabilities that reflector doesn't one can write just CapabilityBoundingSet= as reflector doesn't need any capabilities. In addition this is future-proof against capabilities that are added to new kernel releases.

RestrictAddressFamilies=~AF_AX25 AF_IPX AF_APPLETALK AF_X25 AF_DECnet AF_KEY AF_NETLINK AF_PACKET AF_RDS AF_PPPOX AF_LLC AF_IB AF_MPLS AF_CAN AF_TIPC AF_BLUETOOTH AF_ALG AF_VSOCK AF_KCM AF_UNIX AF_XDP

Here the line RestrictAddressFamilies=AF_INET AF_INET6 is sufficient and again future-proofs against new address families.

(Right now reflector runs as root, one can also use DynamicUser= and grant AmbientCapabilities=CAP_DAC_OVERRIDE to allow editing the mirrorlist file. Not sure if this is more secure/better.)

Thanks for providing reflector to the community!

Xyne · 2021-07-08 20:04:23

I've updated CapabilityBoundingSet to a whitelist. RestrictAddressFamilies had already been updated to a whitelist (with the addition of AF_UNIX, but maybe that's superfluous), but I forgot to push it after letting it sit in my repo.

I don't really have an opinion about switching to the dynamic user with CAP_DAC_OVERRIDE. My understanding is that the read and write paths are already locked down so it shouldn't make a difference but I understand that conceptually it's better to avoid running as root. I'm open to changing that too.

If you have any good sources beyond that systemd man pages for learning about hardening settings, please share. I need to dive into it seriously at some point instead of just dabbling and re-using submitted settings.

pyfisch · 2021-07-09 08:55:18

> I don't really have an opinion about switching to the dynamic user with CAP_DAC_OVERRIDE. My understanding is that the read and write paths are already locked down so it shouldn't make a difference but I understand that conceptually it's better to avoid running as root. I'm open to changing that too.

I agree, I don't think there is a difference because the paths are locked down. So I would keep it with root for now.

> If you have any good sources beyond that systemd man pages for learning about hardening settings, please share. I need to dive into it seriously at some point instead of just dabbling and re-using submitted settings.

No I mainly used the man pages and systemd-analyze security. I found the analyze command useful as it shows me which settings are already good and where it is important to improve. However I am unsure if the suggested options aren't redundant: for example most services shouldn't change the system clock. If I have DynamicUser=true or specified a regular account the service already can't modify the clock. Then systemd-analyze suggests to add CapabilityBoundingSet=~CAP_SYS_TIME, SystemCallFilter=~@clock and then it still wants me to add ProtectClock=true. I know that these options restrict access to the clock in different ways but as a user of systemd I don't really want to care.

I found a few tutorials but I think most are for an older version of systemd (with fewer options) or contain errors. I looked at a few ArchLinux service files but unfortunately most don't make use of the new settings yet.

adventurer · 2021-07-09 12:01:08

Xyne wrote:

If you have any good sources beyond that systemd man pages for learning about hardening settings, please share.

https://gist.github.com/ageis/f5595e59b … 25a323db04

https://wiki.archlinux.org/title/User:N … sandboxing

https://github.com/krathalan/systemd-sa … guide.conf

https://www.redhat.com/sysadmin/mastering-systemd

https://www.digitalocean.com/community/ … untu-20-04

https://web.archive.org/web/20210430150 … stemd.html

Xyne · 2021-07-10 00:32:22

adventurer wrote:

Xyne wrote:
If you have any good sources beyond that systemd man pages for learning about hardening settings, please share.
https://gist.github.com/ageis/f5595e59b … 25a323db04
https://wiki.archlinux.org/title/User:N … sandboxing
https://github.com/krathalan/systemd-sa … guide.conf
https://www.redhat.com/sysadmin/mastering-systemd
https://www.digitalocean.com/community/ … untu-20-04
https://web.archive.org/web/20210430150 … stemd.html

Thanks!

alex.forencich · 2021-08-25 01:03:07

On the subject of mirror speed testing; I think a reasonable compromise is to ping all of the servers in parallel, throw out all servers that timed out, then speed test the remainder sequentially. It seems that it is common for a number of servers to time out, and waiting 5 seconds for each of them is rather annoying. Sending (some of) the pings out in parallel should have little effect on the accuracy of the latency figures.

digitalone · 2021-10-23 08:28:35

Can't use it as service on my system.

sudo reflector --save /etc/pacman.d/mirrorlist --protocol https --country AU,BA,HR,FR,DE,IT,MC,RS,SI,CH --sort score --number 5

from command line works, but as service it fails:

systemd[1]: Starting Refresh Pacman mirrorlist with Reflector....
reflector[3466]: error: Permission denied
systemd[1]: reflector.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: reflector.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Refresh Pacman mirrorlist with Reflector..

Xyne · 2021-11-12 01:24:39

digitalone wrote:

reflector[3466]: error: Permission denied

What are the permissions on /etc/pacman.d/mirrorlist, /etc/xdg/reflector/reflector.conf and their parent directories?
What are the contents of /etc/xdg/reflector/reflector.conf ?
Is there anything more in the output of journalctl for the service?

JKMooney · 2021-11-24 00:02:18

Forgive me if this has already been asked but I'm struggling to find this in the ArchWiki, man pages, or reflector website. How is "Server Score" computed?

V1del · 2021-11-24 07:55:28

See the Mirror status page

Mirror Status wrote:

Mirror Score: A very rough calculation for ranking mirrors. It is currently calculated as (hours delay + average duration + standard deviation) / completion percentage. Lower is better.

ron9 · 2021-12-12 19:49:28

I still got this:

Traceback (most recent call last):
File "/usr/bin/reflector", line 3, in <module>
import Reflector
ImportError: No module named Reflector

Reflector version 2021.7.8-1. Am I missing someting? It worked yesterday..
.

Xyne · 2021-12-12 21:09:07

edit I'm assuming that you've somehow installed the latest version of Python before the official release. Whatever happened, it seems that you have a mismatch between the version of Python on your system and the one used to package Reflector. A simple rebuild of the reflector package should fix it.

Upgrade to version 2021.7.8-2 which has been built against the latest python package. They should have been installed together. Did you do a partial upgrade ?

Last edited by Xyne (2021-12-12 21:29:41)

ron9 · 2021-12-12 21:41:57

Thank you @Xyne. Didn't have community-testing enabled. Now it work again.

mass_t · 2022-01-31 05:56:07

reflector --sort rate --protocol https
[2022-01-31 08:54:26] WARNING: failed to rate http(s) download (https://mirror.csclub.uwaterloo.ca/arch … mmunity.db): Download timed out after 5 second(s).
[2022-01-31 08:54:32] WARNING: failed to rate http(s) download (https://mirror.umd.edu/archlinux/commun … mmunity.db): Download timed out after 5 second(s).
[2022-01-31 08:54:53] WARNING: failed to rate http(s) download (https://mirrors.lug.mtu.edu/archlinux/c … mmunity.db): Download timed out after 5 second(s).
[2022-01-31 08:55:05] WARNING: failed to rate http(s) download (https://mirrors.kernel.org/archlinux/co … mmunity.db): Download timed out after 5 second(s).
[2022-01-31 08:55:10] WARNING: failed to rate http(s) download (https://ftp.rnl.tecnico.ulisboa.pt/pub/ … mmunity.db): Download timed out after 5 second(s).
[2022-01-31 08:55:18] WARNING: failed to rate http(s) download (https://ftp.jaist.ac.jp/pub/Linux/ArchL … mmunity.db): Download timed out after 5 second(s).
^C%

Xyne · 2022-01-31 07:58:08

mass_t wrote:

reflector --sort rate --protocol https
[2022-01-31 08:54:26] WARNING: failed to rate http(s) download (https://mirror.csclub.uwaterloo.ca/arch … mmunity.db): Download timed out after 5 second(s).
[2022-01-31 08:54:32] WARNING: failed to rate http(s) download (https://mirror.umd.edu/archlinux/commun … mmunity.db): Download timed out after 5 second(s).
[2022-01-31 08:54:53] WARNING: failed to rate http(s) download (https://mirrors.lug.mtu.edu/archlinux/c … mmunity.db): Download timed out after 5 second(s).
[2022-01-31 08:55:05] WARNING: failed to rate http(s) download (https://mirrors.kernel.org/archlinux/co … mmunity.db): Download timed out after 5 second(s).
[2022-01-31 08:55:10] WARNING: failed to rate http(s) download (https://ftp.rnl.tecnico.ulisboa.pt/pub/ … mmunity.db): Download timed out after 5 second(s).
[2022-01-31 08:55:18] WARNING: failed to rate http(s) download (https://ftp.jaist.ac.jp/pub/Linux/ArchL … mmunity.db): Download timed out after 5 second(s).
^C%

You're trying to rate over 678 mirrors. Of course some of them will time out and the whole process will take ages. Don't do that. Use some filter options and then rate the shortlisted servers, e.g.

reflector --latest 5 --protocol https --sort rate

Throw in a country filter too to limit the results to servers near you.

Obistron · 2022-07-03 10:49:56

Hi guys I have a problem with reflector.
I noticed that some times I have no problems, other times when I update the system (to update I always use the command: yay), after all the update packages are downloaded, so it should just install, I get an error like this:

error: unable to download package 'npm-8.11.0-1-any.pkg.tar.zst.sig' from archlinux.thaller.ws:
warning: some files cannot be downloaded
error: unable to perform the requested operation (unexpected error)
Errors occurred, no packages were updated.

If after these errors I try again to type yay, the packages are installed without errors.

Typing the command systemctl --failed I got this:
UNIT LOAD ACTIVE SUB DESCRIPTION
● reflector.service loaded failed failed Refresh Pacman mirrorlist with Reflector.

This is my reflector.conf : https://pastebin.com/qLswd2ND

PS Sometimes even without errors in systemctl I have the same problem

What can I do?

Xyne · 2022-07-03 11:04:57

When Reflector fails, what is the error message ("systemctl status reflector.service" or "journalctl -xeu reflector.service")?

There's nothing obviously wrong with your reflector setup. If reflector randomly fails to fetch the mirrorstatus page and pacman downloads randomly fail from different up-to-date servers then it seems that you have an unreliable network connection.

Obistron · 2022-07-03 18:28:28

Xyne wrote:

When Reflector fails, what is the error message ("systemctl status reflector.service" or "journalctl -xeu reflector.service")?
There's nothing obviously wrong with your reflector setup. If reflector randomly fails to fetch the mirrorstatus page and pacman downloads randomly fail from different up-to-date servers then it seems that you have an unreliable network connection.

Output of systemctl status reflector.service: https://pastebin.com/hc5REX3V

But I don't understand what the connection problem may be, because the packages are downloaded very quickly
There is the problem after downloading them and before installation

Last edited by Obistron (2022-07-03 18:34:06)

Xyne · 2022-07-03 19:56:16

Obistron wrote:

Output of systemctl status reflector.service: https://pastebin.com/hc5REX3V
lug 02 16:00:49 ArchSte reflector[505]: error: failed to retrieve mirrorstatus data: URLError: <urlopen error [Errno -3] Temporary failure in name resolution>

Download speed is independent of host name resolution. The error indicates a problem with your DNS. If you are not running a local DNS server then this is a problem with your internet service provider's DNS configuration. Some providers have temporary problems that resolve themselves once they have finished propagating all of their DNS records.

You can temporarily use a third-party DNS provider by adding a nameserver to /etc/resolv.conf. Here are some examples (uncomment the nameserver line of your choice).

# DNS.WATCH
# nameserver 84.200.69.80
# nameserver 84.200.70.40

# CloudFlare
# nameserver 1.1.1.1
# nameserver 1.0.0.1

# Google
# nameserver 8.8.8.8
# nameserver 8.8.4.4

Keep in mind that a DNS query will be sent to the selected server for every single website that you visit. That doesn't really matter when you are using your ISP's DNS servers because your ISP already sees all of your connections. It's up to you to choose which third party is the lesser evil. You should be able to find other public DNS servers with an online search.

You can also try setting up your own local DNS server with something like unbound but that requires a bit more work than temporarily changing the name server.

Obistron · 2022-07-04 13:02:38

Xyne wrote:

Obistron wrote:
Output of systemctl status reflector.service: https://pastebin.com/hc5REX3V
lug 02 16:00:49 ArchSte reflector[505]: error: failed to retrieve mirrorstatus data: URLError: <urlopen error [Errno -3] Temporary failure in name resolution>
Download speed is independent of host name resolution. The error indicates a problem with your DNS. If you are not running a local DNS server then this is a problem with your internet service provider's DNS configuration. Some providers have temporary problems that resolve themselves once they have finished propagating all of their DNS records.
You can temporarily use a third-party DNS provider by adding a nameserver to /etc/resolv.conf. Here are some examples (uncomment the nameserver line of your choice).
# DNS.WATCH
# nameserver 84.200.69.80
# nameserver 84.200.70.40

# CloudFlare
# nameserver 1.1.1.1
# nameserver 1.0.0.1

# Google
# nameserver 8.8.8.8
# nameserver 8.8.4.4
Keep in mind that a DNS query will be sent to the selected server for every single website that you visit. That doesn't really matter when you are using your ISP's DNS servers because your ISP already sees all of your connections. It's up to you to choose which third party is the lesser evil. You should be able to find other public DNS servers with an online search.
You can also try setting up your own local DNS server with something like unbound but that requires a bit more work than temporarily changing the name server.

At home I have a DNS server with pihole, maybe in some adlist there are some packages url.
Now I try to take a look at the pihole logs. Thanks for your help

Fixxer · 2022-12-29 20:39:57

@Xyne

I have a question, if could be possible to implement option to exclude one or more countries by negating (!) or separate option for exclude country by code?

Last edited by Fixxer (2022-12-29 21:56:52)

Xyne · 2023-01-02 20:32:53

Fixxer wrote:

@Xyne
I have a question, if could be possible to implement option to exclude one or more countries by negating (!) or separate option for exclude country by code?

Sure. I'll add it to the todo list.

Arch Linux

#151 2021-05-11 12:07:40

Re: Reflector reborn

#152 2021-05-31 02:51:11

Re: Reflector reborn

#153 2021-06-06 02:10:16

Re: Reflector reborn

#154 2021-07-08 10:36:38

Re: Reflector reborn

#155 2021-07-08 20:04:23

Re: Reflector reborn

#156 2021-07-09 08:55:18

Re: Reflector reborn

#157 2021-07-09 12:01:08

Re: Reflector reborn

#158 2021-07-10 00:32:22

Re: Reflector reborn

#159 2021-08-25 01:03:07

Re: Reflector reborn

#160 2021-10-23 08:28:35

Re: Reflector reborn

#161 2021-11-12 01:24:39

Re: Reflector reborn

#162 2021-11-24 00:02:18

Re: Reflector reborn

#163 2021-11-24 07:55:28

Re: Reflector reborn

#164 2021-12-12 19:49:28

Re: Reflector reborn

#165 2021-12-12 21:09:07

Re: Reflector reborn

#166 2021-12-12 21:41:57

Re: Reflector reborn

#167 2022-01-31 05:56:07

Re: Reflector reborn

#168 2022-01-31 07:58:08

Re: Reflector reborn

#169 2022-07-03 10:49:56

Re: Reflector reborn

#170 2022-07-03 11:04:57

Re: Reflector reborn

#171 2022-07-03 18:28:28

Re: Reflector reborn

#172 2022-07-03 19:56:16

Re: Reflector reborn

#173 2022-07-04 13:02:38

Re: Reflector reborn

#174 2022-12-29 20:39:57

Re: Reflector reborn

#175 2023-01-02 20:32:53

Re: Reflector reborn

Board footer