You are not logged in.
Version 2020.8.19 adds systemd integration with a custom configuration file. See the man page, configuration file (/etc/xdg/reflector/reflector.conf) and wiki for details.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Should the folder /etc/xdg/reflector and file /etc/xdg/reflector/reflector.conf already exist? Because on my system they don't. And the wiki make it sound like they are already there.
EDIT: Nevermind, it seems I had a mirror that didn't get updates for quite awhile and I had an old version of reflector.
Last edited by starquake (2020-08-26 02:49:28)
Offline
Hello,
I have a problem with reflector.service. As root, when I run "systemctl start reflector.service", the service fails:
# systemctl start reflector.service
Job for reflector.service failed because the control process exited with error code.
See "systemctl status reflector.service" and "journalctl -xe" for details.
Here is the result of "systemctl status reflector.service" right after that:
# systemctl status reflector.service
● reflector.service - Refresh Pacman mirrorlist with Reflector.
Loaded: loaded (/usr/lib/systemd/system/reflector.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2020-11-02 22:28:14 +04; 8min ago
Docs: https://wiki.archlinux.org/index.php/Reflector
Main PID: 451347 (code=exited, status=1/FAILURE)
Nov 02 22:28:14 sheldon systemd[1]: Starting Refresh Pacman mirrorlist with Reflector....
Nov 02 22:28:14 sheldon reflector[451375]: error: Permission denied
Nov 02 22:28:14 sheldon systemd[1]: reflector.service: Main process exited, code=exited, status=1/FAILURE
Nov 02 22:28:14 sheldon systemd[1]: reflector.service: Failed with result 'exit-code'.
Nov 02 22:28:14 sheldon systemd[1]: Failed to start Refresh Pacman mirrorlist with Reflector..
But, if I run (as root) :
reflector --verbose --latest 5 --sort rate --save /etc/pacman.d/mirrorlist
then, the mirrorlist file is written correctly.
I have another Arch linux machine and the service is working perfectly. I don't understand.
Can someone help me please (answer below)?
EDIT: I found! I didn't remember I played with the owner and permissions of /etc/pacman.d/mirrorlist. Permissions where set as 755 and the file was owned by my usual user. I changed the owner to root and modified the permissions to 644 on this file and it works now. It is strange though because the service is run by root so it should be able to overwrite the file anyway. Well, that's fine now.
Last edited by berturion (2020-11-02 18:46:48)
Offline
Offline
What does the journal say?
My suspicion is a problem with your internet connectivity.
Inofficial first vice preseident of the Rust Evangelism Strike Force
Offline
What does the journal say?
My suspicion is a problem with your internet connectivity.
reflector[449]: error: failed to retrieve mirrorstatus data: URLError: <urlopen error [Errno -3] Temporary failure in name resolution>
That is what it says. Should I paste my whole journalctl?
But the network online target happened way before the reflector.service starts..
Last edited by riazufila (2020-12-02 13:43:36)
Offline
I'm not sure whether here is the right place for that, or not.
If not, you can also post a new topic on this here: https://bbs.archlinux.org/viewforum.php?id=18
Maybe some mod can clear this up.
In any case, the message confirms my suspicion.
Your DNS is not ready at the time reflector.service is being started.
Inofficial first vice preseident of the Rust Evangelism Strike Force
Offline
I'm not sure whether here is the right place for that, or not.
If not, you can also post a new topic on this here: https://bbs.archlinux.org/viewforum.php?id=18
Maybe some mod can clear this up.
In any case, the message confirms my suspicion.
Your DNS is not ready at the time reflector.service is being started.
Does this mean, network-online.target doesn't really help? Asking this cause reflector is set up with that and I can confirm that the network-online.target is satisfied before reflector.service is executed.
Offline
Probably. What service(s) do you use to connect to the network / internet?
I could think of NetworkManager being the problem, especially if NetworkManager-wait-online.service is not enabled.
Also you can increase the NM_ONLINE_TIMEOUT by systemd editing the unit.
That of course is, iff you use NetworkManager.
Inofficial first vice preseident of the Rust Evangelism Strike Force
Offline
Probably. What service(s) do you use to connect to the network / internet?
I could think of NetworkManager being the problem, especially if NetworkManager-wait-online.service is not enabled.
Also you can increase the NM_ONLINE_TIMEOUT by systemd editing the unit.
That of course is, iff you use NetworkManager.
Yes. I use NetworkManager. The wait service is turned on and enabled which I can confirm. And I've also increased the value to 60 from 30 but the problems still persists.
Last edited by riazufila (2020-12-02 15:19:20)
Offline
Following the discussion here, threading support has been removed from the mirror rating to avoid skewing results and the target file has been changed to the community database (5 MB). Rating mirrors by speed will now take a little longer but the results should be more reliable. As mentioned in that thread, I personally think ranking mirrors by speed is mostly a waste of time and bandwidth. If you want to increase download speeds, I strongly recommend powerpill.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
I wonder if this is a sane way of configuration and usage of reflector?
Since I travel quite I lot I prefer to manually run reflector depending on the change of location/country. For that I made my own script in /home/user/bin/syu:
#!/bin/bash
reflector --verbose --connection-timeout 1 --protocol https --country 'Austria,Belgium,Canada,Czechia,Finland,France,Germany,Hungary,Iceland,Italy,Netherlands,Norway,Slovakia,Slovenia,Sweden,Switzerland,United Kingdom' --age 24 --latest 7 --sort rate
yay -Syyu
Basically what I wanted to achieve:
1. connection-tiimeout: For all coming server checks (https,age,rate,..) limit the check timeout to 1sec. Everything that takes longer is probably slow.
2. protocol: From all available servers, remove all that don't support https
3. country: From all available servers, use only the ones I prefer. Those are mostly that countries that provide me with the fastest speeds
4. age: Use only mirrors that are up to date
5. latest: From the selection that is now left, select the 7 latest updated servers
6. sort: now test them for their speed and use the fastest one
I thought this would be the most resource friendly, fastest and most sane way of using reflector for my case?
Thanks!
Last edited by Utini (2020-12-06 10:16:31)
Setup 1: Thinkpad T14s G3, 14" FHD - R7 6850U - 32GB RAM - 2TB Solidigm P44 Pro NVME
Setup 2: Thinkpad X1E G1, 15.6" FHD - i7-8850H - 32GB RAM - NVIDIA GTX 1050Ti - 2x 1TB Samsung 970 Pro NVME
Accessories: Filco Majestouch TKL MX-Brown Mini Otaku, Benq XL2420T (144Hz), Lo(w)gitech G400, Puretrak Talent, Sennheiser HD800S + Meier Daccord FF + Meier Classic FF
Offline
Utini, that reflector use seems reasonable - though if you are scripting it, I'd expect there'd be a good way to determine what country you are currently in, and just use mirrors in that country. But much more importantly - and the reason I commented - DON'T use `-Syyu'. The additional 'y' either does absolutely nothing useful for you, or it just wastes more time, but it does pointlessly waste mirror server bandwidth. Admittedly this use of bandwidth is fairly trivial, but as there is absolutely no benefit to using the extra 'y', just don't: that is not the proper way to update a system.
"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman
Offline
schard wrote:Probably. What service(s) do you use to connect to the network / internet?
I could think of NetworkManager being the problem, especially if NetworkManager-wait-online.service is not enabled.
Also you can increase the NM_ONLINE_TIMEOUT by systemd editing the unit.
That of course is, iff you use NetworkManager.Yes. I use NetworkManager. The wait service is turned on and enabled which I can confirm. And I've also increased the value to 60 from 30 but the problems still persists.
I have exactly the same error message:
reflector[449]: error: failed to retrieve mirrorstatus data: URLError: <urlopen error [Errno -3] Temporary failure in name resolution>
However in my case this particular error only occurs if I try to run the reflector.service in a VirtualBox Guest Installation of ArchLinux. It doesn't occur if I run
systemctl start reflector
as root on the host of the VirtualBox Guest. I experimented a little bit with the file
/usr/lib/systemd/system/reflector.service
and found that removing
AF_UNIX
from
RestrictAddressFamilies=~AF_AX25 AF_IPX AF_APPLETALK AF_X25 AF_DECnet AF_KEY AF_NETLINK AF_PACKET AF_RDS AF_PPPOX AF_LLC AF_IB AF_MPLS AF_CAN AF_TIPC AF_BLUETOOTH AF_ALG AF_VSOCK AF_KCM AF_UNIX AF_XDP
makes the service work in the Archlinux Virtualbox Guest as well. Does anyone know what could be the reason for that? More importantly from a security standpoint what kind of attack becomes possible if one removes AF_UNIX from RestrictAddressFamilies deny list? Note that one has to call
systemctl daemon-reload
after making changes to reflector.service.
Offline
Thanks for the tool Xyne.
Attached below is a patch to restore the old download_timeout option -- I am sufficiently far from the highest-scoring mirrors that even sampling a small number of them for speed is unbearably slow without it.
The implementation is admittedly hacky, but it's better than nothing.
If this isn't the proper route to contribution, please say so.
--- Reflector.py 2020-12-03 00:00:00.000000000 +0000
+++ Reflector.py 2020-12-06 00:00:00.000000000 +0000
@@ -37,6 +37,7 @@
import shlex
import socket
import subprocess
+import signal
import sys
import tempfile
import time
@@ -59,6 +60,7 @@
MIRRORLIST_ENTRY_FORMAT = "Server = " + MIRROR_URL_FORMAT + "\n"
DEFAULT_CONNECTION_TIMEOUT = 5
+DEFAULT_DOWNLOAD_TIMEOUT = 5
DEFAULT_CACHE_TIMEOUT = 300
SORT_TYPES = {
@@ -131,6 +133,12 @@
# ------------------------------ Miscellaneous ------------------------------- #
+class SlowConn(Exception):
+ pass
+
+def dieIfSlow(sig, frame):
+ raise SlowConn()
+
def get_logger():
'''
Get the logger used by this module. Use this to be sure that the right logger
@@ -193,7 +201,8 @@
# ---------------------------------- Rating ---------------------------------- #
-def rate_rsync(db_url, connection_timeout=DEFAULT_CONNECTION_TIMEOUT):
+def rate_rsync(db_url, connection_timeout=DEFAULT_CONNECTION_TIMEOUT,
+ download_timeout=DEFAULT_DOWNLOAD_TIMEOUT):
'''
Download a database via rsync and return the time and rate of the download.
'''
@@ -203,6 +212,8 @@
'--contimeout={}'.format(connection_timeout),
db_url
]
+ signal.signal(signal.SIGALRM, dieIfSlow)
+ signal.alarm(download_timeout)
try:
with tempfile.TemporaryDirectory() as tmpdir:
time_0 = time.time()
@@ -217,16 +228,20 @@
)
ratio = size / time_delta
return time_delta, ratio
- except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError):
+ except (subprocess.CalledProcessError, subprocess.TimeoutExpired,
+ FileNotFoundError, SlowConn):
return 0, 0
-def rate_http(db_url, connection_timeout=DEFAULT_CONNECTION_TIMEOUT):
+def rate_http(db_url, connection_timeout=DEFAULT_CONNECTION_TIMEOUT,
+ download_timeout=DEFAULT_DOWNLOAD_TIMEOUT):
'''
Download a database via any protocol supported by urlopen and return the time
and rate of the download.
'''
req = urllib.request.Request(url=db_url)
+ signal.signal(signal.SIGALRM, dieIfSlow)
+ signal.alarm(download_timeout)
try:
with urllib.request.urlopen(req, None, connection_timeout) as handle:
time_0 = time.time()
@@ -234,11 +249,13 @@
time_delta = time.time() - time_0
ratio = size / time_delta
return time_delta, ratio
- except (OSError, urllib.error.HTTPError, http.client.HTTPException):
+ except (OSError, urllib.error.HTTPError, http.client.HTTPException,
+ SlowConn):
return 0, 0
-def rate(mirrors, connection_timeout=DEFAULT_CONNECTION_TIMEOUT):
+def rate(mirrors, connection_timeout=DEFAULT_CONNECTION_TIMEOUT,
+ download_timeout=DEFAULT_DOWNLOAD_TIMEOUT):
'''
Rate mirrors by timing the download of the community repo's database from
each one.
@@ -265,9 +282,11 @@
scheme = urllib.parse.urlparse(url).scheme
if scheme == 'rsync':
- time_delta, ratio = rate_rsync(db_url, connection_timeout)
+ time_delta, ratio = rate_rsync(db_url, connection_timeout,
+ download_timeout)
else:
- time_delta, ratio = rate_http(db_url, connection_timeout)
+ time_delta, ratio = rate_http(db_url, connection_timeout,
+ download_timeout)
kibps = ratio / 1024.0
logger.info(fmt.format(url, kibps, time_delta))
@@ -468,11 +487,13 @@
def __init__(
self,
connection_timeout=DEFAULT_CONNECTION_TIMEOUT,
+ download_timeout=DEFAULT_DOWNLOAD_TIMEOUT,
cache_timeout=DEFAULT_CACHE_TIMEOUT,
min_completion_pct=1.0,
url=URL
):
self.connection_timeout = connection_timeout
+ self.download_timeout = download_timeout
self.cache_timeout = cache_timeout
self.min_completion_pct = min_completion_pct
self.url = url
@@ -601,10 +622,10 @@
help='The number of seconds to wait before a connection times out. Default: %(default)s'
)
-# parser.add_argument(
-# '--download-timeout', type=int, metavar='n',
-# help='The number of seconds to wait before a download times out. The threshold is checked after each chunk is read, so the actual timeout may take longer.'
-# )
+ parser.add_argument(
+ '--download-timeout', type=int, metavar='n', default=DEFAULT_DOWNLOAD_TIMEOUT,
+ help='The number of seconds to wait before a download times out. Default: %(default)s'
+ )
parser.add_argument(
'--list-countries', action=ListCountries, nargs=0,
@@ -763,7 +784,7 @@
if not mirrorstatus:
mirrorstatus = MirrorStatus(
connection_timeout=options.connection_timeout,
- # download_timeout=options.download_timeout,
+ download_timeout=options.download_timeout,
cache_timeout=options.cache_timeout,
min_completion_pct=(options.completion_percent / 100.),
url=options.url
Last edited by Gesh (2020-12-06 20:11:22)
Offline
after making changes to reflector.service.[/quote]
Might I suggest something that will survive an update (if this isn't fixed in the next update of reflector)? Copy /usr/lib/systemd/system/reflector.service to /etc/systemd/system/reflector.service and perform the modification on that, run a systemctl daemon-reload then restart reflector and you should be golden.
Offline
Oooor you could both look at "systemctl edit reflector.service" whcih will guide you in using drop-in overrides like you're supposed to.
Managing AUR repos The Right Way -- aurpublish (now a standalone tool)
Offline
Oooor you could both look at "systemctl edit reflector.service" whcih will guide you in using drop-in overrides like you're supposed to.
Completely forgot about this, yes, definitely this way is best. Especially if you've set your preferred $EDITOR.
Offline
@Utini
If you are using the latest 7, using the age criterion before it is probably useless. The only way it would change anything is if there were fewer than 7 mirrors that had synced in the last 24 hours, but that is unlikely with a list that long.
@archlinux20130905
The current service file is the one submitted by the Arch iso devs. I admit that I didn't bother researching all of the options and the reason for their inclusion, so I can't answer your question about which vulnerabilities its removal would create. If there is a good argument for removing it (utility for users vs security vulnerabilities) then I will consider it, but given that it seems to be limited to a few use cases, I recommend using "systemctl edit reflector.service" as suggested by eschwartz.
@Gesh
Thanks for the patch. I've applied it with some changes. It revealed an existing bug in the way some options were passed through to the download functions (they weren't ). I've also created a context manager to time downloads and restore previous alarms when done. I'll upload it shortly.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Recently got tired of editing the mirrorlist every time I do an update, so thanks a lot for the script.
I changed the script a bit, I'm here to share my changes. The '--sort country' option isn't very useful right now, as it sorts countries by alphabetical order, which is never what you want. I added another option to move a selected country (or countries) to the top of the mirrorlist, when sorting.
I see other people have posted patches here, I hope that's the correct way to contribute to the tool.
diff --git a/Reflector.py b/Reflector.py
index 7b6b7ab..b21836f 100644
--- a/Reflector.py
+++ b/Reflector.py
@@ -225,13 +225,26 @@ class DownloadTimer():
# --------------------------------- Sorting ---------------------------------- #
-def sort(mirrors, by=None, **kwargs): # pylint: disable=invalid-name
+def sorting_key_gen(spec_items_priority, item):
+ """
+ Handles the generation of the key for sorting
+ """
+ if item not in spec_items_priority:
+ return len(spec_items_priority), item
+ else:
+ return spec_items_priority.index(item), item
+
+
+def sort(mirrors, by=None, spec_items_priority=None, **kwargs): # pylint: disable=invalid-name
'''
Sort mirrors by different criteria.
'''
# Ensure that "mirrors" is a list that can be sorted.
if not isinstance(mirrors, list):
mirrors = list(mirrors)
+ # ensure spec_items_priority is around
+ if spec_items_priority is None:
+ spec_items_priority = []
if by == 'age':
mirrors.sort(key=lambda m: m['last_sync'], reverse=True)
@@ -242,7 +255,7 @@ def sort(mirrors, by=None, **kwargs): # pylint: disable=invalid-name
else:
try:
- mirrors.sort(key=lambda m: m[by])
+ mirrors.sort(key=lambda m: sorting_key_gen(spec_items_priority, m[by]))
except KeyError as err:
raise MirrorStatusError('attempted to sort mirrors by unrecognized criterion: "{}"'.format(by)) from err
@@ -603,7 +616,7 @@ class MirrorStatus():
msf = MirrorStatusFilter(min_completion_pct=self.min_completion_pct, **kwargs)
yield from msf.filter_mirrors(mirrors)
- def sort(self, mirrors, **kwargs):
+ def sort(self, mirrors, countries_priority=None, **kwargs):
'''
Sort mirrors by various criteria.
'''
@@ -611,7 +624,7 @@ class MirrorStatus():
mirrors = self.get_mirrors()
kwargs.setdefault('connection_timeout', self.connection_timeout)
kwargs.setdefault('download_timeout', self.download_timeout)
- yield from sort(mirrors, **kwargs)
+ yield from sort(mirrors, spec_items_priority=countries_priority, **kwargs)
def rate(self, mirrors=None, **kwargs):
'''
@@ -717,6 +730,16 @@ def add_arguments(parser):
help='Sort the mirrorlist. {}.'.format(sort_help)
)
+ parser.add_argument(
+ '--priority_country',
+ help='List of countries to put on top of the list when sorting by country.'
+ 'Multiple countries may be selected using commas (e.g. "France,Germany") or by passing this option multiple times. '
+ 'Use "--list-countries" to see which are available.',
+ action='append',
+ dest='priority_countries',
+ metavar='<priority_country>'
+ )
+
parser.add_argument(
'--verbose', action='store_true',
help='Print extra information to STDERR. Only works with some options.'
@@ -832,7 +855,7 @@ def parse_args(args=None):
)
parser = add_arguments(parser)
options = parser.parse_args(args)
- for list_arg in ('countries', 'protocols'):
+ for list_arg in ('countries', 'protocols', 'priority_countries'):
setattr(options, list_arg, list(split_list_args(getattr(options, list_arg))))
return options
@@ -883,7 +906,10 @@ def process_options(options, mirrorstatus=None, mirrors=None):
mirrors = itertools.islice(mirrors, options.fastest)
if options.sort and not (options.sort == 'rate' and options.fastest):
- mirrors = mirrorstatus.sort(mirrors, by=options.sort)
+ if options.sort == 'country':
+ mirrors = mirrorstatus.sort(mirrors, by=options.sort, countries_priority=options.priority_countries)
+ else:
+ mirrors = mirrorstatus.sort(mirrors, by=options.sort)
if options.number:
mirrors = list(mirrors)[:options.number]
(Sorry, if I've messed something up, a bit new to sharing my code with others.)
Last edited by contra (2020-12-07 18:41:51)
Offline
@contra
I like the idea and the patch is fine, although I'll likely tweak the implementation a bit to handle both country names and country codes synonymously. I wonder if it's necessary though to add a new option. If a user has country preferences then I expect that they would filter by country prior to sorting. In that case, I would just use the arguments passed with "--country" to set the sort order.
Thoughts?
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Have the same "failed to retrieve mirrorstatus data" issue as @archlinux20130905 on a fresh Arch installation when running "systemctl start reflector". Same errors too. Runs ok from command line. Using iwd for wireless networking and it works well with all the other programs and services - doesn't look like connectivity issue.
Tried fixes suggested in this thread with no success.
Interestingly enough "systemctl start reflector" works well on Arch ISO when booted live. I tried diffing reflector.conf and reflector.service with versions that I have installed and found minor differences in .conf (applying those didn't help either) and no differences in .service file. ISO has some overrides applied to its configuration - applying the same archiso override didn't seem to help either.
I found fairly recent ticket on Arch bug tracker regarding this exact issue - FS#68557 : reflector - systemd service fails to start due to a name resolution failure], but it was closed by @Foxboron without any explanation stating only that it is not a bug. Looking a bit more on wiki pages and google didn't yield any results.
If this is a known "not a bug" issue -- can someone kindly point to explanation / resolution. I have trouble tracking further difference in configuration between Arch ISO and my install. This smells a lot like systemd permissions issue.
Offline
The linked "bug" (FS#68557) has a comment in the closing notes that the issue was resolved by the submitter after fixing an error in resolv.conf. If you're getting name resolution failures then I suspect that your system's network settings are misconfigured, especially if name resolution works on the live medium on the same hardware with the same network.
<edit>
Search the forum for help with name resolution errors. For example, I just saw this thread in the list of active topics.
</edit>
If you can reproduce the error in a terminal, post the exact command along with the downloaded mirrorstatus data ($XDG_CACHE_HOME/mirrorstatus.json, which defaults to ~/.cache/mirrorstatus.json) and I will examine it for possible bugs.
Last edited by Xyne (2020-12-17 22:13:01)
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Thank you @Xyne. I had a closer look at my DNS configuration per your suggestion. I use iwd with systemd-resolved, but not systemd-networkd (iwd has an option to be a simple network manager in simpler setups). When I installed systemd-resolved, I left the original /etc/resolv.conf untouched. Like I mentioned above - DNS name resolution worked well for me everywhere. If you leave the original /etc/resolv.conf around, documentation says it makes DNS config management more backwards compatible. It worked in my case, so I didn't need to change it.
This time I went and replaced /etc/resolf.conf with a link as per documentation in wiki. After that systemctl start reflector began working with no issues. Need to dig deeper about what the difference is - still not sure why reflector.service wouldn't work in that backwards compatible mode. May be this is because I don't run systemd-networkd (which has some magic when working with systemd-resolved in tandem. Arch ISO has both enabled and may be that is why reflector.service doesn't have issues there).
Thank you for your help!
Last edited by romstor (2020-12-18 00:35:59)
Offline
When I installed systemd-resolved, I left the original /etc/resolv.conf untouched. Like I mentioned above - DNS name resolution worked well for me everywhere. If you leave the original /etc/resolv.conf around, documentation says it makes DNS config management more backwards compatible.
It was bound to break something eventually. There are software that perform domain name resolution themselves using the name servers from /etc/resolv.conf instead of using glibc provided functions (which would go through nss-resolve because of /etc/nsswitch.conf).
https://wiki.archlinux.org/index.php/Sy … solved#DNS says that leaving /etc/resolv.conf as a file is "less disruptive" in the context that if something else is managing it. If no other software is managing it or you're not managing it manually, then you'll want to symlink it to /run/systemd/resolve/stub-resolv.conf (or /run/systemd/resolve/resolv.conf if the software that reads /etc/resolv.conf insists on the listed name servers being real DNS servers).
Offline