Reflector reborn

Franklin Yu · 2023-03-25 04:32:18

Feature request: minimize churn

I want to minimize churn by allowing any subset of top N (after whatever filter I specified). Let’s say N is 6, and the mirrors are A-F; I want Reflector to pick the top 3, let’s say A B C, but if the existing list is A B D (for example C suddenly gets a little faster), I want Reflector to keep the current mirror list of A B D. This way if C and D keep switching between 3rd and 4th place, it doesn’t change the mirror list as often.

This is a preliminary idea, because I don’t have good idea what behavior I want with “--sort”. Would love to learn what you think.

Trilby · 2023-03-25 14:19:24

Franklin, what benefit(s) would that feature provide?

In your example, if C/D keep switching places, they are very similar in the relevant metrics. So why would it matter if the mirror list changed? It could be seen as a "needless" change, but does that carry any cost at all?

Franklin Yu · 2023-03-25 19:10:13

In my case I use etckeeper to track changes in my “/etc/” directory. In theory I can also ignore this file, but it sounds like something I do want to track.

Trilby · 2023-03-25 20:23:06

Thanks that makes sense as a fair use-case for such a feature.

Xyne · 2023-04-10 20:00:17

Reflector would have to read and compare the generated mirrorlist to the user's current mirrorlist and then create some sort of similarity metric to determine if they are similar enough to ignore the new list. That is quite far out of the scope of reflector and will not be implemented.

You would write your own wrapper script to generate a new list in tmpfs, diff that with the existing file and update the file only if you consider the changes significant.

Alternatively, given that the file is dynamically generated, why not just stop tracking it? In what scenario would you need to retrieve an old mirrorlist when you could just generate a new one?

kplant · 2023-05-29 12:45:32

@Xyne looks like after the most recent repo changes, an update to how reflector is scoring mirrors is needed.

Currently DB_SUBPATH is 'community/os/x86_64/community.db' - this file is now only 45 bytes, could we change this to 'extra/os/x86_64/extra.db'?

Last edited by kplant (2023-05-29 13:33:49)

airbreather · 2023-06-18 10:46:03

kplant wrote:

@Xyne looks like after the most recent repo changes, an update to how reflector is scoring mirrors is needed.
Currently DB_SUBPATH is 'community/os/x86_64/community.db' - this file is now only 45 bytes, could we change this to 'extra/os/x86_64/extra.db'?

Just came here to post this as well. After making this change in my local /usr/lib/python3.11/site-packages/Reflector.py file, the results from Reflector are a lot better now.

Xyne · 2023-06-24 14:50:38

Sorry, I've been neglecting my packages lately. I need to figure out what happened to the site and then I'll update reflector.

edit
The site's back up and reflector has been updated.

Last edited by Xyne (2023-06-28 01:01:47)

xerus · 2023-08-04 21:10:04

It seems this thread is also the place for bugs/feature requests?

I have a permanent reflector conf in the xdg dir shared across my devices, which so far filtered by my current country to effectively limit their number.
But if I take my laptop to another country, I would have to adjust that.

How about an option to use the current country, estimated by geoip, or a rough geographical area as the current country might not have an own mirror (or just fall back to worldwide in that case)?
I don't think edge cases matter here, but I think somehow filtering by distance is a good way find fast mirrors without needing to query too many.

tkit · 2023-09-19 10:23:54

Description:
Reflector cannot find any mirrors for China

Additional info:
* package version(s)
```
➜ ~ LANG=C pacman -Qi reflector
Name : reflector
Version : 2023-1
Description : A Python 3 module and script to retrieve and filter the latest Pacman mirror list.
Architecture : any
URL : https://xyne.dev/projects/reflector
Licenses : GPL2
Groups : None
Provides : None
Depends On : python
Optional Deps : rsync: rate rsync mirrors [installed]
Required By : None
Optional For : None
Conflicts With : None
Replaces : None
Installed Size : 165.05 KiB
Packager : Xyne <xyne@archlinux.org>
Build Date : Wed Jun 28 08:57:34 2023
Install Date : Mon Jul 24 20:01:41 2023
Install Reason : Explicitly installed
Install Script : No
Validated By : Signature
* config and/or log files etc.
* link to upstream bug report, if any
```
Steps to reproduce:
```
➜ ~ reflector -c China
error: no mirrors found
➜ ~ reflector -c CN
error: no mirrors found
➜ ~ reflector -c JP
################################################################################
################# Arch Linux mirrorlist generated by Reflector #################
################################################################################

# With: reflector -c JP
# When: 2023-09-19 10:23:07 UTC
# From: https://archlinux.org/mirrors/status/json/
# Retrieved: 2023-09-19 10:23:07 UTC
# Last Check: 2023-09-19 10:16:15 UTC

Server = http://ftp.jaist.ac.jp/pub/Linux/ArchLi … o/os/$arch
Server = https://ftp.jaist.ac.jp/pub/Linux/ArchL … o/os/$arch
Server = rsync://ftp.jaist.ac.jp/pub/Linux/ArchLinux/$repo/os/$arch
Server = http://mirrors.cat.net/archlinux/$repo/os/$arch
Server = https://mirrors.cat.net/archlinux/$repo/os/$arch
Server = http://mirror.nishi.network/archlinux/$repo/os/$arch
Server = https://mirror.nishi.network/archlinux/$repo/os/$arch
Server = http://repo.jing.rocks/archlinux/$repo/os/$arch
Server = https://repo.jing.rocks/archlinux/$repo/os/$arch
Server = rsync://repo.jing.rocks/archlinux/$repo/os/$arch
➜ ~ reflector --list-countries | grep China
China CN 40
➜ ~ reflector --list-countries | awk 'NR==1 || /China/'
Country Code Count
China CN 40
➜ ~
```

Awebb · 2023-09-19 10:28:53

That is not reflector's fault:

https://archlinux.org/mirrors/status/#errorlog

tkit · 2023-09-19 10:37:18

Awebb wrote:

That is not reflector's fault:
https://archlinux.org/mirrors/status/#errorlog

All mirrors from China are down?

Awebb · 2023-09-19 18:09:08

tkit wrote:

Awebb wrote:
That is not reflector's fault:
https://archlinux.org/mirrors/status/#errorlog
All mirrors from China are down?

They were when we had the conversation earlier. Check again. reflector -c CN now delivers something and servers are up again in the list. Whatever happened there, it hit all mirrors in China.

Last edited by Awebb (2023-09-19 20:39:17)

jasonkhanlar · 2024-03-08 10:44:03

I am confused by

`reflector --delay 0.1 --latest 200 --protocol https --age 12 --sort rate --save /etc/pacman.d/mirrorlist.reflector` outputs 2 warnings:

[2024-03-08 04:58:56] WARNING: failed to rate http(s) download (https://mirrors.neusoft.edu.cn/archlinux/extra/os/x86_64/extra.db): Download timed out after 5 second(s).
[2024-03-08 05:00:01] WARNING: failed to rate http(s) download (https://mirror.cyberbits.asia/archlinux/extra/os/x86_64/extra.db): <urlopen error [Errno 101] Network is unreachable>

and the file contains those two domains as mirrors at the end of /etc/pacman.d/mirrorlist.reflector:
mirrors.neusoft.edu.cn and mirror.cyberbits.asia
and they are not commented out

If it is intentional to have those two mirrors included at the end, which according to data at https://archlinux.org/mirrors/status/ appear to fit the rest of the criteria in the command line, then there should be command line switch option to exclude entries that have warnings. I manually removed the entries myself for now.

Last edited by jasonkhanlar (2024-03-08 11:02:39)

Xyne · 2024-03-13 19:47:14

jasonkhanlar wrote:

]there should be command line switch option to exclude entries that have warnings. I manually removed the entries myself for now.

I've added it to the todo list.

kero · 2024-08-27 13:11:00

Hi,
For some unknown reason, when I try to run reflector via its service file (systemctl start reflector.service) AND the destination file has another name than "mirrorlist" (eg: /etc/pacman.d/mirrorlistbis), I get an error. Here is the journalctl relevant part:

août 27 14:37:03 kdesktop systemd[1]: Starting Refresh Pacman mirrorlist with Reflector....
août 27 14:37:08 kdesktop reflector[402978]: error: Read-only file system
août 27 14:37:08 kdesktop systemd[1]: reflector.service: Main process exited, code=exited, status=1/FAILURE
août 27 14:37:08 kdesktop systemd[1]: reflector.service: Failed with result 'exit-code'.

Notes:
- I CAN successfully use the reflector.service if I don't modifiy the --save option, leaving thes standard /etc/pacman.d/mirrorlist (which confirms that my filesystem isn't ReadOnly).
- I CAN successfully use reflector to write another file than "mirrorlist" (eg.: /etc/pacman.d/mirrorlistbis), but only if using directly the /usr/bin executable (which also confirms that my filesystem isn't ReadOnly).

So: I am doing something wrong or should I suspect a bug ? And in this case, where should I write ? The official page doesn't give informations about that.

Thank you.

Arch Linux

#176 2023-03-25 04:32:18

Re: Reflector reborn

#177 2023-03-25 14:19:24

Re: Reflector reborn

#178 2023-03-25 19:10:13

Re: Reflector reborn

#179 2023-03-25 20:23:06

Re: Reflector reborn

#180 2023-04-10 20:00:17

Re: Reflector reborn

#181 2023-05-29 12:45:32

Re: Reflector reborn

#182 2023-06-18 10:46:03

Re: Reflector reborn

#183 2023-06-24 14:50:38

Re: Reflector reborn

#184 2023-08-04 21:10:04

Re: Reflector reborn

#185 2023-09-19 10:23:54

Re: Reflector reborn

#186 2023-09-19 10:28:53

Re: Reflector reborn

#187 2023-09-19 10:37:18

Re: Reflector reborn

#188 2023-09-19 18:09:08

Re: Reflector reborn

#189 2024-03-08 10:44:03

Re: Reflector reborn

#190 2024-03-13 19:47:14

Re: Reflector reborn

#191 2024-08-27 13:11:00

Re: Reflector reborn

Board footer