You are not logged in.

#176 2023-03-25 04:32:18

Franklin Yu
Member
Registered: 2022-11-14
Posts: 4

Re: Reflector reborn

Feature request: minimize churn

I want to minimize churn by allowing any subset of top N (after whatever filter I specified). Let’s say N is 6, and the mirrors are A-F; I want Reflector to pick the top 3, let’s say A B C, but if the existing list is A B D (for example C suddenly gets a little faster), I want Reflector to keep the current mirror list of A B D. This way if C and D keep switching between 3rd and 4th place, it doesn’t change the mirror list as often.

This is a preliminary idea, because I don’t have good idea what behavior I want with “--sort”. Would love to learn what you think.

Offline

#177 2023-03-25 14:19:24

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,447
Website

Re: Reflector reborn

Franklin, what benefit(s) would that feature provide?

In your example, if C/D keep switching places, they are very similar in the relevant metrics.  So why would it matter if the mirror list changed?  It could be seen as a "needless" change, but does that carry any cost at all?


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#178 2023-03-25 19:10:13

Franklin Yu
Member
Registered: 2022-11-14
Posts: 4

Re: Reflector reborn

In my case I use etckeeper to track changes in my “/etc/” directory. In theory I can also ignore this file, but it sounds like something I do want to track.

Offline

#179 2023-03-25 20:23:06

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,447
Website

Re: Reflector reborn

Thanks that makes sense as a fair use-case for such a feature.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#180 2023-04-10 20:00:17

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Reflector reborn

Reflector would have to read and compare the generated mirrorlist to the user's current mirrorlist and then create some sort of similarity metric to determine if they are similar enough to ignore the new list. That is quite far out of the scope of reflector and will not be implemented.

You would write your own wrapper script to generate a new list in tmpfs, diff that with the existing file and update the file only if you consider the changes significant.

Alternatively, given that the file is dynamically generated, why not just stop tracking it? In what scenario would you need to retrieve an old mirrorlist when you could just generate a new one?


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#181 2023-05-29 12:45:32

kplant
Member
Registered: 2023-05-29
Posts: 1

Re: Reflector reborn

@Xyne looks like after the most recent repo changes, an update to how reflector is scoring mirrors is needed.

Currently DB_SUBPATH is 'community/os/x86_64/community.db' - this file is now only 45 bytes, could we change this to 'extra/os/x86_64/extra.db'?

Last edited by kplant (2023-05-29 13:33:49)

Offline

#182 2023-06-18 10:46:03

airbreather
Member
Registered: 2010-03-17
Posts: 4

Re: Reflector reborn

kplant wrote:

@Xyne looks like after the most recent repo changes, an update to how reflector is scoring mirrors is needed.

Currently DB_SUBPATH is 'community/os/x86_64/community.db' - this file is now only 45 bytes, could we change this to 'extra/os/x86_64/extra.db'?

Just came here to post this as well.  After making this change in my local /usr/lib/python3.11/site-packages/Reflector.py file, the results from Reflector are a lot better now.

Offline

#183 2023-06-24 14:50:38

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Reflector reborn

Sorry, I've been neglecting my packages lately. I need to figure out what happened to the site and then I'll update reflector.

edit
The site's back up and reflector has been updated.

Last edited by Xyne (2023-06-28 01:01:47)


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#184 2023-08-04 21:10:04

xerus
Member
Registered: 2021-05-11
Posts: 34

Re: Reflector reborn

It seems this thread is also the place for bugs/feature requests?

I have a permanent reflector conf in the xdg dir shared across my devices, which so far filtered by my current country to effectively limit their number.
But if I take my laptop to another country, I would have to adjust that.

How about an option to use the current country, estimated by geoip, or a rough geographical area as the current country might not have an own mirror (or just fall back to worldwide in that case)?
I don't think edge cases matter here, but I think somehow filtering by distance is a good way find fast mirrors without needing to query too many.

Offline

#185 2023-09-19 10:23:54

tkit
Member
Registered: 2017-02-25
Posts: 15

Re: Reflector reborn

Description:
Reflector cannot find any mirrors for China

Additional info:
* package version(s)
```
➜ ~ LANG=C pacman -Qi reflector
Name : reflector
Version : 2023-1
Description : A Python 3 module and script to retrieve and filter the latest Pacman mirror list.
Architecture : any
URL : https://xyne.dev/projects/reflector
Licenses : GPL2
Groups : None
Provides : None
Depends On : python
Optional Deps : rsync: rate rsync mirrors [installed]
Required By : None
Optional For : None
Conflicts With : None
Replaces : None
Installed Size : 165.05 KiB
Packager : Xyne <xyne@archlinux.org>
Build Date : Wed Jun 28 08:57:34 2023
Install Date : Mon Jul 24 20:01:41 2023
Install Reason : Explicitly installed
Install Script : No
Validated By : Signature
* config and/or log files etc.
* link to upstream bug report, if any
```
Steps to reproduce:
```
➜ ~ reflector -c China
error: no mirrors found
➜ ~ reflector -c CN
error: no mirrors found
➜ ~ reflector -c JP
################################################################################
################# Arch Linux mirrorlist generated by Reflector #################
################################################################################

# With:       reflector -c JP
# When:       2023-09-19 10:23:07 UTC
# From:       https://archlinux.org/mirrors/status/json/
# Retrieved:  2023-09-19 10:23:07 UTC
# Last Check: 2023-09-19 10:16:15 UTC

Server = http://ftp.jaist.ac.jp/pub/Linux/ArchLi … o/os/$arch
Server = https://ftp.jaist.ac.jp/pub/Linux/ArchL … o/os/$arch
Server = rsync://ftp.jaist.ac.jp/pub/Linux/ArchLinux/$repo/os/$arch
Server = http://mirrors.cat.net/archlinux/$repo/os/$arch
Server = https://mirrors.cat.net/archlinux/$repo/os/$arch
Server = http://mirror.nishi.network/archlinux/$repo/os/$arch
Server = https://mirror.nishi.network/archlinux/$repo/os/$arch
Server = http://repo.jing.rocks/archlinux/$repo/os/$arch
Server = https://repo.jing.rocks/archlinux/$repo/os/$arch
Server = rsync://repo.jing.rocks/archlinux/$repo/os/$arch
➜ ~ reflector --list-countries | grep China
China CN 40
➜ ~ reflector --list-countries | awk 'NR==1 || /China/'
Country Code Count
China CN 40
➜ ~
```

Offline

#186 2023-09-19 10:28:53

Awebb
Member
Registered: 2010-05-06
Posts: 6,275

Re: Reflector reborn

That is not reflector's fault:

https://archlinux.org/mirrors/status/#errorlog

Offline

#187 2023-09-19 10:37:18

tkit
Member
Registered: 2017-02-25
Posts: 15

Re: Reflector reborn

Awebb wrote:

All mirrors from China are down?

Offline

#188 2023-09-19 18:09:08

Awebb
Member
Registered: 2010-05-06
Posts: 6,275

Re: Reflector reborn

tkit wrote:
Awebb wrote:

All mirrors from China are down?

They were when we had the conversation earlier. Check again. reflector -c CN now delivers something and servers are up again in the list. Whatever happened there, it hit all mirrors in China.

Last edited by Awebb (2023-09-19 20:39:17)

Offline

#189 2024-03-08 10:44:03

jasonkhanlar
Member
Registered: 2020-08-30
Posts: 7

Re: Reflector reborn

I am confused by


`reflector --delay 0.1 --latest 200 --protocol https --age 12 --sort rate --save /etc/pacman.d/mirrorlist.reflector` outputs 2 warnings:

[2024-03-08 04:58:56] WARNING: failed to rate http(s) download (https://mirrors.neusoft.edu.cn/archlinux/extra/os/x86_64/extra.db): Download timed out after 5 second(s).
[2024-03-08 05:00:01] WARNING: failed to rate http(s) download (https://mirror.cyberbits.asia/archlinux/extra/os/x86_64/extra.db): <urlopen error [Errno 101] Network is unreachable>

and the file contains those two domains as mirrors at the end of /etc/pacman.d/mirrorlist.reflector:
mirrors.neusoft.edu.cn and mirror.cyberbits.asia
and they are not commented out

If it is intentional to have those two mirrors included at the end, which according to data at https://archlinux.org/mirrors/status/ appear to fit the rest of the criteria in the command line, then there should be command line switch option to exclude entries that have warnings. I manually removed the entries myself for now.

Last edited by jasonkhanlar (2024-03-08 11:02:39)

Offline

#190 2024-03-13 19:47:14

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Reflector reborn

jasonkhanlar wrote:

]there should be command line switch option to exclude entries that have warnings. I manually removed the entries myself for now.

I've added it to the todo list.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

Board footer

Powered by FluxBB