You are not logged in.

#1 2021-02-07 07:32:39

westandskif
Member
From: Belarus, Minsk
Registered: 2021-02-07
Posts: 11

rate-mirrors: Everyday-use client-side map-aware tool

Hello everyone!

I often update the system and always want to work with ± fastest up-to-date mirrors.
Also I live in a country (Belarus) with a far from being stable internet -- it's quite fast, but connection speeds vary significantly even over the course of the day.

So I wanted a tool which is better than Reflector IMHO:
- doesn't spend eternity on trying to load a file of a fixed size from a half-dead mirror -- ~30 seconds for everything
- needs no options and just works
- knows something about internet infrastructure (submarine cables and at least numbers of internet exchanges per country)
- knows when the speed has stabilized and is able to stop
- tests concurrently during the phase #1
- re-tests top N mirrors
- is ok to run before each system update.

=== edited on 6/30/2021 ===
- AUR rate-mirrors-bin
- AUR rate-mirrors
- https://github.com/westandskif/rate-mirrors

Last edited by westandskif (2021-06-30 20:13:53)

Offline

#2 2021-02-07 17:09:36

philo
Member
Registered: 2015-01-26
Posts: 251

Re: rate-mirrors: Everyday-use client-side map-aware tool

Reflector is not good enough?

https://wiki.archlinux.org/index.php/Reflector

Offline

#3 2021-02-07 20:33:03

westandskif
Member
From: Belarus, Minsk
Registered: 2021-02-07
Posts: 11

Re: rate-mirrors: Everyday-use client-side map-aware tool

philo wrote:

This is the tool which made me write my own - I used to use Reflector quite a lot but:
- it wastes too much time on testing too slow mirrors
- it’s impossible to take N mirrors per country, you only can list countries and after sorting I get Germany only (just because its mirrors are often more relevant, but they are far from being closest and fastest for me; while only let’s say half an hour ahead in terms of sync time)
- speed is worse
- it doesn’t allow you to just run it in a brainless mode and find a fast mirror from a country you never expected

Offline

#4 2021-02-08 18:54:17

adventurer
Member
Registered: 2014-05-04
Posts: 119

Re: rate-mirrors: Everyday-use client-side map-aware tool

Interesting approach, indeed! I'm living in Germany, and rate-arch-mirrors created a list with quite a number of mirrors that are much faster than the fastest German mirror.

Some thoughts:

- There is no option to limit the list to https mirrors.
- The mirror score is not taken into account. In my understanding the score is a measure of mirror reliability. Limiting the speed test to mirrors with the lowest score might make rate-arch-mirrors even faster.
- For some options it's not clear to me what they really mean. A more verbose explanation would be helpful.

Offline

#5 2021-02-08 20:53:28

ugjka
Member
From: Latvia
Registered: 2014-04-01
Posts: 1,794
Website

Re: rate-mirrors: Everyday-use client-side map-aware tool

Is Updating crates.io index supposed to take a while when building the package?


https://ugjka.net
paru > yay | webcord > discord
pacman -S spotify-launcher
mount /dev/disk/by-...

Offline

#6 2021-02-09 14:30:12

westandskif
Member
From: Belarus, Minsk
Registered: 2021-02-07
Posts: 11

Re: rate-mirrors: Everyday-use client-side map-aware tool

ugjka wrote:

Is Updating crates.io index supposed to take a while when building the package?

Yes hmm it refreshes the index and fetches dependencies.
I have a point in my TODO list to a provide pre-compiled version too -- hopefully will do it soon.

Offline

#7 2021-02-09 14:37:00

westandskif
Member
From: Belarus, Minsk
Registered: 2021-02-07
Posts: 11

Re: rate-mirrors: Everyday-use client-side map-aware tool

adventurer wrote:

Interesting approach, indeed! I'm living in Germany, and rate-arch-mirrors created a list with quite a number of mirrors that are much faster than the fastest German mirror.

Some thoughts:

- There is no option to limit the list to https mirrors.
- The mirror score is not taken into account. In my understanding the score is a measure of mirror reliability. Limiting the speed test to mirrors with the lowest score might make rate-arch-mirrors even faster.
- For some options it's not clear to me what they really mean. A more verbose explanation would be helpful.

Thank you very much for the feedback!
- I've added "https only" option to my TODO list
- will do! there's definitely a room for another option -- to choose a strategy of sampling country mirrors (shuffle/top-score)
- I see, I'll do my best, but probably I'll still need some help in the end (at least with pointing out what is unclear smile )

Offline

#8 2021-02-09 14:53:15

progandy
Member
Registered: 2012-05-17
Posts: 5,184

Re: rate-mirrors: Everyday-use client-side map-aware tool

You could also keep the mirror filtering separate and have an option to receive the json data from stdin or a file if you want to have less options.


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

#9 2021-02-22 09:48:31

westandskif
Member
From: Belarus, Minsk
Registered: 2021-02-07
Posts: 11

Re: rate-mirrors: Everyday-use client-side map-aware tool

adventurer wrote:

- There is no option to limit the list to https mirrors.
- The mirror score is not taken into account. In my understanding the score is a measure of mirror reliability. Limiting the speed test to mirrors with the lowest score might make rate-arch-mirrors even faster.
- For some options it's not clear to me what they really mean. A more verbose explanation would be helpful.

I've released a new version:

rate-arch-mirrors --protocol=https  # to limit the list to https mirrors
rate-arch-mirrors --sort-mirrors-by=score_asc  # this sorting is on by default (lower score is better)

Also I've updated help texts, could you please share your feedback?


ugjka wrote:

Is Updating crates.io index supposed to take a while when building the package?

Could you please try installing pre-built binary? https://aur.archlinux.org/packages/rate … rrors-bin/

Offline

#10 2021-02-24 11:11:23

adventurer
Member
Registered: 2014-05-04
Posts: 119

Re: rate-mirrors: Everyday-use client-side map-aware tool

Nice, improvements! Thank you. Though I'm not sure if it really got any faster by sorting for https only and score ...

The help texts are clearer now.

One thought: Considering --concurrency=8 , isn't rate-arch-mirrors affected by the problem discussed in https://bbs.archlinux.org/viewtopic.php?id=261040, as well?

Last edited by adventurer (2021-02-24 11:11:42)

Offline

#11 2021-02-24 15:43:14

westandskif
Member
From: Belarus, Minsk
Registered: 2021-02-07
Posts: 11

Re: rate-mirrors: Everyday-use client-side map-aware tool

adventurer wrote:

Nice, improvements! Thank you. Though I'm not sure if it really got any faster by sorting for https only and score ...

My pleasure! smile

Regarding "sorting by score & https only" - yeah, I find it best to take both http & https, while the best sorting option for the next run is never known.
Sometimes random sorting may reveal a mirror unreachable otherwise (e.g. we have 20 mirrors in a country, while testing 2-3) -- e.g a mirror may be not the most stable on average, not the most recent in terms of delay after sync, but it may have 100% of bandwidth available.


adventurer wrote:

One thought: Considering --concurrency=8 , isn't rate-arch-mirrors affected by the problem discussed in https://bbs.archlinux.org/viewtopic.php?id=261040, as well?

There are 2 testing phases:
1. "Dirty check": 8 mirrors are tested concurrently. My assumption is that a fast mirror should be notable even in such conditions. Also the tool waits until a speed is ±stable for every mirror.
2. "Final test": the tool re-tests top N mirrors with no concurrency (1 by 1). Also the tool waits until a speed is twice as stable as in the "Dirty check" phase

So, having this run right before the system upgrade gives a chance to see speeds, similar to what you see here

Offline

#12 2021-02-24 18:48:20

adventurer
Member
Registered: 2014-05-04
Posts: 119

Re: rate-mirrors: Everyday-use client-side map-aware tool

1. "Dirty check": 8 mirrors are tested concurrently. My assumption is that a fast mirror should be notable even in such conditions. Also the tool waits until a speed is ±stable for every mirror.

Hm, Xyne, the Reflector creator, writes in above mentioned thread:

I completely agree that parallel speed testing makes absolutely no sense if the user connection is the bottleneck.
...
I'll change the default number of threads to 1 but leave the option for those with better connections.

I tend to agree with him. Even the results of the "dirty check" might be bogus if the available bandwidth is not sufficient which means that the pre-selected mirrors in that first step might not necessarily be the fastest.

Offline

#13 2021-02-24 19:59:01

westandskif
Member
From: Belarus, Minsk
Registered: 2021-02-07
Posts: 11

Re: rate-mirrors: Everyday-use client-side map-aware tool

adventurer wrote:

I tend to agree with him. Even the results of the "dirty check" might be bogus if the available bandwidth is not sufficient which means that the pre-selected mirrors in that first step might not necessarily be the fastest.

If this is the case, then it makes sense to adjust concurrency using the option.
I believe this should apply to connection of any speed: e.g. my connection is 100Mbps, so 8 even mirror connections would compete for my bandwidth, resulting in 1.25MB/s each.
At this same step (one out of seven) we’ll pick a mirror with smallest time to connect along with the one with the highest speed. There will be 6 more steps like this.
After that, top performers will be retested.
We should be really lucky to have top mirrors perfectly paired at every step.

Everything I mentioned above doesn’t change the fact, that the results may be bogus in the case you mentioned. But also same will happen if you use concurrency=1 and there’s a sudden drop in your downstream.

My point is:
The tool with default settings helps you find fast (not the fastest) mirrors within 15-25 seconds.

If the goal is to find the fastest ones, then I would set concurrency to 1, test every mirror in every country (increasing limits), run this test multiple times to find out top mirrors. However this result would reflect the past:) the longer the test takes — the more outdated the results will be.

Offline

#14 2021-06-30 20:01:09

westandskif
Member
From: Belarus, Minsk
Registered: 2021-02-07
Posts: 11

Re: rate-mirrors: Everyday-use client-side map-aware tool

update:
Updated and renamed the tool, now it supports rating lists of mirrors passed via stdin (just in case), arch and manjaro ones.
Also i've updated the first post with latest links.

Offline

#15 2021-07-01 11:11:10

adventurer
Member
Registered: 2014-05-04
Posts: 119

Re: rate-mirrors: Everyday-use client-side map-aware tool

Hi, I tried the new version but ran into troubles.

Executing

rate-mirrors arch --max-delay=21600 --sort-mirrors-by=score_asc

worked as expected but adding more options resulted in errors, e.g.:

rate-mirrors arch --max-delay=21600 --sort-mirrors-by=score_asc --country-neighbors-per-country=3

gave an error:

error: Found argument '--country-neighbors-per-country' which wasn't expected, or isn't valid in this context

The same happened when adding other options.

Offline

#16 2021-07-01 11:54:25

westandskif
Member
From: Belarus, Minsk
Registered: 2021-02-07
Posts: 11

Re: rate-mirrors: Everyday-use client-side map-aware tool

adventurer wrote:

but adding more options resulted in errors, e.g.:

rate-mirrors arch --max-delay=21600 --sort-mirrors-by=score_asc --country-neighbors-per-country=3

gave an error:

error: Found argument '--country-neighbors-per-country' which wasn't expected, or isn't valid in this context

The same happened when adding other options.

Hi! Thank you for trying it! smile

There are base options which have their own help

rate-mirrors --help

and which go before a sub-command like this:

rate-mirrors --country-neighbors-per-country=3 arch

And there are sub-command specific options which have their help like:

rate-mirrors arch --help

and which go after the sub-command like this:

rate-mirrors --country-neighbors-per-country=3 arch --max-delay=21600 --sort-mirrors-by=score_asc

Offline

#17 2021-07-01 17:18:35

adventurer
Member
Registered: 2014-05-04
Posts: 119

Re: rate-mirrors: Everyday-use client-side map-aware tool

Ah - thanks a lot! It works now! The distinction between the options before and after the sub-command had not been obvious to me. Perhaps this should be documented more prominently/clearly ...

Btw.: There is a typo on your github site here :

(--sort-mirrors-by=store_asc option)

Offline

#18 2021-07-01 20:16:28

westandskif
Member
From: Belarus, Minsk
Registered: 2021-02-07
Posts: 11

Re: rate-mirrors: Everyday-use client-side map-aware tool

adventurer wrote:

Ah - thanks a lot! It works now! The distinction between the options before and after the sub-command had not been obvious to me. Perhaps this should be documented more prominently/clearly ...

Btw.: There is a typo on your github site here :

(--sort-mirrors-by=store_asc option)

Glad to help smile
I've fixed the typo, thank you!

Regarding the docs, I'll try to elaborate on this.

Offline

Board footer

Powered by FluxBB