Reflector returns "failed to rate http(s) download: Download ......" !

im1st · 2021-01-12 22:13:53

Hi there

after about 2 month I've updated my Arch and now I get a problem with reflector

it returns

failed to rate http(s) download (https://arch.jensgutermuth.de/community/os/x86_64/community.db): Download timed out after 5 second(s).

(this url for example) and repeat a lot (not for every url) and i have to metion that I can access this url by browser and also when I put it in the first line on mirrorlist, it works but reflector returns error

it happens when I use -f or --fastest option

what is the problem?

thanks in advance

Nikita790 · 2021-01-13 03:48:21

Hey, it is because the download-timeout is set lower then the time it takes to connect to the server, add this flag to reflector " --download-timeout " and set it to a higher number, you can use 60seconds (it is what i use) for example

im1st · 2021-01-13 07:36:18

Nikita790 wrote:

Hey, it is because the download-timeout is set lower then the time it takes to connect to the server, add this flag to reflector " --download-timeout " and set it to a higher number, you can use 60seconds (it is what i use) for example

grateful
the default is 5, in other hand, if it takes more than 5 second to connect to a server, the server is down or I can not connect to it (I'm from Iran, they like to block IPs a lot )
Dont know why but it happens a lot and because of that it takes a lot of time to reflector takes action

after I posted this topic, I've used " --download-timeout 2 " and find reflector became better and worked but even after, I couldnt reach the highest speed like before!

anyway, Thanks a lot Nikita

Vextric · 2021-12-28 16:52:05

Hi I know it's kind of old, but I'm from Iran too and I get the timeout regardless of timeout duration. Can you please tell me that which command do you use to update mirrors these days? thanks!

schard · 2021-12-28 18:52:05

You should open an issue on reflector's Bug tracker. If the measuring of a mirror times out during rating, it should be disqualified i.e. excluded.

Xyne · 2022-01-16 17:43:39

This isn't a bug. Sorting only changes the order of selected mirrors. If your selected mirrors are timing out then you should use different selection criteria. You can find a longer reply here.

stojshic · 2022-01-17 23:53:27

Reflector also does not work for me too, today... I tried with --latest, tried with country france, germany, serbia, every time, for every mirror, it just gets warning about failing to rate... :-/

Xyne · 2022-01-19 10:04:28

stojshic wrote:

Reflector also does not work for me too, today... I tried with --latest, tried with country france, germany, serbia, every time, for every mirror, it just gets warning about failing to rate... :-/

I can't reproduce the error. Does it work without rating? Post the exact command that you are using.

> reflector --latest 5 --sort rate --country france,germany,serbia
################################################################################
################# Arch Linux mirrorlist generated by Reflector #################
################################################################################

# With:       reflector --latest 5 --sort rate --country france,germany,serbia
# When:       2022-01-19 10:02:14 UTC
# From:       https://archlinux.org/mirrors/status/json/
# Retrieved:  2022-01-19 10:01:15 UTC
# Last Check: 2022-01-19 09:51:47 UTC

Server = http://mirror.cyberbits.eu/archlinux/$repo/os/$arch
Server = https://mirror.cyberbits.eu/archlinux/$repo/os/$arch
Server = https://mirror.chaoticum.net/arch/$repo/os/$arch
Server = http://mirror.chaoticum.net/arch/$repo/os/$arch
Server = rsync://rsync.cyberbits.eu/archlinux/$repo/os/$arch

> pacman -Qi reflector
Name            : reflector
Version         : 2021.11-2
Description     : A Python 3 module and script to retrieve and filter the latest Pacman mirror list.
Architecture    : any
URL             : https://xyne.dev/projects/reflector
Licenses        : GPL
Groups          : None
Provides        : None
Depends On      : python3
Optional Deps   : rsync: rate rsync mirrors [installed]
Required By     : None
Optional For    : pm2ml  powerpill
Conflicts With  : None
Replaces        : None
Installed Size  : 103.17 KiB
Packager        : Xyne
Build Date      : 2022-01-16 W02-7 18:49:41 +0100
Install Date    : 2022-01-19 W03-3 11:00:30 +0100
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

Fixxer · 2022-01-19 10:08:26

But /bin/sh(it) happens

reflector --latest 5 --sort rate
[2022-01-19 11:06:58] WARNING: failed to rate http(s) download (http://mirror.lty.me/archlinux/community/os/x86_64/community.db): Download timed out after 5 second(s).
################################################################################
################# Arch Linux mirrorlist generated by Reflector #################
################################################################################

# With:       reflector --latest 5 --sort rate
# When:       2022-01-19 10:06:59 UTC
# From:       https://archlinux.org/mirrors/status/json/
# Retrieved:  2022-01-19 10:06:34 UTC
# Last Check: 2022-01-19 09:51:47 UTC

Server = http://mirror.telepoint.bg/archlinux/$repo/os/$arch
Server = https://mirror.telepoint.bg/archlinux/$repo/os/$arch
Server = http://mirror.cyberbits.eu/archlinux/$repo/os/$arch
Server = rsync://mirror.telepoint.bg/archlinux/$repo/os/$arch
Server = http://mirror.lty.me/archlinux/$repo/os/$arch

PeterLustknabePan · 2022-01-31 11:07:42

Same for me

Xyne · 2022-01-31 21:25:49

> reflector --latest 5 --sort rate
################################################################################
################# Arch Linux mirrorlist generated by Reflector #################
################################################################################

# With:       reflector --latest 5 --sort rate
# When:       2022-01-31 21:12:55 UTC
# From:       https://archlinux.org/mirrors/status/json/
# Retrieved:  2022-01-31 21:12:50 UTC
# Last Check: 2022-01-31 21:00:05 UTC

Server = https://ftp.halifax.rwth-aachen.de/archlinux/$repo/os/$arch
Server = https://archlinux.thaller.ws/$repo/os/$arch
Server = http://archlinux.thaller.ws/$repo/os/$arch
Server = rsync://ftp.halifax.rwth-aachen.de/archlinux/$repo/os/$arch
Server = rsync://archlinux.thaller.ws/archlinux/$repo/os/$arch

Xyne wrote:

This isn't a bug. Sorting only changes the order of selected mirrors. If your selected mirrors are timing out then you should use different selection criteria. You can find a longer reply here.

If the download times out then that server ends up at the bottom of the sorted list, which is where it belongs because the connection is slow. If you want to hide the warning then you can increase the timeout interval with the command-line options but the connection itself will still be slow.

Again, the better solution is to use appropriate filter options (e.g.--country) to select mirrors closer to you with better connections.

Neitsab · 2022-03-11 14:36:18

Hi Xyne, I have also been facing this error consistently recently while using the official Arch installation media to provision VMs:

% reflector --country France,Germany \
          --latest 5 \
          --protocol https \
          --sort rate
[2022-03-11 15:28:27] WARNING: failed to rate http(s) download (https://archlinux.mailtunnel.eu/community/os/x86_64/community.db): Download timed out after 5 second(s).
[2022-03-11 15:28:33] WARNING: failed to rate http(s) download (https://mirror.luzea.de/archlinux/community/os/x86_64/community.db): Download timed out after 5 second(s).
[2022-03-11 15:28:47] WARNING: failed to rate http(s) download (https://archlinux.thaller.ws/community/os/x86_64/community.db): Download timed out after 5 second(s).
[2022-03-11 15:28:59] WARNING: failed to rate http(s) download (https://mirror.pseudoform.org/community/os/x86_64/community.db): Download timed out after 5 second(s).
[2022-03-11 15:29:06] WARNING: failed to rate http(s) download (https://de.arch.mirror.kescher.at/community/os/x86_64/community.db): Download timed out after 5 second(s).
################################################################################
################# Arch Linux mirrorlist generated by Reflector #################
################################################################################

# With:       reflector --country France,Germany --latest 5 --protocol https --sort rate
# When:       2022-03-11 14:29:06 UTC
# From:       https://archlinux.org/mirrors/status/json/
# Retrieved:  2022-03-11 14:28:21 UTC
# Last Check: 2022-03-11 13:56:42 UTC

Server = https://archlinux.mailtunnel.eu/$repo/os/$arch
Server = https://mirror.luzea.de/archlinux/$repo/os/$arch
Server = https://archlinux.thaller.ws/$repo/os/$arch
Server = https://mirror.pseudoform.org/$repo/os/$arch
Server = https://de.arch.mirror.kescher.at/$repo/os/$arch

As you can see from the timestamps it took quite a while in between errors.

I am tethered to my phone so it certainly isn't the greatest connection, but despite all the warnings I get between 1 and 3 MB/s downloads with pacman and no repo/package download failure. That's why to me it feels like there is a certain something specific to reflector which triggers those failures, but I don't know enough to say what.

Thanks

Xyne · 2022-03-12 10:50:10

The community database is 6 MB. The download times out after 5 s. If your average download rate with your connection over 5 seconds is less than 1.2 MB/s then it will time out. This isn't a reflector issue, unless mirrors start blocking requests from reflector due to the number of people misusing the rate sorting.

Either increase the timeout, or don't even bother rate-sorting on a connection that maxes out at 3MB. With Pacman's parallel downloads (or powerpill, if anyone else still uses it), the connection will be saturated at that bandwidth.

Neitsab · 2022-03-12 17:40:51

Thanks for your reply. I was operating under the assumption that the "download timed out" message meant the connection had failed/broken up, but hadn't considered the 5s delay applied to the allotted time to download a file from the servers. But your math checks out and it does explain why only community.db is affected. If pacman has even just a 10s timeout delay, that would explain the difference in behaviour I observed.

Slightly tangential, but do we really live in a world where 1 MB/s (so about 8-10 Mbps, not even speaking about the ludicrous 3 MB/s (!) you mention) is considered too low a bandwidth to bother with? It was only a few years ago when I was delighted to reach those heights, because it meant I could stream a 720p movie without worrying too much about buffering... It certainly sounds exaggerated to me to dismiss it as a valid bandwidth threshold for general computing since I do torrents and other large transfers on it, but anyway I'm digressing.

So you feel like people abuse the rate sorting functionality? It is enabled in the Archiso reflector config, and I see it as one of the stronger selling points of the software, but I think I can understand what you mean especially in conjunction with ParallelDownloads (which is disabled by default OTOH). What would you recommend people do?

Cheers

Trilby · 2022-03-12 18:49:23

Neitsab wrote:

do we really live in a world where 1 MB/s (so about 8-10 Mbps, not even speaking about the ludicrous 3 MB/s (!) you mention) is considered too low a bandwidth to bother with?

No one has suggested any such thing here. The defaut timeout is simply the default. The fact that there is a simple option (command line flag) to change this timeout indicates that it is seen as "worth bothering with".

EDIT: though the comment on a connection that maxes out at 3MB/s could be read differently - but it's still not suggesting that users of such connections are not worth bothering with - but only that there is no point trying to find fast mirrors if the mirrors are not the bottleneck. If the connection from your computer to the wider internet is a massive bottleneck, then rating mirror speed is irrelevant as all attempted measures would be capped by the local connection rather than providing any information about the mirror. In such cases (where the local connection is the bottleneck) just pick mirrors that are good based on other criteria.

Neitsab wrote:

So you feel like people abuse the rate sorting functionality? It is enabled in the Archiso reflector config

Suggesting that some users may abuse it does not imply that all use of sorting is abuse. The link you provided shows filters *before* sorting - so not abuse. I don't think an author of a bit of software would include a command line flag for their software that they considered every use of an abuse.

Last edited by Trilby (2022-03-12 18:53:31)

Xyne · 2022-03-16 20:45:43

Trilby's response is spot-on. I'm not disparaging your bandwidth at all. My point is simply that once a mirror saturates your bandwidth, you simply cannot receive the data any faster no matter how much faster the mirror could serve it. A Ferrari and a Lada stuck in the same traffic jam will move at the same speed. Rating mirrors by speed is mostly useful when your bandwidth exceeds the server's bandwidth, but even that is less useful now that we can just increment the number of parallel downloads in Pacman.

The main abuse of the rating function that I see is rating hundreds of mirrors without first filtering them. That's several GB of data transfer every time just to find the "fastest" mirror. If the user's bandwidth is already saturated then it makes no difference, and if not then the bandwidth is probably so high that the difference in download time is negligible.

In my opinion it only makes sense to sort by rate if the chosen filters somehow return a set of servers with significantly different bandwidths, some of which are much slower than the user's connection.

Subject-17 · 2022-09-19 21:51:48

EDIT: nevermind, it's a problem with my cable. FYI when using iperf also try the -R flag. I'm getting 1 gig Tx but <1meg Rx

Last edited by Subject-17 (2022-09-20 03:39:49)

mablue · 2024-04-15 05:24:31

Nikita790 wrote:

Hey, it is because the download-timeout is set lower then the time it takes to connect to the server, add this flag to reflector " --download-timeout " and set it to a higher number, you can use 60seconds (it is what i use) for example

Thanks Problem [SOLVED] by adding

--download-timeout 60

to my command

Last edited by mablue (2024-04-15 05:25:52)

Arch Linux

#1 2021-01-12 22:13:53

Reflector returns "failed to rate http(s) download: Download ......" !

#2 2021-01-13 03:48:21

Re: Reflector returns "failed to rate http(s) download: Download ......" !

#3 2021-01-13 07:36:18

Re: Reflector returns "failed to rate http(s) download: Download ......" !

#4 2021-12-28 16:52:05

Re: Reflector returns "failed to rate http(s) download: Download ......" !

#5 2021-12-28 18:52:05

Re: Reflector returns "failed to rate http(s) download: Download ......" !

#6 2022-01-16 17:43:39

Re: Reflector returns "failed to rate http(s) download: Download ......" !

#7 2022-01-17 23:53:27

Re: Reflector returns "failed to rate http(s) download: Download ......" !

#8 2022-01-19 10:04:28

Re: Reflector returns "failed to rate http(s) download: Download ......" !

#9 2022-01-19 10:08:26

Re: Reflector returns "failed to rate http(s) download: Download ......" !

#10 2022-01-31 11:07:42

Re: Reflector returns "failed to rate http(s) download: Download ......" !

#11 2022-01-31 21:25:49

Re: Reflector returns "failed to rate http(s) download: Download ......" !

#12 2022-03-11 14:36:18

Re: Reflector returns "failed to rate http(s) download: Download ......" !

#13 2022-03-12 10:50:10

Re: Reflector returns "failed to rate http(s) download: Download ......" !

#14 2022-03-12 17:40:51

Re: Reflector returns "failed to rate http(s) download: Download ......" !

#15 2022-03-12 18:49:23

Re: Reflector returns "failed to rate http(s) download: Download ......" !

#16 2022-03-16 20:45:43

Re: Reflector returns "failed to rate http(s) download: Download ......" !

#17 2022-09-19 21:51:48

Re: Reflector returns "failed to rate http(s) download: Download ......" !

#18 2024-04-15 05:24:31

Re: Reflector returns "failed to rate http(s) download: Download ......" !

Board footer