Reflector reborn

Gesh · 2012-12-17 13:28:13

As a workaround in the meantime, you can unset XDG_CACHE_HOME before calling reflector, which will make it use a directory in /tmp
This is caused by Arch supporting the XDG specified environment variables without creating the directories they point to.

Xyne · 2012-12-17 17:16:26

The new version is in [community] now, so workarounds are no longer necessary.

adventurer · 2014-05-23 18:14:08

Hi Xyne,

as a rather new Arch user I've started to use Reflector just a couple of days ago.

I'm using it as suggested on the Reflector wiki page:

reflector --verbose -l 20 -p http --sort rate --save /etc/pacman.d/mirrorlist

It works very well but I have some question regarding the arguments and filters:

1. How is "--sort rank" different from "--fastest n"? My interpretation is that "--sort rank" already delivers the fastest mirrors so that "--fastest n" only limits that list to n mirrors but doesn't change their sequence. Is this correct?
2. Does "rank" correspond to the "Duration" mentioned on the Mirror Status page? Is replacing "--sort rank" with "--sort score" recommended? What's your opinion on this?
3. Does using the "--threads n" argument make a big performance difference? In my tests it was faster but that might have been due to caching in dnsmasq.

Thanks for a very useful tool!

Last edited by adventurer (2014-05-23 18:14:32)

Xyne · 2014-05-24 14:21:08

Hi adventurer,

First a minor disclaimer: I did not write the wiki page nor do I personally recommend using reflector that way. On my own systems I use a static mirrorlist and only use reflector indirectly through powerpill for faster downloads. That stated, there's nothing necessary wrong with that command. I simply prefer to choose which mirrors to trust for database downloads manually.

1. There is no --sort rank. I assume that you mean --sort rate, which sorts the mirrors by download rate after filters have been applied. For example, if you filter by country then sort by rate, only the mirrors from the selected country will be rated by speed. When you use --fastest n, reflector will rate all the mirrors (in the absence of other filters) and then return the n fastest ones. To rate a mirror, reflector downloads the core database from each mirror which can take a very long time and waste both your own and the mirrors' bandwidth if you try to rate too many (the full list contains over 200 mirrors).

2. "rate" is the download rate determined by downloading the core database from the mirror to your computer. It is specific to your connection, which in turn depends on your ISP, geographic location, etc. The score on the Mirror Status page is determined by the Arch Linux server. I don't know exactly how they calculate it but it depends on the state of synchronization of the mirror and perhaps on the connection.

Again, given that I use powerpill, I am not concerned with per-mirror download rates so i never use --sort rate. If I had to choose, I would probably prefer score over rate as I would rather have the most up-to-date mirror than the fastest one.

3. It depends on your connection to the mirrors. If each connection is only using part of your bandwidth then increasing the number of threads will speed up the rating operation. The operation will take the least time when your bandwidth is maxed, but that will also give skewed results because the calculated download rate to some servers will be reduced due to connection saturation on your end. I believe that increasing the number of threads until your bandwidth is almost maxed will give the best results.

Trilby · 2014-05-24 16:43:31

Xyne wrote:

For example, if you filter by country then sort by rate, only the mirrors from the selected country will be rated by speed. When you use --fastest n, reflector will rate all the mirrors (in the absence of other filters) and then return the n fastest ones.

Can you clarify this - that statement seems to contradict the description in the help output:

  -f n, --fastest n     Return the n fastest mirrors that meet the other
                        criteria. Do not use this option without other
                        filtering options.

Specifically, you just mentioned that this parameter would get the fastest N of all mirrors in the absence of other filters, while the help output indicates it finds the fastest N out of those that meet all other provided criteria.

adventurer · 2014-05-24 16:54:20

Thanks a lot, Xyne, your reply was very helpful! BTW, "rank" was a typo, indeed, I meant "rate", of course

But I still have a couple of questions if you don't mind.

Xyne wrote:

1. There is no --sort rank. I assume that you mean --sort rate, which sorts the mirrors by download rate after filters have been applied. For example, if you filter by country then sort by rate, only the mirrors from the selected country will be rated by speed. When you use --fastest n, reflector will rate all the mirrors (in the absence of other filters) and then return the n fastest ones.

You wrote: "in the absence of other filters". But if there are other filters? More specifically, what's the difference between

reflector -l 20 -p http --sort rate

and

reflector -l 20 -p http --fastest 10

and

reflector -l 20 -p http --sort rate --fastest 10

The different behaviour of reflector in those cases is not yet clear to me.

If I had to choose, I would probably prefer score over rate as I would rather have the most up-to-date mirror than the fastest one.

But isn't that already taken into account by using --latest n, which "limits the list to the n most recently synchronized servers."? In other words, what's the benefit of --sort score if you already use --latest n ? How are both related?

Last edited by adventurer (2014-05-24 16:56:09)

Xyne · 2014-05-30 20:12:13

adventurer wrote:

reflector -l 20 -p http --sort rate

That will generate a list of the 20 most recently synchronized mirrors, sorted by rate.

adventurer wrote:

reflector -l 20 -p http --fastest 10

That will give you an unsorted list of the fastest 10 mirrors among the 20 most recently sync'd.

adventurer wrote:

reflector -l 20 -p http --sort rate --fastest 10

That will give you the same as above but sorted by rate.

adventurer wrote:

But isn't that already taken into account by using --latest n, which "limits the list to the n most recently synchronized servers."? In other words, what's the benefit of --sort score if you already use --latest n ? How are both related?

My understanding is that some mirrors do not fully synchronize so they may appear in the latest 20 yet still be missing packages. Those mirrors should have lower scores. I'm not really sure how the it works though and I believe that most mirrors fully sync each time they do so the score is probably irrelevant most of the time for end users.

Xyne · 2014-05-31 03:19:18

Trilby wrote:

Specifically, you just mentioned that this parameter would get the fastest N of all mirrors in the absence of other filters, while the help output indicates it finds the fastest N out of those that meet all other provided criteria.

Sorry, I missed your post.

I don't see any contradiction. Each filter is an additional criterion. If you apply no filters then all mirrors meet your criteria.

The "fastest" filter is applied to the remaining servers after all other filters have been applied (as I recall). If no other filters have been applied then it will rate all of the mirrors and return the fastest n mirrors. If you have applied other filters such as "country", then it will only rate the mirrors that match the target country and return the n fastest among those.

Trilby · 2014-05-31 03:22:00

Thanks your previous post already cleared up any confusion I had.

adventurer · 2014-05-31 13:46:04

Xyne, thanks again for your post. Again very helpful and much appreciated.

Xyne · 2014-08-15 21:27:38

Updated to 2014.8. Changelog:

new --info option for returning mirror info instead of a mirror list
sorting and filtering by country now works with country codes (also included in the output of --list-countries)

Examples:

$ reflector --list-countries
Australia                                   AU 15
Austria                                     AT  1
Bangladesh                                  BD  2
Belarus                                     BY  2
Belgium                                     BE  3
Brazil                                      BR  2
Bulgaria                                    BG  1
Canada                                      CA  9
China                                       CN  7
Colombia                                    CO  1
Croatia                                     HR  1
Czech Republic                              CZ  8
Denmark                                     DK  2
Ecuador                                     EC  2
Estonia                                     EE  2
Finland                                     FI  2
France                                      FR 11
Germany                                     DE 44
Greece                                      GR  6
Hong Kong                                   HK  2
Hungary                                     HU  1
Iceland                                     IS  3
India                                       IN  2
Indonesia                                   ID  1
Ireland                                     IE  1
Israel                                      IL  1
Italy                                       IT  1
Japan                                       JP  4
Kazakhstan                                  KZ  2
Korea (the Republic of)                     KR  2
Lithuania                                   LT  1
Luxembourg                                  LU  1
Macedonia (the former Yugoslav Republic of) MK  2
Netherlands                                 NL  9
New Caledonia                               NC  4
New Zealand                                 NZ  1
Norway                                      NO  3
Philippines                                 PH  1
Poland                                      PL  4
Portugal                                    PT  3
Romania                                     RO  3
Russian Federation                          RU  2
Serbia                                      RS  1
Singapore                                   SG  2
Slovakia                                    SK  3
Spain                                       ES  2
Sri Lanka                                   LK  1
Sweden                                      SE  6
Switzerland                                 CH  1
Taiwan (Province of China)                  TW  4
United Kingdom                              GB  8
United States                               US 45

$ reflector -l 5 -c US --info
http://mirrors.kernel.org/archlinux/$repo/os/$arch
completion_pct  : 1.0
country         : United States
country_code    : US
delay           : 2292
duration_avg    : 0.702270154442106
duration_stddev : 0.598905511896356
last_sync       : 2014-08-15 21:00:01 UTC
protocol        : http
score           : 1.9378423330051289

https://mirrors.kernel.org/archlinux/$repo/os/$arch
completion_pct  : 1.0
country         : United States
country_code    : US
delay           : 2306
duration_avg    : 0.532400697469711
duration_stddev : 0.169243353766836
last_sync       : 2014-08-15 21:00:01 UTC
protocol        : https
score           : 1.3421996067921027

rsync://mirrors.kernel.org/archlinux/$repo/os/$arch
completion_pct  : 1.0
country         : United States
country_code    : US
delay           : 2306
duration_avg    : 4.76898066486631
duration_stddev : 3.57896372787145
last_sync       : 2014-08-15 21:00:01 UTC
protocol        : rsync
score           : 8.988499948293315

http://www.gtlib.gatech.edu/pub/archlinux/$repo/os/$arch
completion_pct  : 1.0
country         : United States
country_code    : US
delay           : 2374
duration_avg    : 0.458140905414309
duration_stddev : 0.208162151169742
last_sync       : 2014-08-15 20:50:01 UTC
protocol        : http
score           : 1.3257475010284956

rsync://rsync.gtlib.gatech.edu/archlinux/$repo/os/$arch
completion_pct  : 1.0
country         : United States
country_code    : US
delay           : 2374
duration_avg    : 1.36274749892099
duration_stddev : 0.567510278579429
last_sync       : 2014-08-15 20:50:01 UTC
protocol        : rsync
score           : 2.5897022219448633

karol · 2014-08-18 22:33:47

One more thing: the current reflector doesn't provide pacman-mirrorlist anymore.

Xyne · 2014-08-18 22:52:15

I don't remember where it was discussed before and I don't have time to check right now, but given that the package would have to include or overwrite the mirrorlist file it would have to conflict with the mirrorlist package.

Nietzsche · 2014-10-26 18:51:31

Hello, I really like your script, but I keep getting errors with it on a specific machine. Here's the traceback:

sudo reflector --sort rate -n 10 --save /etc/pacman.d/mirrorlist
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3.4/threading.py", line 921, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.4/threading.py", line 869, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.4/site-packages/Reflector.py", line 372, in worker
    size = len(f.read())
  File "/usr/lib/python3.4/http/client.py", line 512, in read
    s = self._safe_read(self.length)
  File "/usr/lib/python3.4/http/client.py", line 664, in _safe_read
    raise IncompleteRead(b''.join(s), amt)
http.client.IncompleteRead: IncompleteRead(37189 bytes read, 82939 more expected)

Xyne · 2014-10-26 22:07:37

It seems that one of the mirrors is misreporting the size of a file or failing to send it.

I have added an exception handler to work around it. Can you please upgrade to the latest version of Reflector in the repo on my site and let me know if it works? When I know that it does, I will push the update to [community].

Nietzsche · 2014-10-27 12:39:32

Looks like the same error to me.

pacman -Qi reflector | grep Version
Version               : 2014.10-1

sudo reflector --sort rate -n 15 --save /etc/pacman.d/mirrorlist
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/lib/python3.4/threading.py", line 921, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.4/threading.py", line 869, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.4/site-packages/Reflector.py", line 370, in worker
    size = len(f.read())
  File "/usr/lib/python3.4/http/client.py", line 512, in read
    s = self._safe_read(self.length)
  File "/usr/lib/python3.4/http/client.py", line 664, in _safe_read
    raise IncompleteRead(b''.join(s), amt)
http.client.IncompleteRead: IncompleteRead(37189 bytes read, 82923 more expected)

Thanks for the fast reply, byt the way.

Jristz · 2014-10-27 22:59:52

Is that one mirror, I use 200 and because that one mirror giving the problem.
I can confirm in my system that waiting 20 mins to let it finish not work, reflector simply stop in that problem.

egroeg · 2014-10-31 21:42:48

Jristz wrote:

Is that one mirror, I use 200 and because that one mirror giving the problem.
I can confirm in my system that waiting 20 mins to let it finish not work, reflector simply stop in that problem.

What mirror was it?

Xyne · 2014-11-01 16:25:11

Sorry for the late reply. Please try reflector 2014-11 and let me know if it works now.

Nietzsche · 2014-11-02 13:08:34

It works perfectly now.

Xyne · 2014-11-02 13:27:04

Thanks. I'll push it to [community].

skanky · 2015-10-02 09:04:22

Hi Xyne, just to let you know, I'm getting an error trying to run reflector, since the most recent python update to 3.5.

Traceback (most recent call last):
  File "/usr/bin/reflector", line 3, in <module>
    import Reflector
ImportError: No module named 'Reflector'

Last edited by skanky (2015-10-02 09:05:36)

karol · 2015-10-02 22:05:20

Works for me on my 32-bit Arch with [testing] repos enabled.

predmijat · 2015-10-03 09:40:23

It doesn't work for me too - x64 with Python 3.5, same error as skanky.

karol · 2015-10-03 20:36:49

Which version are you using? 2014.11-2 ?
https://projects.archlinux.org/svntogit … a04c649232

Arch Linux

#51 2012-12-17 13:28:13

Re: Reflector reborn

#52 2012-12-17 17:16:26

Re: Reflector reborn

#53 2014-05-23 18:14:08

Re: Reflector reborn

#54 2014-05-24 14:21:08

Re: Reflector reborn

#55 2014-05-24 16:43:31

Re: Reflector reborn

#56 2014-05-24 16:54:20

Re: Reflector reborn

#57 2014-05-30 20:12:13

Re: Reflector reborn

#58 2014-05-31 03:19:18

Re: Reflector reborn

#59 2014-05-31 03:22:00

Re: Reflector reborn

#60 2014-05-31 13:46:04

Re: Reflector reborn

#61 2014-08-15 21:27:38

Re: Reflector reborn

#62 2014-08-18 22:33:47

Re: Reflector reborn

#63 2014-08-18 22:52:15

Re: Reflector reborn

#64 2014-10-26 18:51:31

Re: Reflector reborn

#65 2014-10-26 22:07:37

Re: Reflector reborn

#66 2014-10-27 12:39:32

Re: Reflector reborn

#67 2014-10-27 22:59:52

Re: Reflector reborn

#68 2014-10-31 21:42:48

Re: Reflector reborn

#69 2014-11-01 16:25:11

Re: Reflector reborn

#70 2014-11-02 13:08:34

Re: Reflector reborn

#71 2014-11-02 13:27:04

Re: Reflector reborn

#72 2015-10-02 09:04:22

Re: Reflector reborn

#73 2015-10-02 22:05:20

Re: Reflector reborn

#74 2015-10-03 09:40:23

Re: Reflector reborn

#75 2015-10-03 20:36:49

Re: Reflector reborn

Board footer