You are not logged in.

#51 2012-12-17 13:28:13

Gesh
Member
Registered: 2010-04-12
Posts: 5

Re: Reflector reborn

As a workaround in the meantime, you can unset XDG_CACHE_HOME before calling reflector, which will make it use a directory in /tmp
This is caused by Arch supporting the XDG specified environment variables without creating the directories they point to.

Offline

#52 2012-12-17 17:16:26

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Reflector reborn

The new version is in [community] now, so workarounds are no longer necessary.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#53 2014-05-23 18:14:08

adventurer
Member
Registered: 2014-05-04
Posts: 119

Re: Reflector reborn

Hi Xyne,

as a rather new Arch user I've started to use Reflector just a couple of days ago.

I'm using it as suggested on the Reflector wiki page:

reflector --verbose -l 20 -p http --sort rate --save /etc/pacman.d/mirrorlist

It works very well but I have some question regarding the arguments and filters:

1. How is "--sort rank" different from "--fastest n"? My interpretation is that "--sort rank" already delivers the fastest mirrors so that "--fastest n" only limits that list to n mirrors but doesn't change their sequence. Is this correct?
2. Does "rank" correspond to the "Duration" mentioned on the Mirror Status page? Is replacing "--sort rank" with "--sort score" recommended? What's your opinion on this?
3. Does using the "--threads n" argument make a big performance difference? In my tests it was faster but that might have been due to caching in dnsmasq.

Thanks for a very useful tool!

Last edited by adventurer (2014-05-23 18:14:32)

Offline

#54 2014-05-24 14:21:08

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Reflector reborn

Hi adventurer,

First a minor disclaimer: I did not write the wiki page nor do I personally recommend using reflector that way. On my own systems I use a static mirrorlist and only use reflector indirectly through powerpill for faster downloads. That stated, there's nothing necessary wrong with that command. I simply prefer to choose which mirrors to trust for database downloads manually.

1. There is no --sort rank. I assume that you mean --sort rate, which sorts the mirrors by download rate after filters have been applied. For example, if you filter by country then sort by rate, only the mirrors from the selected country will be rated by speed. When you use --fastest n, reflector will rate all the mirrors (in the absence of other filters) and then return the n fastest ones. To rate a mirror, reflector downloads the core database from each mirror which can take a very long time and waste both your own and the mirrors' bandwidth if you try to rate too many (the full list contains over 200 mirrors).

2. "rate" is the download rate determined by downloading the core database from the mirror to your computer. It is specific to your connection, which in turn depends on your ISP, geographic location, etc. The score on the Mirror Status page is determined by the Arch Linux server. I don't know exactly how they calculate it but it depends on the state of synchronization of the mirror and perhaps on the connection.

Again, given that I use powerpill, I am not concerned with per-mirror download rates so i never use --sort rate. If I had to choose, I would probably prefer score over rate as I would rather have the most up-to-date mirror than the fastest one.

3. It depends on your connection to the mirrors. If each connection is only using part of your bandwidth then increasing the number of threads will speed up the rating operation. The operation will take the least time when your bandwidth is maxed, but that will also give skewed results because the calculated download rate to some servers will be reduced due to connection saturation on your end. I believe that increasing the number of threads until your bandwidth is almost maxed will give the best results.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#55 2014-05-24 16:43:31

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,422
Website

Re: Reflector reborn

Xyne wrote:

For example, if you filter by country then sort by rate, only the mirrors from the selected country will be rated by speed. When you use --fastest n, reflector will rate all the mirrors (in the absence of other filters) and then return the n fastest ones.

Can you clarify this - that statement seems to contradict the description in the help output:

  -f n, --fastest n     Return the n fastest mirrors that meet the other
                        criteria. Do not use this option without other
                        filtering options.

Specifically, you just mentioned that this parameter would get the fastest N of all mirrors in the absence of other filters, while the help output indicates it finds the fastest N out of those that meet all other provided criteria.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#56 2014-05-24 16:54:20

adventurer
Member
Registered: 2014-05-04
Posts: 119

Re: Reflector reborn

Thanks a lot, Xyne, your reply was very helpful! BTW, "rank" was a typo, indeed, I meant "rate", of course wink

But I still have a couple of questions if you don't mind.

Xyne wrote:

1. There is no --sort rank. I assume that you mean --sort rate, which sorts the mirrors by download rate after filters have been applied. For example, if you filter by country then sort by rate, only the mirrors from the selected country will be rated by speed. When you use --fastest n, reflector will rate all the mirrors (in the absence of other filters) and then return the n fastest ones.

You wrote: "in the absence of other filters". But if there are other filters? More specifically, what's the difference between

reflector -l 20 -p http --sort rate

and

reflector -l 20 -p http --fastest 10

and

reflector -l 20 -p http --sort rate --fastest 10

The different behaviour of reflector in those cases is not yet clear to me.

If I had to choose, I would probably prefer score over rate as I would rather have the most up-to-date mirror than the fastest one.

But isn't that already taken into account by using --latest n, which "limits the list to the n most recently synchronized servers."? In other words, what's the benefit of --sort score if you already use --latest n ? How are both related?

Last edited by adventurer (2014-05-24 16:56:09)

Offline

#57 2014-05-30 20:12:13

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Reflector reborn

adventurer wrote:
reflector -l 20 -p http --sort rate

That will generate a list of the 20 most recently synchronized mirrors, sorted by rate.

adventurer wrote:
reflector -l 20 -p http --fastest 10

That will give you an unsorted list of the fastest 10 mirrors among the 20 most recently sync'd.

adventurer wrote:
reflector -l 20 -p http --sort rate --fastest 10

That will give you the same as above but sorted by rate.

adventurer wrote:

But isn't that already taken into account by using --latest n, which "limits the list to the n most recently synchronized servers."? In other words, what's the benefit of --sort score if you already use --latest n ? How are both related?

My understanding is that some mirrors do not fully synchronize so they may appear in the latest 20 yet still be missing packages. Those mirrors should have lower scores. I'm not really sure how the it works though and I believe that most mirrors fully sync each time they do so the score is probably irrelevant most of the time for end users.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#58 2014-05-31 03:19:18

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Reflector reborn

Trilby wrote:

Specifically, you just mentioned that this parameter would get the fastest N of all mirrors in the absence of other filters, while the help output indicates it finds the fastest N out of those that meet all other provided criteria.

Sorry, I missed your post.

I don't see any contradiction. Each filter is an additional criterion. If you apply no filters then all mirrors meet your criteria.

The "fastest" filter is applied to the remaining servers after all other filters have been applied (as I recall). If no other filters have been applied then it will rate all of the mirrors and return the fastest n mirrors. If you have applied other filters such as "country", then it will only rate the mirrors that match the target country and return the n fastest among those.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#59 2014-05-31 03:22:00

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,422
Website

Re: Reflector reborn

Thanks your previous post already cleared up any confusion I had.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#60 2014-05-31 13:46:04

adventurer
Member
Registered: 2014-05-04
Posts: 119

Re: Reflector reborn

Xyne, thanks again for your post. Again very helpful and much appreciated.

Offline

#61 2014-08-15 21:27:38

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Reflector reborn

Updated to 2014.8. Changelog:

  • new --info option for returning mirror info instead of a mirror list

  • sorting and filtering by country now works with country codes (also included in the output of --list-countries)

Examples:

$ reflector --list-countries
Australia                                   AU 15
Austria                                     AT  1
Bangladesh                                  BD  2
Belarus                                     BY  2
Belgium                                     BE  3
Brazil                                      BR  2
Bulgaria                                    BG  1
Canada                                      CA  9
China                                       CN  7
Colombia                                    CO  1
Croatia                                     HR  1
Czech Republic                              CZ  8
Denmark                                     DK  2
Ecuador                                     EC  2
Estonia                                     EE  2
Finland                                     FI  2
France                                      FR 11
Germany                                     DE 44
Greece                                      GR  6
Hong Kong                                   HK  2
Hungary                                     HU  1
Iceland                                     IS  3
India                                       IN  2
Indonesia                                   ID  1
Ireland                                     IE  1
Israel                                      IL  1
Italy                                       IT  1
Japan                                       JP  4
Kazakhstan                                  KZ  2
Korea (the Republic of)                     KR  2
Lithuania                                   LT  1
Luxembourg                                  LU  1
Macedonia (the former Yugoslav Republic of) MK  2
Netherlands                                 NL  9
New Caledonia                               NC  4
New Zealand                                 NZ  1
Norway                                      NO  3
Philippines                                 PH  1
Poland                                      PL  4
Portugal                                    PT  3
Romania                                     RO  3
Russian Federation                          RU  2
Serbia                                      RS  1
Singapore                                   SG  2
Slovakia                                    SK  3
Spain                                       ES  2
Sri Lanka                                   LK  1
Sweden                                      SE  6
Switzerland                                 CH  1
Taiwan (Province of China)                  TW  4
United Kingdom                              GB  8
United States                               US 45
$ reflector -l 5 -c US --info
http://mirrors.kernel.org/archlinux/$repo/os/$arch
completion_pct  : 1.0
country         : United States
country_code    : US
delay           : 2292
duration_avg    : 0.702270154442106
duration_stddev : 0.598905511896356
last_sync       : 2014-08-15 21:00:01 UTC
protocol        : http
score           : 1.9378423330051289

https://mirrors.kernel.org/archlinux/$repo/os/$arch
completion_pct  : 1.0
country         : United States
country_code    : US
delay           : 2306
duration_avg    : 0.532400697469711
duration_stddev : 0.169243353766836
last_sync       : 2014-08-15 21:00:01 UTC
protocol        : https
score           : 1.3421996067921027

rsync://mirrors.kernel.org/archlinux/$repo/os/$arch
completion_pct  : 1.0
country         : United States
country_code    : US
delay           : 2306
duration_avg    : 4.76898066486631
duration_stddev : 3.57896372787145
last_sync       : 2014-08-15 21:00:01 UTC
protocol        : rsync
score           : 8.988499948293315

http://www.gtlib.gatech.edu/pub/archlinux/$repo/os/$arch
completion_pct  : 1.0
country         : United States
country_code    : US
delay           : 2374
duration_avg    : 0.458140905414309
duration_stddev : 0.208162151169742
last_sync       : 2014-08-15 20:50:01 UTC
protocol        : http
score           : 1.3257475010284956

rsync://rsync.gtlib.gatech.edu/archlinux/$repo/os/$arch
completion_pct  : 1.0
country         : United States
country_code    : US
delay           : 2374
duration_avg    : 1.36274749892099
duration_stddev : 0.567510278579429
last_sync       : 2014-08-15 20:50:01 UTC
protocol        : rsync
score           : 2.5897022219448633

My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#62 2014-08-18 22:33:47

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Reflector reborn

One more thing: the current reflector doesn't provide pacman-mirrorlist anymore.

Offline

#63 2014-08-18 22:52:15

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Reflector reborn

I don't remember where it was discussed before and I don't have time to check right now, but given that the package would have to include or overwrite the mirrorlist file it would have to conflict with the mirrorlist package.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#64 2014-10-26 18:51:31

Nietzsche
Member
Registered: 2014-10-26
Posts: 15

Re: Reflector reborn

Hello, I really like your script, but I keep getting errors with it on a specific machine. Here's the traceback:

sudo reflector --sort rate -n 10 --save /etc/pacman.d/mirrorlist
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3.4/threading.py", line 921, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.4/threading.py", line 869, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.4/site-packages/Reflector.py", line 372, in worker
    size = len(f.read())
  File "/usr/lib/python3.4/http/client.py", line 512, in read
    s = self._safe_read(self.length)
  File "/usr/lib/python3.4/http/client.py", line 664, in _safe_read
    raise IncompleteRead(b''.join(s), amt)
http.client.IncompleteRead: IncompleteRead(37189 bytes read, 82939 more expected)

Offline

#65 2014-10-26 22:07:37

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Reflector reborn

It seems that one of the mirrors is misreporting the size of a file or failing to send it.

I have added an exception handler to work around it. Can you please upgrade to the latest version of Reflector in the repo on my site and let me know if it works? When I know that it does, I will push the update to [community].


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#66 2014-10-27 12:39:32

Nietzsche
Member
Registered: 2014-10-26
Posts: 15

Re: Reflector reborn

Looks like the same error to me.

pacman -Qi reflector | grep Version
Version               : 2014.10-1

sudo reflector --sort rate -n 15 --save /etc/pacman.d/mirrorlist
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/lib/python3.4/threading.py", line 921, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.4/threading.py", line 869, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.4/site-packages/Reflector.py", line 370, in worker
    size = len(f.read())
  File "/usr/lib/python3.4/http/client.py", line 512, in read
    s = self._safe_read(self.length)
  File "/usr/lib/python3.4/http/client.py", line 664, in _safe_read
    raise IncompleteRead(b''.join(s), amt)
http.client.IncompleteRead: IncompleteRead(37189 bytes read, 82923 more expected)

Thanks for the fast reply, byt the way.

Offline

#67 2014-10-27 22:59:52

Jristz
Member
From: America/Santiago
Registered: 2011-06-11
Posts: 1,022

Re: Reflector reborn

Is that one mirror, I use 200 and because that one mirror giving the problem.
I can confirm in my system that waiting 20 mins to let it finish not work, reflector simply stop in that problem.


Well, I suppose that this is somekind of signature, no?

Offline

#68 2014-10-31 21:42:48

egroeg
Member
Registered: 2011-02-23
Posts: 40

Re: Reflector reborn

Jristz wrote:

Is that one mirror, I use 200 and because that one mirror giving the problem.
I can confirm in my system that waiting 20 mins to let it finish not work, reflector simply stop in that problem.

What mirror was it?

Offline

#69 2014-11-01 16:25:11

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Reflector reborn

Sorry for the late reply. Please try reflector 2014-11 and let me know if it works now.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#70 2014-11-02 13:08:34

Nietzsche
Member
Registered: 2014-10-26
Posts: 15

Re: Reflector reborn

It works perfectly now.

Offline

#71 2014-11-02 13:27:04

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Reflector reborn

Thanks. I'll push it to [community].


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#72 2015-10-02 09:04:22

skanky
Member
From: WAIS
Registered: 2009-10-23
Posts: 1,847

Re: Reflector reborn

Hi Xyne, just to let you know, I'm getting an error trying to run reflector, since the most recent python update to 3.5.

Traceback (most recent call last):
  File "/usr/bin/reflector", line 3, in <module>
    import Reflector
ImportError: No module named 'Reflector'

Last edited by skanky (2015-10-02 09:05:36)


"...one cannot be angry when one looks at a penguin."  - John Ruskin
"Life in general is a bit shit, and so too is the internet. And that's all there is." - scepticisle

Offline

#73 2015-10-02 22:05:20

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Reflector reborn

Works for me on my 32-bit Arch with [testing] repos enabled.

Offline

#74 2015-10-03 09:40:23

predmijat
Member
Registered: 2014-09-30
Posts: 39

Re: Reflector reborn

It doesn't work for me too - x64 with Python 3.5, same error as skanky.

Offline

#75 2015-10-03 20:36:49

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Reflector reborn

Which version are you using? 2014.11-2 ?
https://projects.archlinux.org/svntogit … a04c649232

Offline

Board footer

Powered by FluxBB