You are not logged in.
As a workaround in the meantime, you can unset XDG_CACHE_HOME before calling reflector, which will make it use a directory in /tmp
This is caused by Arch supporting the XDG specified environment variables without creating the directories they point to.
Offline
The new version is in [community] now, so workarounds are no longer necessary.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Hi Xyne,
as a rather new Arch user I've started to use Reflector just a couple of days ago.
I'm using it as suggested on the Reflector wiki page:
reflector --verbose -l 20 -p http --sort rate --save /etc/pacman.d/mirrorlist
It works very well but I have some question regarding the arguments and filters:
1. How is "--sort rank" different from "--fastest n"? My interpretation is that "--sort rank" already delivers the fastest mirrors so that "--fastest n" only limits that list to n mirrors but doesn't change their sequence. Is this correct?
2. Does "rank" correspond to the "Duration" mentioned on the Mirror Status page? Is replacing "--sort rank" with "--sort score" recommended? What's your opinion on this?
3. Does using the "--threads n" argument make a big performance difference? In my tests it was faster but that might have been due to caching in dnsmasq.
Thanks for a very useful tool!
Last edited by adventurer (2014-05-23 18:14:32)
Offline
Hi adventurer,
First a minor disclaimer: I did not write the wiki page nor do I personally recommend using reflector that way. On my own systems I use a static mirrorlist and only use reflector indirectly through powerpill for faster downloads. That stated, there's nothing necessary wrong with that command. I simply prefer to choose which mirrors to trust for database downloads manually.
1. There is no --sort rank. I assume that you mean --sort rate, which sorts the mirrors by download rate after filters have been applied. For example, if you filter by country then sort by rate, only the mirrors from the selected country will be rated by speed. When you use --fastest n, reflector will rate all the mirrors (in the absence of other filters) and then return the n fastest ones. To rate a mirror, reflector downloads the core database from each mirror which can take a very long time and waste both your own and the mirrors' bandwidth if you try to rate too many (the full list contains over 200 mirrors).
2. "rate" is the download rate determined by downloading the core database from the mirror to your computer. It is specific to your connection, which in turn depends on your ISP, geographic location, etc. The score on the Mirror Status page is determined by the Arch Linux server. I don't know exactly how they calculate it but it depends on the state of synchronization of the mirror and perhaps on the connection.
Again, given that I use powerpill, I am not concerned with per-mirror download rates so i never use --sort rate. If I had to choose, I would probably prefer score over rate as I would rather have the most up-to-date mirror than the fastest one.
3. It depends on your connection to the mirrors. If each connection is only using part of your bandwidth then increasing the number of threads will speed up the rating operation. The operation will take the least time when your bandwidth is maxed, but that will also give skewed results because the calculated download rate to some servers will be reduced due to connection saturation on your end. I believe that increasing the number of threads until your bandwidth is almost maxed will give the best results.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
For example, if you filter by country then sort by rate, only the mirrors from the selected country will be rated by speed. When you use --fastest n, reflector will rate all the mirrors (in the absence of other filters) and then return the n fastest ones.
Can you clarify this - that statement seems to contradict the description in the help output:
-f n, --fastest n Return the n fastest mirrors that meet the other
criteria. Do not use this option without other
filtering options.
Specifically, you just mentioned that this parameter would get the fastest N of all mirrors in the absence of other filters, while the help output indicates it finds the fastest N out of those that meet all other provided criteria.
"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman
Offline
Thanks a lot, Xyne, your reply was very helpful! BTW, "rank" was a typo, indeed, I meant "rate", of course
But I still have a couple of questions if you don't mind.
1. There is no --sort rank. I assume that you mean --sort rate, which sorts the mirrors by download rate after filters have been applied. For example, if you filter by country then sort by rate, only the mirrors from the selected country will be rated by speed. When you use --fastest n, reflector will rate all the mirrors (in the absence of other filters) and then return the n fastest ones.
You wrote: "in the absence of other filters". But if there are other filters? More specifically, what's the difference between
reflector -l 20 -p http --sort rate
and
reflector -l 20 -p http --fastest 10
and
reflector -l 20 -p http --sort rate --fastest 10
The different behaviour of reflector in those cases is not yet clear to me.
If I had to choose, I would probably prefer score over rate as I would rather have the most up-to-date mirror than the fastest one.
But isn't that already taken into account by using --latest n, which "limits the list to the n most recently synchronized servers."? In other words, what's the benefit of --sort score if you already use --latest n ? How are both related?
Last edited by adventurer (2014-05-24 16:56:09)
Offline
reflector -l 20 -p http --sort rate
That will generate a list of the 20 most recently synchronized mirrors, sorted by rate.
reflector -l 20 -p http --fastest 10
That will give you an unsorted list of the fastest 10 mirrors among the 20 most recently sync'd.
reflector -l 20 -p http --sort rate --fastest 10
That will give you the same as above but sorted by rate.
But isn't that already taken into account by using --latest n, which "limits the list to the n most recently synchronized servers."? In other words, what's the benefit of --sort score if you already use --latest n ? How are both related?
My understanding is that some mirrors do not fully synchronize so they may appear in the latest 20 yet still be missing packages. Those mirrors should have lower scores. I'm not really sure how the it works though and I believe that most mirrors fully sync each time they do so the score is probably irrelevant most of the time for end users.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Specifically, you just mentioned that this parameter would get the fastest N of all mirrors in the absence of other filters, while the help output indicates it finds the fastest N out of those that meet all other provided criteria.
Sorry, I missed your post.
I don't see any contradiction. Each filter is an additional criterion. If you apply no filters then all mirrors meet your criteria.
The "fastest" filter is applied to the remaining servers after all other filters have been applied (as I recall). If no other filters have been applied then it will rate all of the mirrors and return the fastest n mirrors. If you have applied other filters such as "country", then it will only rate the mirrors that match the target country and return the n fastest among those.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Thanks your previous post already cleared up any confusion I had.
"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman
Offline
Xyne, thanks again for your post. Again very helpful and much appreciated.
Offline
Updated to 2014.8. Changelog:
new --info option for returning mirror info instead of a mirror list
sorting and filtering by country now works with country codes (also included in the output of --list-countries)
Examples:
$ reflector --list-countries
Australia AU 15
Austria AT 1
Bangladesh BD 2
Belarus BY 2
Belgium BE 3
Brazil BR 2
Bulgaria BG 1
Canada CA 9
China CN 7
Colombia CO 1
Croatia HR 1
Czech Republic CZ 8
Denmark DK 2
Ecuador EC 2
Estonia EE 2
Finland FI 2
France FR 11
Germany DE 44
Greece GR 6
Hong Kong HK 2
Hungary HU 1
Iceland IS 3
India IN 2
Indonesia ID 1
Ireland IE 1
Israel IL 1
Italy IT 1
Japan JP 4
Kazakhstan KZ 2
Korea (the Republic of) KR 2
Lithuania LT 1
Luxembourg LU 1
Macedonia (the former Yugoslav Republic of) MK 2
Netherlands NL 9
New Caledonia NC 4
New Zealand NZ 1
Norway NO 3
Philippines PH 1
Poland PL 4
Portugal PT 3
Romania RO 3
Russian Federation RU 2
Serbia RS 1
Singapore SG 2
Slovakia SK 3
Spain ES 2
Sri Lanka LK 1
Sweden SE 6
Switzerland CH 1
Taiwan (Province of China) TW 4
United Kingdom GB 8
United States US 45
$ reflector -l 5 -c US --info
http://mirrors.kernel.org/archlinux/$repo/os/$arch
completion_pct : 1.0
country : United States
country_code : US
delay : 2292
duration_avg : 0.702270154442106
duration_stddev : 0.598905511896356
last_sync : 2014-08-15 21:00:01 UTC
protocol : http
score : 1.9378423330051289
https://mirrors.kernel.org/archlinux/$repo/os/$arch
completion_pct : 1.0
country : United States
country_code : US
delay : 2306
duration_avg : 0.532400697469711
duration_stddev : 0.169243353766836
last_sync : 2014-08-15 21:00:01 UTC
protocol : https
score : 1.3421996067921027
rsync://mirrors.kernel.org/archlinux/$repo/os/$arch
completion_pct : 1.0
country : United States
country_code : US
delay : 2306
duration_avg : 4.76898066486631
duration_stddev : 3.57896372787145
last_sync : 2014-08-15 21:00:01 UTC
protocol : rsync
score : 8.988499948293315
http://www.gtlib.gatech.edu/pub/archlinux/$repo/os/$arch
completion_pct : 1.0
country : United States
country_code : US
delay : 2374
duration_avg : 0.458140905414309
duration_stddev : 0.208162151169742
last_sync : 2014-08-15 20:50:01 UTC
protocol : http
score : 1.3257475010284956
rsync://rsync.gtlib.gatech.edu/archlinux/$repo/os/$arch
completion_pct : 1.0
country : United States
country_code : US
delay : 2374
duration_avg : 1.36274749892099
duration_stddev : 0.567510278579429
last_sync : 2014-08-15 20:50:01 UTC
protocol : rsync
score : 2.5897022219448633
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
One more thing: the current reflector doesn't provide pacman-mirrorlist anymore.
Offline
I don't remember where it was discussed before and I don't have time to check right now, but given that the package would have to include or overwrite the mirrorlist file it would have to conflict with the mirrorlist package.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Hello, I really like your script, but I keep getting errors with it on a specific machine. Here's the traceback:
sudo reflector --sort rate -n 10 --save /etc/pacman.d/mirrorlist
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python3.4/threading.py", line 921, in _bootstrap_inner
self.run()
File "/usr/lib/python3.4/threading.py", line 869, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.4/site-packages/Reflector.py", line 372, in worker
size = len(f.read())
File "/usr/lib/python3.4/http/client.py", line 512, in read
s = self._safe_read(self.length)
File "/usr/lib/python3.4/http/client.py", line 664, in _safe_read
raise IncompleteRead(b''.join(s), amt)
http.client.IncompleteRead: IncompleteRead(37189 bytes read, 82939 more expected)
Offline
It seems that one of the mirrors is misreporting the size of a file or failing to send it.
I have added an exception handler to work around it. Can you please upgrade to the latest version of Reflector in the repo on my site and let me know if it works? When I know that it does, I will push the update to [community].
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Looks like the same error to me.
pacman -Qi reflector | grep Version
Version : 2014.10-1
sudo reflector --sort rate -n 15 --save /etc/pacman.d/mirrorlist
Exception in thread Thread-5:
Traceback (most recent call last):
File "/usr/lib/python3.4/threading.py", line 921, in _bootstrap_inner
self.run()
File "/usr/lib/python3.4/threading.py", line 869, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.4/site-packages/Reflector.py", line 370, in worker
size = len(f.read())
File "/usr/lib/python3.4/http/client.py", line 512, in read
s = self._safe_read(self.length)
File "/usr/lib/python3.4/http/client.py", line 664, in _safe_read
raise IncompleteRead(b''.join(s), amt)
http.client.IncompleteRead: IncompleteRead(37189 bytes read, 82923 more expected)
Thanks for the fast reply, byt the way.
Offline
Is that one mirror, I use 200 and because that one mirror giving the problem.
I can confirm in my system that waiting 20 mins to let it finish not work, reflector simply stop in that problem.
Well, I suppose that this is somekind of signature, no?
Offline
Is that one mirror, I use 200 and because that one mirror giving the problem.
I can confirm in my system that waiting 20 mins to let it finish not work, reflector simply stop in that problem.
What mirror was it?
Offline
Sorry for the late reply. Please try reflector 2014-11 and let me know if it works now.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
It works perfectly now.
Offline
Thanks. I'll push it to [community].
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Hi Xyne, just to let you know, I'm getting an error trying to run reflector, since the most recent python update to 3.5.
Traceback (most recent call last):
File "/usr/bin/reflector", line 3, in <module>
import Reflector
ImportError: No module named 'Reflector'
Last edited by skanky (2015-10-02 09:05:36)
"...one cannot be angry when one looks at a penguin." - John Ruskin
"Life in general is a bit shit, and so too is the internet. And that's all there is." - scepticisle
Offline
Works for me on my 32-bit Arch with [testing] repos enabled.
Offline
It doesn't work for me too - x64 with Python 3.5, same error as skanky.
Offline
Which version are you using? 2014.11-2 ?
https://projects.archlinux.org/svntogit … a04c649232
Offline