Reflector reborn

chr0nik · 2011-11-13 02:24:01

graysky wrote:

Stupid question: how are people using reflector?

I just run it manually every so often (I'm sure there are better ways) followed by the pacman command you mentioned:

[notme@nothere bin]$ cat update-mirrors.sh 
mv /etc/pacman.d/mirrorlist /etc/pacman.d/mirrorlist.backup
reflector -l 15 --sort rate --save /etc/pacman.d/mirrorlist
diff /etc/pacman.d/mirrorlist /etc/pacman.d/mirrorlist.backup

Last edited by chr0nik (2011-11-13 02:25:48)

Xyne · 2011-11-17 08:59:47

graysky wrote:

Stupid question: how are people using reflector?

I don't use it directly. I have a fixed mirrorlist for syncing the database. When I upgrade the system with powerpill-light, it uses Reflector at run-time to get a list of the most up-to-date mirrors for parallel downloads.

Xyne · 2012-03-10 19:47:30

npouillard wrote:

Nice script, thanks Xyne.
What about putting the command used to generate the file in the generated file?

Done.

# Arch Linux mirrorlist generated by Reflector
# With:       /usr/bin/reflector -l 5
# When:       2012-03-10 19:46:34 UTC
# From:       https://www.archlinux.org/mirrors/status/json/
# Retrieved:  2012-03-10 19:46:34 UTC
# Last Check: 2012-03-10 18:54:32 UTC

Server = ftp://ftp.las.ic.unicamp.br/pub/archlinux/$repo/os/$arch
Server = http://www.las.ic.unicamp.br/pub/archlinux/$repo/os/$arch
Server = ftp://ftp.tku.edu.tw/Linux/ArchLinux/$repo/os/$arch
Server = ftp://mirrors.uk2.net/pub/archlinux/$repo/os/$arch
Server = http://ftp.tku.edu.tw/Linux/ArchLinux/$repo/os/$arch

Buce · 2012-03-23 23:07:03

I'm in the same boat as ddffnn, and would like a --sort=score option. Could this be considered for inclusion in reflector? I have a patch for it here.

Xyne · 2012-03-24 09:05:52

I've added "score" and "delay" to the sort options. The new package is available in my repo. I'll push it to [community] sometime this weekend.

Buce · 2012-03-24 15:41:57

Awesome, thanks!

mrman · 2012-03-24 19:42:18

The "score" sort option is sorting by highest score, but lower is better.

# Arch Linux mirrorlist generated by Reflector
# With:       /usr/bin/reflector -c 'United States' --sort score
# When:       2012-03-24 21:13:32 UTC
# From:       https://www.archlinux.org/mirrors/status/json/
# Retrieved:  2012-03-24 21:13:32 UTC
# Last Check: 2012-03-24 20:40:29 UTC

Server = ftp://lug.mtu.edu/archlinux/ftpfull/$repo/os/$arch
Server = http://lug.mtu.edu/archlinux/ftpfull/$repo/os/$arch
Server = http://archlinux.tserver.net/$repo/os/$arch
Server = http://archlinux.supsec.org/$repo/os/$arch
Server = ftp://ftp.osuosl.org/pub/archlinux/$repo/os/$arch
Server = http://mirrors.gigenet.com/archlinux/$repo/os/$arch
Server = http://ftp.osuosl.org/pub/archlinux/$repo/os/$arch
Server = ftp://mirror.us.leaseweb.net/archlinux/$repo/os/$arch
Server = ftp://mirror.ancl.hawaii.edu/linux/archlinux/$repo/os/$arch
Server = http://mirror.us.leaseweb.net/archlinux/$repo/os/$arch
Server = http://mirror.yellowfiber.net/archlinux/$repo/os/$arch
Server = http://mirrors.lax1.thegcloud.com/arch//$repo/os/$arch
Server = http://mirror.ancl.hawaii.edu/linux/archlinux/$repo/os/$arch
Server = ftp://archlinux.supsec.org/pub/linux/arch/$repo/os/$arch
Server = http://mirror.umd.edu/archlinux/$repo/os/$arch
Server = ftp://cosmos.cites.illinois.edu/pub/archlinux/$repo/os/$arch
Server = http://cosmos.cites.illinois.edu/pub/archlinux/$repo/os/$arch
Server = http://mirrors.rutgers.edu/archlinux/$repo/os/$arch
Server = ftp://mirrors.xmission.com/archlinux/$repo/os/$arch
Server = http://mirrors.xmission.com/archlinux/$repo/os/$arch
Server = ftp://locke.suu.edu/linux/dist/archlinux/$repo/os/$arch
Server = ftp://cake.lib.fit.edu/archlinux/$repo/os/$arch
Server = http://mirrors.liquidweb.com/archlinux/$repo/os/$arch
Server = http://archlinux.surlyjake.com/archlinux/$repo/os/$arch
Server = ftp://ftp.gtlib.gatech.edu/pub/archlinux/$repo/os/$arch
Server = http://www.gtlib.gatech.edu/pub/archlinux/$repo/os/$arch
Server = http://cake.lib.fit.edu/archlinux/$repo/os/$arch
Server = ftp://ftp.archlinux.org/$repo/os/$arch
Server = http://mirror.mocker.org/archlinux/$repo/os/$arch
Server = ftp://mirror.rit.edu/archlinux/$repo/os/$arch
Server = http://mirror.ece.vt.edu/archlinux/$repo/os/$arch
Server = http://mirrors.us.kernel.org/archlinux/$repo/os/$arch
Server = http://mirror.rit.edu/archlinux/$repo/os/$arch

Last edited by mrman (2012-03-24 21:19:20)

Xyne · 2012-03-25 09:28:37

mrman wrote:

The "score" sort option is sorting by highest score, but lower is better.

fixed

Xyne · 2012-03-26 10:35:13

The latest version has been pushed to [community] (it's already in my repo). It includes the new sort options and a fix for yesterdays change in the JSON output.

swanson · 2012-03-26 11:04:38

When doing; export LANG=C ; reflector --list-countries

I get this;
Reflector.MirrorStatusError: "failed to retrieve mirror data: time data '2012-03-26T10:20:01Z' does not match format '%Y-%m-%d %H:%M:%S'"

I've got swedish date and time settings if that might be the fault? (And just reinstalled reflector as well)

Xyne · 2012-03-26 15:50:40

@swanson
That's what the post above yours was about. It should be fixed now.

leyan · 2012-03-28 16:15:06

Nice script !

Just a small typo in the help, a "Limit" is missing for the --grep option. Also, when trying to save on a location without permission (typically /etc/pacman.d/mirrorlist without being root), the exception is not correctly handled and itself throws an exception.

I would propose something like :

@@ -336,7 +336,7 @@
help='Return the n fastest mirrors that meet the other criteria.')

filters.add_argument('--grep', dest='grep', metavar='<regex>', action='append',
- help=' the list to URLs that match at least one of the given regular expressions.')
+ help='Limit the list to URLs that match at least one of the given regular expressions.')

filters.add_argument('-l', '--latest', dest='latest', type=int, metavar='n',
help='Limit the list to the n most recently synchronized servers.')
@@ -416,7 +416,7 @@
f.write(mirrorlist)
f.close()
except IOError as e:
- stderr.write(e)
+ stderr.write('error: %s\n' % e.strerror)
exit(1)
else:
print(mirrorlist)

Xyne · 2012-03-28 17:04:32

@leyan
Thanks, I've updated the code. I'll upload it as soon as I figure out wtf is going on with gpg.

Please use code tags in the future,

like this

medianmajik · 2012-06-15 19:47:49

Mirror Script that I don't quite have running correctly.

#!/bin/bash
## Written by MedianMajik.

find /etc/pacman.d/ -type f -name "mirrorlist.*" -printf '%A+ \n' -print0 -delete
find /etc/pacman.d/ -type f -name "mirrorlist" -printf 'mv "%h/%f" "%h/%f.SAF\n"' ; 
reflector -c "United States" -a 1 -f 6 --sort rate --save /etc/pacman.d/mirrorlist
diff /etc/pacman.d/mirrorlist /etc/pacman.d/mirrorlist.SAF >> /etc/pacman.d/mirror.log
pacman -Syy
echo "Available packages for @box as of `date +%c`" > /etc/pacman.d/package.log
pacman -Qu >> /etc/pacman.d/package.log

Standard Ouput and Standard Error

mv "/etc/pacman.d/mirrorlist" "/etc/pacman.d/mirrorlist.SAF
"Traceback (most recent call last):
  File "/usr/bin/reflector", line 5, in <module>
    Reflector.main()
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 405, in main
    ms, mirrors = process_options(options)
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 378, in process_options
    mirrors = ms.sort(mirrors, 'rate')
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 164, in sort
    mirrors = self.rate(mirrors)
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 224, in rate
    q_in.join()
  File "/usr/lib/python3.2/queue.py", line 82, in join
    self.all_tasks_done.wait()
  File "/usr/lib/python3.2/threading.py", line 235, in wait
    waiter.acquire()
KeyboardInterrupt
/home/js/.scripts/update-mirrors.sh: line 7: /etc/pacman.d/mirror.log: Permission denied
warning: database file for 'extra' does not exist
error: you cannot perform this operation unless you are root.
/home/js/.scripts/update-mirrors.sh: line 9: /etc/pacman.d/package.log: Permission denied
/home/js/.scripts/update-mirrors.sh: line 10: /etc/pacman.d/package.log: Permission denied

Xyne · 2012-06-18 00:23:16

@medianmajik
What is the script doing and what do you expect it to do?

All I see is that you are killing it with a KeyboardInterrupt, which is not a bug. I could update the code to catch that and avoid the tracedump, but that would only be a cosmetic change.

Btw, you shouldn't use "pacman -Syy" in a script unless you update the system immediately afterward. It can break the system. Example:

pacman -Syy
# some time later
pacman -S <some library>

Now only the library has been updated without updating packages that depend on it. Those packages depend on the old version and will no longer work. If this affects a critical package, the system will be unusable. There is a thread somewhere on the forum that goes into detail about this.

Look at the way paconky handles update detection. You can also use an alternative database with pacman ("pacman -b /some/temporary/path ...") but you will need to copy or symlink in the local database.

TheSaint · 2012-07-26 14:51:50

I'm facing some problem

reflector -f 10 --save /tmp/mirrorlist
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python3.2/threading.py", line 740, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.2/threading.py", line 693, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 281, in worker
    l = len(f.read())
  File "/usr/lib/python3.2/http/client.py", line 496, in read
    s = self._safe_read(self.length)
  File "/usr/lib/python3.2/http/client.py", line 590, in _safe_read
    chunk = self.fp.read(min(amt, MAXAMOUNT))
  File "/usr/lib/python3.2/socket.py", line 276, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.2/threading.py", line 740, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.2/threading.py", line 693, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 281, in worker
    l = len(f.read())
  File "/usr/lib/python3.2/http/client.py", line 496, in read
    s = self._safe_read(self.length)
  File "/usr/lib/python3.2/http/client.py", line 590, in _safe_read
    chunk = self.fp.read(min(amt, MAXAMOUNT))
  File "/usr/lib/python3.2/socket.py", line 276, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python3.2/threading.py", line 740, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.2/threading.py", line 693, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 280, in worker
    with urllib.request.urlopen(req, None, self.connection_timeout) as f:
  File "/usr/lib/python3.2/urllib/request.py", line 138, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.2/urllib/request.py", line 369, in open
    response = self._open(req, data)
  File "/usr/lib/python3.2/urllib/request.py", line 387, in _open
    '_open', req)
  File "/usr/lib/python3.2/urllib/request.py", line 347, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.2/urllib/request.py", line 1155, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/usr/lib/python3.2/urllib/request.py", line 1140, in do_open
    r = h.getresponse()
  File "/usr/lib/python3.2/http/client.py", line 1049, in getresponse
    response.begin()
  File "/usr/lib/python3.2/http/client.py", line 346, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.2/http/client.py", line 308, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.2/socket.py", line 276, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

The version is the latest

Name           : reflector
Version        : 2012.7.15-1
URL            : http://xyne.archlinux.ca/projects/reflector
Licenses       : GPL

I think it need some fix. I'll look at is.

Last edited by TheSaint (2012-07-26 14:54:33)

Xyne · 2012-07-26 21:15:22

I've released an update to catch those errors. Let me know if they persist.

TheSaint · 2012-07-28 16:18:08

Just THANK YOU

Gesh · 2012-12-12 16:24:33

Thanks a lot. This script is a life saver.
However, it has the unfortunate bug of requiring the existence of $HOME/.cache in order to run.
Also, where are the sources for your programs? It would be very helpful if I could see what could
have caused the problem.

Thanks,
Gesh

Gesh · 2012-12-12 16:36:07

Sorry. Reading through Reflector.py I found that it reads XDG_CACHE_HOME which for some
reason is set on my computer. Unsetting it makes the problems vanish.

Gesh

browntown · 2012-12-15 04:47:22

I'm new and might be doing something wrong but when executing the command

# reflector --verbose -l 5 --sort rate --save /etc/pacman.d/mirrorlist

I get the following error

[root@archbang ~]# reflector --verbose -l 5 --sort rate --save /etc/pacman.d/mirrorlist
error: failed to cache JSON data ([Errno 2] No such file or directory: '/root/.cache/mirrorstatus.json')

Let me know what information I can give that can help problemsolve.

karol · 2012-12-15 05:12:44

Have you tried running reflector as a regular user with sudo?
I did run the command you used as root and it worked fine.

ewaller · 2012-12-15 05:24:44

browntown wrote:

I'm new and might be doing something wrong but when executing the command
# reflector --verbose -l 5 --sort rate --save /etc/pacman.d/mirrorlist 
I get the following error
[root@archbang ~]# reflector --verbose -l 5 --sort rate --save /etc/pacman.d/mirrorlist
error: failed to cache JSON data ([Errno 2] No such file or directory: '/root/.cache/mirrorstatus.json')
Let me know what information I can give that can help problemsolve.

If only you were running Arch, we could help you.

browntown · 2012-12-15 06:20:45

<Insert smarta$$ response here>
$ uname -r
3.6.10-1-ARCH

$ sudo did the trick thanks for the tip karol

Xyne · 2012-12-15 23:56:28

The "no such file or directory" errors should be fixed with the version in my repo. I'll push it to [community] soonish.

Arch Linux

#26 2011-11-13 02:24:01

Re: Reflector reborn

#27 2011-11-17 08:59:47

Re: Reflector reborn

#28 2012-03-10 19:47:30

Re: Reflector reborn

#29 2012-03-23 23:07:03

Re: Reflector reborn

#30 2012-03-24 09:05:52

Re: Reflector reborn

#31 2012-03-24 15:41:57

Re: Reflector reborn

#32 2012-03-24 19:42:18

Re: Reflector reborn

#33 2012-03-25 09:28:37

Re: Reflector reborn

#34 2012-03-26 10:35:13

Re: Reflector reborn

#35 2012-03-26 11:04:38

Re: Reflector reborn

#36 2012-03-26 15:50:40

Re: Reflector reborn

#37 2012-03-28 16:15:06

Re: Reflector reborn

#38 2012-03-28 17:04:32

Re: Reflector reborn

#39 2012-06-15 19:47:49

Re: Reflector reborn

#40 2012-06-18 00:23:16

Re: Reflector reborn

#41 2012-07-26 14:51:50

Re: Reflector reborn

#42 2012-07-26 21:15:22

Re: Reflector reborn

#43 2012-07-28 16:18:08

Re: Reflector reborn

#44 2012-12-12 16:24:33

Re: Reflector reborn

#45 2012-12-12 16:36:07

Re: Reflector reborn

#46 2012-12-15 04:47:22

Re: Reflector reborn

#47 2012-12-15 05:12:44

Re: Reflector reborn

#48 2012-12-15 05:24:44

Re: Reflector reborn

#49 2012-12-15 06:20:45

Re: Reflector reborn

#50 2012-12-15 23:56:28

Re: Reflector reborn

Board footer