You are not logged in.

#26 2011-11-13 02:24:01

chr0nik
Member
From: Michigan, USA
Registered: 2011-03-13
Posts: 31

Re: Reflector reborn

graysky wrote:

Stupid question: how are people using reflector?

I just run it manually every so often (I'm sure there are better ways) followed by the pacman command you mentioned:

[notme@nothere bin]$ cat update-mirrors.sh 
mv /etc/pacman.d/mirrorlist /etc/pacman.d/mirrorlist.backup
reflector -l 15 --sort rate --save /etc/pacman.d/mirrorlist
diff /etc/pacman.d/mirrorlist /etc/pacman.d/mirrorlist.backup

Last edited by chr0nik (2011-11-13 02:25:48)

Offline

#27 2011-11-17 08:59:47

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,965
Website

Re: Reflector reborn

graysky wrote:

Stupid question: how are people using reflector?

I don't use it directly. I have a fixed mirrorlist for syncing the database. When I upgrade the system with powerpill-light, it uses Reflector at run-time to get a list of the most up-to-date mirrors for parallel downloads.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#28 2012-03-10 19:47:30

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,965
Website

Re: Reflector reborn

npouillard wrote:

Nice script, thanks Xyne.

What about putting the command used to generate the file in the generated file?

Done.

# Arch Linux mirrorlist generated by Reflector
# With:       /usr/bin/reflector -l 5
# When:       2012-03-10 19:46:34 UTC
# From:       https://www.archlinux.org/mirrors/status/json/
# Retrieved:  2012-03-10 19:46:34 UTC
# Last Check: 2012-03-10 18:54:32 UTC

Server = ftp://ftp.las.ic.unicamp.br/pub/archlinux/$repo/os/$arch
Server = http://www.las.ic.unicamp.br/pub/archlinux/$repo/os/$arch
Server = ftp://ftp.tku.edu.tw/Linux/ArchLinux/$repo/os/$arch
Server = ftp://mirrors.uk2.net/pub/archlinux/$repo/os/$arch
Server = http://ftp.tku.edu.tw/Linux/ArchLinux/$repo/os/$arch

My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#29 2012-03-23 23:07:03

Buce
Member
From: Duluth, Minnesota, US
Registered: 2009-03-17
Posts: 23

Re: Reflector reborn

I'm in the same boat as ddffnn, and would like a --sort=score option. Could this be considered for inclusion in reflector? I have a patch for it here.

Offline

#30 2012-03-24 09:05:52

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,965
Website

Re: Reflector reborn

I've added "score" and "delay" to the sort options. The new package is available in my repo. I'll push it to [community] sometime this weekend.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#31 2012-03-24 15:41:57

Buce
Member
From: Duluth, Minnesota, US
Registered: 2009-03-17
Posts: 23

Re: Reflector reborn

Awesome, thanks!

Offline

#32 2012-03-24 19:42:18

mrman
Member
Registered: 2010-02-05
Posts: 26

Re: Reflector reborn

The "score" sort option is sorting by highest score, but lower is better.

# Arch Linux mirrorlist generated by Reflector
# With:       /usr/bin/reflector -c 'United States' --sort score
# When:       2012-03-24 21:13:32 UTC
# From:       https://www.archlinux.org/mirrors/status/json/
# Retrieved:  2012-03-24 21:13:32 UTC
# Last Check: 2012-03-24 20:40:29 UTC

Server = ftp://lug.mtu.edu/archlinux/ftpfull/$repo/os/$arch
Server = http://lug.mtu.edu/archlinux/ftpfull/$repo/os/$arch
Server = http://archlinux.tserver.net/$repo/os/$arch
Server = http://archlinux.supsec.org/$repo/os/$arch
Server = ftp://ftp.osuosl.org/pub/archlinux/$repo/os/$arch
Server = http://mirrors.gigenet.com/archlinux/$repo/os/$arch
Server = http://ftp.osuosl.org/pub/archlinux/$repo/os/$arch
Server = ftp://mirror.us.leaseweb.net/archlinux/$repo/os/$arch
Server = ftp://mirror.ancl.hawaii.edu/linux/archlinux/$repo/os/$arch
Server = http://mirror.us.leaseweb.net/archlinux/$repo/os/$arch
Server = http://mirror.yellowfiber.net/archlinux/$repo/os/$arch
Server = http://mirrors.lax1.thegcloud.com/arch//$repo/os/$arch
Server = http://mirror.ancl.hawaii.edu/linux/archlinux/$repo/os/$arch
Server = ftp://archlinux.supsec.org/pub/linux/arch/$repo/os/$arch
Server = http://mirror.umd.edu/archlinux/$repo/os/$arch
Server = ftp://cosmos.cites.illinois.edu/pub/archlinux/$repo/os/$arch
Server = http://cosmos.cites.illinois.edu/pub/archlinux/$repo/os/$arch
Server = http://mirrors.rutgers.edu/archlinux/$repo/os/$arch
Server = ftp://mirrors.xmission.com/archlinux/$repo/os/$arch
Server = http://mirrors.xmission.com/archlinux/$repo/os/$arch
Server = ftp://locke.suu.edu/linux/dist/archlinux/$repo/os/$arch
Server = ftp://cake.lib.fit.edu/archlinux/$repo/os/$arch
Server = http://mirrors.liquidweb.com/archlinux/$repo/os/$arch
Server = http://archlinux.surlyjake.com/archlinux/$repo/os/$arch
Server = ftp://ftp.gtlib.gatech.edu/pub/archlinux/$repo/os/$arch
Server = http://www.gtlib.gatech.edu/pub/archlinux/$repo/os/$arch
Server = http://cake.lib.fit.edu/archlinux/$repo/os/$arch
Server = ftp://ftp.archlinux.org/$repo/os/$arch
Server = http://mirror.mocker.org/archlinux/$repo/os/$arch
Server = ftp://mirror.rit.edu/archlinux/$repo/os/$arch
Server = http://mirror.ece.vt.edu/archlinux/$repo/os/$arch
Server = http://mirrors.us.kernel.org/archlinux/$repo/os/$arch
Server = http://mirror.rit.edu/archlinux/$repo/os/$arch

Last edited by mrman (2012-03-24 21:19:20)

Offline

#33 2012-03-25 09:28:37

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,965
Website

Re: Reflector reborn

mrman wrote:

The "score" sort option is sorting by highest score, but lower is better.

fixed


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#34 2012-03-26 10:35:13

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,965
Website

Re: Reflector reborn

The latest version has been pushed to [community] (it's already in my repo). It includes the new sort options and a fix for yesterdays change in the JSON output.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#35 2012-03-26 11:04:38

swanson
Member
From: Sweden
Registered: 2011-02-05
Posts: 759

Re: Reflector reborn

When doing; export LANG=C ; reflector --list-countries


I get this;
Reflector.MirrorStatusError: "failed to retrieve mirror data: time data '2012-03-26T10:20:01Z' does not match format '%Y-%m-%d %H:%M:%S'"

I've got swedish date and time settings  if that might be the fault? (And just reinstalled reflector as well)

Offline

#36 2012-03-26 15:50:40

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,965
Website

Re: Reflector reborn

@swanson
That's what the post above yours was about. It should be fixed now.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#37 2012-03-28 16:15:06

leyan
Member
Registered: 2011-01-25
Posts: 1

Re: Reflector reborn

Nice script !

Just a small typo in the help, a "Limit" is missing for the --grep option. Also, when trying to save on a location without permission (typically /etc/pacman.d/mirrorlist without being root), the exception is not correctly handled and itself throws an exception.

I would propose something like :

@@ -336,7 +336,7 @@
                     help='Return the n fastest mirrors that meet the other criteria.')

   filters.add_argument('--grep', dest='grep', metavar='<regex>', action='append',
-                    help=' the list to URLs that match at least one of the given regular expressions.')
+                    help='Limit the list to URLs that match at least one of the given regular expressions.')

   filters.add_argument('-l', '--latest', dest='latest', type=int, metavar='n',
                     help='Limit the list to the n most recently synchronized servers.')
@@ -416,7 +416,7 @@
       f.write(mirrorlist)
       f.close()
     except IOError as e:
-      stderr.write(e)
+      stderr.write('error: %s\n' % e.strerror)
       exit(1)
   else:
     print(mirrorlist)

Offline

#38 2012-03-28 17:04:32

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,965
Website

Re: Reflector reborn

@leyan
Thanks, I've updated the code. I'll upload it as soon as I figure out wtf is going on with gpg.

Please use code tags in the future,

like this

My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#39 2012-06-15 19:47:49

medianmajik
Member
Registered: 2010-05-27
Posts: 53

Re: Reflector reborn

Mirror Script that I don't quite have running correctly.

#!/bin/bash
## Written by MedianMajik.

find /etc/pacman.d/ -type f -name "mirrorlist.*" -printf '%A+ \n' -print0 -delete
find /etc/pacman.d/ -type f -name "mirrorlist" -printf 'mv "%h/%f" "%h/%f.SAF\n"' ; 
reflector -c "United States" -a 1 -f 6 --sort rate --save /etc/pacman.d/mirrorlist
diff /etc/pacman.d/mirrorlist /etc/pacman.d/mirrorlist.SAF >> /etc/pacman.d/mirror.log
pacman -Syy
echo "Available packages for @box as of `date +%c`" > /etc/pacman.d/package.log
pacman -Qu >> /etc/pacman.d/package.log

Standard Ouput and Standard Error

mv "/etc/pacman.d/mirrorlist" "/etc/pacman.d/mirrorlist.SAF
"Traceback (most recent call last):
  File "/usr/bin/reflector", line 5, in <module>
    Reflector.main()
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 405, in main
    ms, mirrors = process_options(options)
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 378, in process_options
    mirrors = ms.sort(mirrors, 'rate')
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 164, in sort
    mirrors = self.rate(mirrors)
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 224, in rate
    q_in.join()
  File "/usr/lib/python3.2/queue.py", line 82, in join
    self.all_tasks_done.wait()
  File "/usr/lib/python3.2/threading.py", line 235, in wait
    waiter.acquire()
KeyboardInterrupt
/home/js/.scripts/update-mirrors.sh: line 7: /etc/pacman.d/mirror.log: Permission denied
warning: database file for 'extra' does not exist
error: you cannot perform this operation unless you are root.
/home/js/.scripts/update-mirrors.sh: line 9: /etc/pacman.d/package.log: Permission denied
/home/js/.scripts/update-mirrors.sh: line 10: /etc/pacman.d/package.log: Permission denied

One thing at a time

Offline

#40 2012-06-18 00:23:16

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,965
Website

Re: Reflector reborn

@medianmajik
What is the script doing and what do you expect it to do?

All I see is that you are killing it with a KeyboardInterrupt, which is not a bug. I could update the code to catch that and avoid the tracedump, but that would only be a cosmetic change.


Btw, you shouldn't use "pacman -Syy" in a script unless you update the system immediately afterward. It can break the system. Example:

pacman -Syy
# some time later
pacman -S <some library>

Now only the library has been updated without updating packages that depend on it. Those packages depend on the old version and will no longer work. If this affects a critical package, the system will be unusable. There is a thread somewhere on the forum that goes into detail about this.

Look at the way paconky handles update detection. You can also use an alternative database with pacman ("pacman -b /some/temporary/path ...") but you will need to copy or symlink in the local database.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#41 2012-07-26 14:51:50

TheSaint
Member
From: my computer
Registered: 2007-08-19
Posts: 1,523

Re: Reflector reborn

I'm facing some problem

reflector -f 10 --save /tmp/mirrorlist
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python3.2/threading.py", line 740, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.2/threading.py", line 693, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 281, in worker
    l = len(f.read())
  File "/usr/lib/python3.2/http/client.py", line 496, in read
    s = self._safe_read(self.length)
  File "/usr/lib/python3.2/http/client.py", line 590, in _safe_read
    chunk = self.fp.read(min(amt, MAXAMOUNT))
  File "/usr/lib/python3.2/socket.py", line 276, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.2/threading.py", line 740, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.2/threading.py", line 693, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 281, in worker
    l = len(f.read())
  File "/usr/lib/python3.2/http/client.py", line 496, in read
    s = self._safe_read(self.length)
  File "/usr/lib/python3.2/http/client.py", line 590, in _safe_read
    chunk = self.fp.read(min(amt, MAXAMOUNT))
  File "/usr/lib/python3.2/socket.py", line 276, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python3.2/threading.py", line 740, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.2/threading.py", line 693, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.2/site-packages/Reflector.py", line 280, in worker
    with urllib.request.urlopen(req, None, self.connection_timeout) as f:
  File "/usr/lib/python3.2/urllib/request.py", line 138, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.2/urllib/request.py", line 369, in open
    response = self._open(req, data)
  File "/usr/lib/python3.2/urllib/request.py", line 387, in _open
    '_open', req)
  File "/usr/lib/python3.2/urllib/request.py", line 347, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.2/urllib/request.py", line 1155, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/usr/lib/python3.2/urllib/request.py", line 1140, in do_open
    r = h.getresponse()
  File "/usr/lib/python3.2/http/client.py", line 1049, in getresponse
    response.begin()
  File "/usr/lib/python3.2/http/client.py", line 346, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.2/http/client.py", line 308, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.2/socket.py", line 276, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

The version is the latest

Name           : reflector
Version        : 2012.7.15-1
URL            : http://xyne.archlinux.ca/projects/reflector
Licenses       : GPL

I think it need some fix. I'll look at is.

Last edited by TheSaint (2012-07-26 14:54:33)


do it good first, it will be faster than do it twice the saint wink

Offline

#42 2012-07-26 21:15:22

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,965
Website

Re: Reflector reborn

I've released an update to catch those errors. Let me know if they persist.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#43 2012-07-28 16:18:08

TheSaint
Member
From: my computer
Registered: 2007-08-19
Posts: 1,523

Re: Reflector reborn

Just THANK YOU


do it good first, it will be faster than do it twice the saint wink

Offline

#44 2012-12-12 16:24:33

Gesh
Member
Registered: 2010-04-12
Posts: 5

Re: Reflector reborn

Thanks a lot. This script is a life saver.
However, it has the unfortunate bug of requiring the existence of $HOME/.cache in order to run.
Also, where are the sources for your programs? It would be very helpful if I could see what could
have caused the problem.

Thanks,
Gesh

Offline

#45 2012-12-12 16:36:07

Gesh
Member
Registered: 2010-04-12
Posts: 5

Re: Reflector reborn

Sorry. Reading through Reflector.py I found that it reads XDG_CACHE_HOME which for some
reason is set on my computer. Unsetting it makes the problems vanish.

Gesh

Offline

#46 2012-12-15 04:47:22

browntown
Member
From: Salem, OR USA
Registered: 2012-11-20
Posts: 63

Re: Reflector reborn

I'm new and might be doing something wrong but when executing the command

# reflector --verbose -l 5 --sort rate --save /etc/pacman.d/mirrorlist 

I get the following error

[root@archbang ~]# reflector --verbose -l 5 --sort rate --save /etc/pacman.d/mirrorlist
error: failed to cache JSON data ([Errno 2] No such file or directory: '/root/.cache/mirrorstatus.json')

Let me know what information I can give that can help problemsolve.

Offline

#47 2012-12-15 05:12:44

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: Reflector reborn

Have you tried running reflector as a regular user with sudo?
I did run the command you used as root and it worked fine.

Offline

#48 2012-12-15 05:24:44

ewaller
Administrator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 20,085

Re: Reflector reborn

browntown wrote:

I'm new and might be doing something wrong but when executing the command

# reflector --verbose -l 5 --sort rate --save /etc/pacman.d/mirrorlist 

I get the following error

[root@archbang ~]# reflector --verbose -l 5 --sort rate --save /etc/pacman.d/mirrorlist
error: failed to cache JSON data ([Errno 2] No such file or directory: '/root/.cache/mirrorstatus.json')

Let me know what information I can give that can help problemsolve.

If only you were running Arch, we could help you.


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

Offline

#49 2012-12-15 06:20:45

browntown
Member
From: Salem, OR USA
Registered: 2012-11-20
Posts: 63

Re: Reflector reborn

<Insert smarta$$ response here>
$ uname -r
3.6.10-1-ARCH

$ sudo did the trick thanks for the tip karol

Offline

#50 2012-12-15 23:56:28

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,965
Website

Re: Reflector reborn

The "no such file or directory" errors should be fixed with the version in my repo. I'll push it to [community] soonish.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

Board footer

Powered by FluxBB