*****

ProtoXyne · 2008-07-24 12:31:43

Please delete this account.

Last edited by ProtoXyne (2010-09-15 17:34:52)

shining · 2008-07-24 15:46:27

I just wonder if Gerbra mirrorcheck tool could not do the same itself more easily.

ProtoXyne · 2008-07-24 16:02:35

Please delete this account.

Last edited by ProtoXyne (2010-09-15 17:36:10)

awagner · 2008-08-19 09:24:01

Hi - and thanks for the contribution first of all!

AFAIU the powerpill thread, the topmost server in the list is in a way special, in that it is the "trusted server" from which the validity of mirrored packages are inferred. Is there any way to have a specific mirror come out topmost in reflector (so that I could, say, have a recently synced but far-away mirror as trusted server in powerpill)?

Xyne · 2008-08-19 11:30:25

awagner wrote:

Hi - and thanks for the contribution first of all!

Thanks

awagner wrote:

AFAIU the powerpill thread, the topmost server in the list is in a way special, in that it is the "trusted server" from which the validity of mirrored packages are inferred. Is there any way to have a specific mirror come out topmost in reflector (so that I could, say, have a recently synced but far-away mirror as trusted server in powerpill)?

You could either edit the output manually the same way that you edit the default mirror list (just move your preferred mirror to the top) or you could add the line "Server = <server>" to the pacman.conf file just before the "Include = <mirror list>" line under each repo. If you do the latter, you would only need to do that once.

I could make reflector check for a file (e.g. "important_mirrors") and parse servers in it so that it would always include a certain mirror at the top and others in the list. I could also use the pacman.conf parser from powerpill to add a "keep top mirror" option to reflector.

EDIT:
I was just thinking about the alterations to reflector that I mentioned and I decided that they would just be unnecessarily complicated and "unarchy" (I would need to add checks and options to specify repos in case of different main mirrors etc).

Last edited by Xyne (2008-08-19 11:56:01)

shining · 2008-08-19 11:42:22

Off Topic : why do you have 2 forums accounts ? it is confusing

Xyne · 2008-08-19 11:59:32

shining wrote:

Off Topic : why do you have 2 forums accounts ? it is confusing

Confusing for you and annoying for me. I wanted to change my account name but someone pointed me to the "sorry, but it would be too much trouble for the admins" thread, so I had no choice but to create a new account. I would let the old one die but it seems silly to repost the powerpill thread and I'm not sure if it would count as spam if I reposted this thread under the new account.

Last edited by Xyne (2008-11-17 04:37:32)

shining · 2008-08-19 12:10:50

Xyne wrote:

shining wrote:
Off Topic : why do you have 2 forums accounts ? it is confusing
Confusing for you and annoying for me. I wanted to change my account name but someone pointed me to the "sorry, but it would be too much trouble for the admins" thread, so I had no choice but to create a new account.

Hmm ok, no problem
Actually, having "a.k.a. Xyne" in the signature of the first account removes a lot of confusion, so don't worry.
I was probably more curious than anything.

Back to the topic, I didn't subscribe to this thread, so I missed your answer one month ago.
I just sent a mail to Gerbra to point him to this thread and to ask him for more information about his tool.

awagner · 2008-08-19 12:58:57

Xyne wrote:

... or you could add the line "Server = <server>" to the pacman.conf file just before the "Include = <mirror list>" line under each repo. If you do the latter, you would only need to do that once.

Thanks, that's a good idea. (I was thinking along the lines of having reflector rank my mirrors by a cronjob but making sure that even without manual intervention I am keeping an "authoritative" server topmost, i.e. exempt from the possible ranking rotations. That will be handled fine by your suggestion, I suppose.)

Cheers,
Andreas

Last edited by awagner (2008-08-19 13:50:31)

GerBra · 2008-08-19 13:22:01

Hi, i found reflector a nice tool.

If you wish i could add this functionality to the script. My script itself is in Ruby and could be found only on my
server ;-) But i put it online now for your interest:
http://users.archlinux.de/~gerbra/mirrorcheck-code/
I have stripped my mirror database (the xml file) cause there are nonpublic informations in.
I left one mirror for example of the xml structure.

The ruby code have had heavy modifications during the versions, so the code is not the best. I'm not a real
developer, but i have commented most things (i hope) ;-)
I think i would make a git repo on archlinux.de for this.

The script itself works on my server, generate the html file and copy this and the news file to the archlinux.de
server.

Currently i'm working on a ruby rails application for the whole application (database organisation, list and statistics, easily add/mod changes and generate emails to the admins and so on). With this i think it would be easy to implement list generation for whole mirrorlists and ranked mirrorlists and....

One hint of (perhaps automated) switching your download mirror: you must consider if you "trust" all mirrors the same. It's good that we become more and more mirrors, but IMHO this leads also to a call for signed packages.

Gerhard

Xyne · 2008-08-19 14:07:26

GerBra wrote:

Hi, i found reflector a nice tool.
If you wish i could add this functionality to the script. My script itself is in Ruby and could be found only on my
server ;-) But i put it online now for your interest:
http://users.archlinux.de/~gerbra/mirrorcheck-code/
I have stripped my mirror database (the xml file) cause there are nonpublic informations in.
I left one mirror for example of the xml structure.
The ruby code have had heavy modifications during the versions, so the code is not the best. I'm not a real
developer, but i have commented most things (i hope) ;-)
I think i would make a git repo on archlinux.de for this.
The script itself works on my server, generate the html file and copy this and the news file to the archlinux.de
server.
Currently i'm working on a ruby rails application for the whole application (database organisation, list and statistics, easily add/mod changes and generate emails to the admins and so on). With this i think it would be easy to implement list generation for whole mirrorlists and ranked mirrorlists and....

Thanks, that sounds good. I'll take a look, although I've never done anything in Ruby, so I'm not sure how much I'll understand... maybe this will give me that reason to learn it that I've been waiting for.

GerBra wrote:

One hint of (perhaps automated) switching your download mirror: you must consider if you "trust" all mirrors the same. It's good that we become more and more mirrors, but IMHO this leads also to a call for signed packages.

I've thought about that before too, but as long as your database is updated from a trusted mirror, any altered package should be rejected due to a mismatched checksum, right? I download all of my packages with powerpill so everything is checked against the database checksums 3 times (aria2 in the metalink, powerpill against the database, then pacman when before it installs). If you have a corrupted database, then there is a problem of course.

EDIT: I just realized that the database download might fail from the main trusted mirror and default to another one. Maybe we could add a "trusted mirror list" parser to pacman so that it would only accept databases from those mirrors (this is all assuming that MD5 checksums are still reasonably good integrity checks).

Last edited by Xyne (2008-08-19 14:19:56)

Xiong Chiamiov · 2008-12-23 20:47:17

Howdy. Decided to try out another one of your scripts, since I've found Powerpill to be so incredibly useful.

I was running the sample code, with a slight modification, since I'd rather have it just replace my mirrorlist (of which I have a backup, of course).

sudo reflector -h 24 -t 25 -o /etc/pacman.d/mirrorlist

It got to the "invoking rankmirrors" part in about a second, then stayed there for a good 5 mintues. A slight bit of experimenting showed the same slowness with any arguments that invoke rankmirrors. The thing that I thought was odd was that I wasn't having any CPU usage from the script, so it didn't seem that it was doing a super-intensive sort or anything.
When I interrupted it, this is what I got:

pearson /etc/pacman.d  $  sudo reflector -h 24 -t 25 -o /etc/pacman.d/mirrorlist
Invoking 'rankmirrors'... this may take a while.
^C
pearson /etc/pacman.d  $  close failed in file object destructor:
Error in sys.excepthook:

Original exception was:

pearson /etc/pacman.d  $

Xyne · 2008-12-23 21:06:08

The rankmirrors script (which is separate from reflector) tests each server passed to it. It's not CPU intensive, but it can take a long time because it's running various network tests (I don't know which).

You can try running rankmirrors with different numbers of servers to see what's taking so long. My recommendation though is to not overwrite your main mirrorlist with the output of reflector unless you check it because it's server-agnostic. If a rogue server were to end up on that list and at the top of your list, pacman would then use it for database updates. It's very unlikely that this would ever happen though.

For the record, powerpill doesn't have to worry about this because it checks package integrity against the database without ever touching the database itself, so it doesn't matter if it grabs a corrupted file off of a rogue server as long as the database is fine.

Xiong Chiamiov · 2008-12-23 21:23:35

Ah, I see. I didn't realize that rankmirrors was a separate application. I don't know how reflector hooks into it (I'm too lazy to take a look), but perhaps you could use the -v (verbose) option of rankmirrors to give some information to the user, just so they know something's happening? Again, I'm making suggestions without seeing the code, so I have no idea how practical it actually is.

So, the additional_mirrors (or whatever it was called) in the powerpill config directory (which, btw, is ~/.config/..., not ~/config, at least for me - the latter is what you have on the powerpill page) lists additional mirrors for powerpill to use when downloading packages, with the idea being that you can use reflector to keep that full of fast, up-to-date mirrors, and have one trusted mirror in your actual /etc/pacman.d/mirrorlist?

shining · 2008-12-23 21:36:08

rankmirrors is a simple python script, it is easy to read. The only thing it does is retrieving 50kb from the server and timing that download.
http://projects.archlinux.org/?p=pacman … b=HEAD#l63

Xiong Chiamiov · 2008-12-23 22:35:16

shining wrote:

rankmirrors is a simple python script, it is easy to read. The only thing it does is retrieving 50kb from the server and timing that download.
http://projects.archlinux.org/?p=pacman … b=HEAD#l63

It was not rankmirrors that I was too lazy to read the source for, but reflector, which, while short, is in Perl, and I'm not getting the Llama book for another two days, so my Perl knowledge is rather limited.

Xyne · 2008-12-23 23:02:50

Xiong Chiamiov wrote:

So, the additional_mirrors (or whatever it was called) in the powerpill config directory (which, btw, is ~/.config/..., not ~/config, at least for me - the latter is what you have on the powerpill page) lists additional mirrors for powerpill to use when downloading packages, with the idea being that you can use reflector to keep that full of fast, up-to-date mirrors, and have one trusted mirror in your actual /etc/pacman.d/mirrorlist?

Yep.

I've fixed the typo on the powerpill info page (thanks for bringing it to my attention).

As for verbose output of rankmirrors, I'll take a look at what I would need to do to implement it.

Xyne · 2008-12-23 23:31:04

I've added verbose output from rankmirrors.

Xiong Chiamiov · 2008-12-24 00:02:23

Xyne wrote:

I've added verbose output from rankmirrors.

Looks nice, thanks.

One last question as far as usage. "reflector -h 24 -t 25 -o reflector_mirrorlist" will output the file to the current directory, but which directory does it need to be in for additional_mirrors.conf to pick it up? ~/.config/powerpill, or /etc/pacman.d? Or rather, from looking at pacman.conf, should I give it an absolute directory?

Xyne · 2008-12-24 00:32:52

The path in additional_mirrors.conf is either absolute or relative to the given configuration directory. If you want to keep it in ~/.config/powerpill/ but store reflector_mirrorlist in /etc/pacman.d/, specify the full path in additional_mirrors.conf ("Include = /etc/pacman.d/reflector_mirrorlist").

Shunpike · 2009-04-23 17:02:49

The --fastest option seems to be broken : the list is not generated by rankmirrors, and the output is not limited to the given number.
The --rank option works fine.

Xyne · 2009-04-23 17:47:42

Thanks, Shunpike. It should work now (perl-xyne-arch>=0.43).

Shunpike · 2009-04-23 18:34:50

Solved, thanks ;-)

Last edited by Shunpike (2009-04-23 18:35:00)

MindTooth · 2009-04-24 07:03:18

root /etc/pacman.d #  reflector -h 24 -t 25 -o reflector_mirrorlist
Use of uninitialized value in numeric le (<=) at /usr/share/perl5/vendor_perl/Xyne/Arch/Reflector.pm line 158.
root /etc/pacman.d #

What can cause this?

Birger

Shunpike · 2009-04-24 07:40:24

If you are using the latest version, syntax has changed, use -f instead of -t ;-)
Try reflector --help too.

Last edited by Shunpike (2009-04-24 07:40:39)

Arch Linux

#1 2008-07-24 12:31:43

*****

#2 2008-07-24 15:46:27

Re: *****

#3 2008-07-24 16:02:35

Re: *****

#4 2008-08-19 09:24:01

Re: *****

#5 2008-08-19 11:30:25

Re: *****

#6 2008-08-19 11:42:22

Re: *****

#7 2008-08-19 11:59:32

Re: *****

#8 2008-08-19 12:10:50

Re: *****

#9 2008-08-19 12:58:57

Re: *****

#10 2008-08-19 13:22:01

Re: *****

#11 2008-08-19 14:07:26

Re: *****

#12 2008-12-23 20:47:17

Re: *****

#13 2008-12-23 21:06:08

Re: *****

#14 2008-12-23 21:23:35

Re: *****

#15 2008-12-23 21:36:08

Re: *****

#16 2008-12-23 22:35:16

Re: *****

#17 2008-12-23 23:02:50

Re: *****

#18 2008-12-23 23:31:04

Re: *****

#19 2008-12-24 00:02:23

Re: *****

#20 2008-12-24 00:32:52

Re: *****

#21 2009-04-23 17:02:49

Re: *****

#22 2009-04-23 17:47:42

Re: *****

#23 2009-04-23 18:34:50

Re: *****

#24 2009-04-24 07:03:18

Re: *****

#25 2009-04-24 07:40:24

Re: *****

Board footer