speculum - Simple, straightforward mirror list optimizer

schard · 2019-02-28 09:04:55

The other day I had a look at the source of reflector, didn't like what I saw, and wrote my own mirror list filter, sorter and limiter from scratch:
AUR | source

It does not intend to be a 1-to-1 copy in functionality of reflector. So it does not do any measurements on its own, but just filters, sorts and limits the data returned by the mirrorlist JSON endpoint.
I, however, believe that it will be sufficient for the masses, and has a sleek and strightforward code base.
Feedback is welcome.

Last edited by schard (2019-02-28 09:10:23)

eschwartz · 2019-02-28 23:22:38

And the first thing it does is add a hundred MB worth of dependencies on blas, lapack, python-numpy and python-pandas.

I'm curious what your exact issue is with reflector. (Aside for the fact that Xyne has *still* not gotten around to uploading anything to git.)

schard · 2019-03-01 07:24:46

Yes, it does. The packages are available and I did want to get rid of the reinvention of the wheel, which the original version was.
I don't think that dependencies are something that must be avoided at all cost.
I am rather fond of using existing libraries and tools if they serve my purpose.

As for reflector's codebase: It seemingly intentionally defies all Python code style guidelines and I am under the impression that the code was is obfuscated to make making it hard to read and comprehend for others, which I do not find trustworthy.

Update
I overreacted with my wording and retract any accusations against the autor of reflector.

Last edited by schard (2019-03-01 15:54:41)

Trilby · 2019-03-01 14:12:56

schard wrote:

I am under the impression that the code was obfuscated to make it hard to read and comprehend for others, which I do not find trustworthy.

I don't think this is called for. I also personally find it quite ironic.

There certainly are different coding styles. And if you don't like the way it was coded and felt you wanted to try your hand at your own version you could tinker with, that is perfectly reasonable.

I find your assertion ironic, though, as I tried to read your code. I couldn't. I actually came to the same conclusions about your code that you did about Xyne's. It looks intentionally obfuscated with a ridiculously vast amount of the code being just pointless wrapper functions resulting in someone reading your code needing to basically memorize a translation table from common library functions to your own function names that you've wrapped them with.

As for using lots of libraries, go for it, if they serve a purpose that simpler built-ins will not accomplish. However, you don't seem to be using Pandas for anything that standard data structures wouldn't do just as well. Yes, don't reinvent the wheel, but also don't needlessly stick wheels on everything just because they are cool.

I had never dug through reflector's code before, but I just did now. It's a bit longer than it would need to be to meet my needs, but then it has lots of features I don't use. However, I find it perfectly readable and well in line with all python coding standards. Personally I find the two-space indent a bit narrow, but the coding standards only specify consistency which reflector's code maintains (and lots of people think I'm a freak as I prefer 3-spaces instead of the traditional 4, so I'm not really one to speak).

Last edited by Trilby (2019-03-01 14:22:43)

schard · 2019-03-01 15:16:52

@Trilby. I criticized the code, not the coder. And I migrated to pandas, because I had to use a lot of hacks and compensations in my original version (you can see all changes in the git log).
It was also not my own idea to do that, as I submitted my code for review.
I found the idea convincing and it significantly reduced the complexity of the code, making it more readable.
As for the wrapper functions, they serve several purposes. If you disagree with parts of my code base, you are welcome to participate in the review I linked to above.

Trilby · 2019-03-01 15:30:23

List comprehension can quite easily do all that filtering that you are relying on pandas for. As for specific comments, I can't really make any as I can't make heads or tails of what most of the code is doing.

As for criticizing code, not the coder, that is simply disingenuous. You said it intentionally violated standards; as code doesn't have intentions, only a coder can, it would seem this is not a critique of the code. You also said it was obfuscated to make it hard to read; code cannot obfuscate itself nor could it have the goal of making itself hard to read. These are accusations of the code's author. And there's nothing a priori wrong with making accusations against the author of some code - but when questioned, you should either provide evidence to back the claims or an acknowledgement that they were unfounded.

Your initial post was much simpler: reflector doesn't suit you. That's fine. And you didn't like what you saw in the code. That's a critique of the code. But your second post took a much different approach which is what I responded to.

In either case, I'll leave this thread be. I didn't have any intent of participating here as I have no interest in the tool presented. I just objected to the unfounded accusations of the author of a similar tool.

Last edited by Trilby (2019-03-01 15:33:39)

eschwartz · 2019-03-01 21:02:00

schard wrote:

Yes, it does. The packages are available and I did want to get rid of the reinvention of the wheel, which the original version was.
I don't think that dependencies are something that must be avoided at all cost.
I am rather fond of using existing libraries and tools if they serve my purpose.

You can do whatever suits you best. But when submitting it to the community, the assumption is the community might be interested in using it too, in which case you might also want to consider what suits them best.

Getting back to the concrete details of my critique, python3 is a fairly common and popular programming toolkit present on most peoples' linux desktops. python-pandas is not (unless you are a data scientist).

Now consider that you've introduced a 100MB dependency to your small script. Do you really think that is the *best* way to advertise? In such situations, you should always ask yourself exactly how much you benefit from the dependency, and how much you lose.

schard wrote:

And I migrated to pandas, because I had to use a lot of hacks and compensations in my original version (you can see all changes in the git log).
It was also not my own idea to do that, as I submitted my code for review.
I found the idea convincing and it significantly reduced the complexity of the code, making it more readable.
As for the wrapper functions, they serve several purposes. If you disagree with parts of my code base, you are welcome to participate in the review I linked to above.

They already listed the caveat when initially mentioning pandas, that it introduces dependencies outside the stdlib. The person who suggested pandas also said "Of course you could just implement it using just standard library tools: [...] And this would still be vastly shorter and more readable than your code..."

Again: is the simplicity of using pandas *really* worth the quite large dependency? What about the alternative implementation which did *not* use pandas, but was still a lot less complex than your code?

schard · 2019-03-01 21:54:23

@eschwartz: I completely understand your critique. However, I'm not AppleSoft and have no interest in having this specific script on every Arch user's PC. Those who like clean code and have enough space for some dependency libraries, may use it... or don't.
I threw this script in here, to qoute the GPL, "in the hope it will be useful". If people don't think so and prefer other solutions, I don't mind.

Furthermore, following your argument, only scripts that are equal in size or larger than pandas may use it (This is an exaggeration, I know, but I hope you get my point). And I don't agree with that.
But again: Everybody feel free to use it if you find it useful. If not, don't.

Update: I have a free weekend ahead of me, and I will consider removing the pandas dependecy if I find a simpler solution.

Last edited by schard (2019-03-01 22:06:14)

eschwartz · 2019-03-01 22:07:05

schard wrote:

Furthermore, following your argument, only scripts that are equal in size or larger than pandas may use it (This is an exaggeration, I know, but I hope you get my point). And I don't agree with that.

Say what?

Any script that is a single line long, but does something that cannot be conveniently done without pandas, should feel free to use it.

Using large mathematics libraries in order to reimplement list comprehension based filtering, does not require pandas and can be conveniently done in simple and arguably elegant-to-some code that doesn't reach out of the stdlib.

But yes, I'll acknowledge I am an anti-fan of programming paradigms where people are afraid to reimplement trivial functions because "muh Unix code reuse" and instead create applications as large or larger than the average web browser *for no concrete benefit*.

Pursuing that line of reasoning, I guess I could say I do not find your code elegant at all -- most of the source code to your script is indirectly located at https://github.com/pandas-dev/pandas and isn't very elegant at all.

schard · 2019-03-02 07:49:14

I took your advise and migrated back to python standard library only.

Xyne · 2019-03-02 11:58:06

schard wrote:

As for reflector's codebase: It seemingly intentionally defies all Python code style guidelines and I am under the impression that the code was is obfuscated to make making it hard to read and comprehend for others, which I do not find trustworthy.

There was no intentional obfuscation, only organic growth over years as new functionality was shoehorned in to meet different needs. It's not pretty but it happens when priorities lie elsewhere.

Anyway, your hostility prompted me to take some time to refactor the code to make it more legible. Some legacy code will have to remain until I can update everything that uses it. I also added some options.

schard · 2019-03-02 13:01:28

Sorry for the hostility, but reading the original code hit a nerve of mine.
I did not mean to personally offend you. But I have (had) a strong opinion on reflector's codebase.

Arch Linux

#1 2019-02-28 09:04:55

speculum - Simple, straightforward mirror list optimizer

#2 2019-02-28 23:22:38

Re: speculum - Simple, straightforward mirror list optimizer

#3 2019-03-01 07:24:46

Re: speculum - Simple, straightforward mirror list optimizer

#4 2019-03-01 14:12:56

Re: speculum - Simple, straightforward mirror list optimizer

#5 2019-03-01 15:16:52

Re: speculum - Simple, straightforward mirror list optimizer

#6 2019-03-01 15:30:23

Re: speculum - Simple, straightforward mirror list optimizer

#7 2019-03-01 21:02:00

Re: speculum - Simple, straightforward mirror list optimizer

#8 2019-03-01 21:54:23

Re: speculum - Simple, straightforward mirror list optimizer

#9 2019-03-01 22:07:05

Re: speculum - Simple, straightforward mirror list optimizer

#10 2019-03-02 07:49:14

Re: speculum - Simple, straightforward mirror list optimizer

#11 2019-03-02 11:58:06

Re: speculum - Simple, straightforward mirror list optimizer

#12 2019-03-02 13:01:28

Re: speculum - Simple, straightforward mirror list optimizer

Board footer