Programming thinking? For a text utility in python

ngoonee · 2014-03-31 14:15:51

I've written some code to process text from this format:-

[V1]
.     E                B
 All hail the power of Jesus' Name!
.    B                E
 Let angels prostrate fall;
.      E               B  E
 Bring forth the royal diadem,
.    E     B/D# C#m7 D  Bsus B
 and crown Him  Lord of all
.      E               B
 Bring forth the royal diadem,
.    E/G#  A   E/B  B   E
 and crown Him Lord  of all.

to and from this format:-

{c:Verse 1}
All h[E]ail the power of [B]Jesus' Name!
Let [B]angels prostrate [E]fall;
Bring [E]forth the royal [B]dia[E]dem,
and [E]crown [B/D#]Him [C#m7]Lord [D]of [Bsus]all [B]
Bring [E]forth the royal [B]diadem,
and [E/G#]crown [A]Him [E/B]Lord [B] of [E]all.

The code works, as far as I can tell. Been using it for a year or so now. Now I
want to expand it, write a simple GUI to allow someone else to easily use it
(rather than having non-computer-literate people copy pasting command line
python).

Looking back at the code I've written (and refactored at least once), it looks
like junk. Its so... procedural. I don't have much (any) formal programming
learning, everything has been picked up on the fly in the course of my
engineering degree, PhD research, and teaching duties.

How do I train myself to think like a programmer. I can already program, and my
programs tend to work more often than not. I can identify weaknesses, for
example overly verbose comments, horrible unit testing (writing code is much
more fun than writing tests), and non-existant pre-planning. But I'm not sure
how to teach myself to systematically design a program that's not just a
single-function "I want to do THIS" app.

For reference (those who have too much free time) here's the link to the two
python files which are my app. I don't expect anyone to help me with the
programming itself, but comments on my style are MUCH appreciated.

[1] - https://www.dropbox.com/s/32o6a0ei6xny3y7/song.py
[2] - https://www.dropbox.com/s/9qpqdbrxiygmlmf/convert.py

P.S. - mods, if this fits better under Off-Topic so be it its a bit of a philosophical question on programming rather than a nuts and bolts one.

jakobcreutzfeldt · 2014-03-31 18:52:35

On first glance, it's not really bad at all. I was expecting to see one massive function or, worse, some 1000-line script all on the top-level, which is what I tend to see in Python coming from people with no programming experience. One thing that I notice is that you do string concatenation with the '+' operator. This is basically an unofficial quirk of CPython and shouldn't be expected to be supported on all systems. It's better to use the join method of strings (e.g. ' '.join(["foo", "bar"]) gives "foo bar"). On the other hand, I'm mainly seeing you use string concatenation to put string variables inside messages, for which string formatting is a better solution because it's much easier to change the message later on:

# no:
raise IOError("No such file: " + path + "\n" + str(e.args))
# yes:
raise IOError("No such file: {0}\n{1}".format(path, str(e.args)))

Note that this kind of string formatting isn't supported before Python 2.6, I think (before that, it would be ("No such file: %s\n%s" % path, str(e.args)).

About the only other thing I might suggest would be to split things up further into modules. e.g. your songOS class (should be SongOS according to the usual Python style) can go in its own songos.py file. Of course, then you wouldn't have just two Python scripts but you would instead need a directory layout with __init__.py files and such. But if you were planning on distributing something, that would be the way to go. Anyway, I find that such compartmentalization helps me maintain larger programs: assume that anything that's not in the current file works like a black box; don't worry about the implementation details (out of site out of mind). But if something breaks, you know exactly where to find it, without having to scroll/search through a huge file. Of course, sometimes you just can't split a file up conveniently. Either way, this is a matter of taste.

I'm assuming you've read them but:
http://www.python.org/dev/peps/pep-0008/
http://www.python.org/dev/peps/pep-0020/

Last edited by jakobcreutzfeldt (2014-03-31 18:54:11)

firecat53 · 2014-03-31 18:56:55

Disclaimer: I've probably become a "programmer" much like you did, so take my opinion with a grain of salt! Your code looks at first glance pretty clean to me. It seems to be (without having spent much time with it) well commented, logically organized and PEP8 compliant. You use a class where a class makes sense and just functions/methods where they make more sense. I really can't answer your question about how to train yourself to be a 'better' programmer because I think through these things in much the same way you do!

Two ways you could maybe try something different are:

1. Unit testing (standing in my glass house here throwing stones...)

2. If you have a lot of if/else statements, you can try a dictionary-type construct where you define a small function, then put the function and the key into a dictionary (ok, yes I see I have even more if/else inside each function).

def up():
    if self.y == 0:
        if self.win_y > 0:
            self.win_y = self.win_y - 1
    else:
        self.y = self.y - 1
def left():
    if self.x == 0:
        if self.win_x > 0:
            self.win_x = self.win_x - 1
    else:
        self.x = self.x - 1

self.keys = {'k': up,
             'h': left}
........

if 0 < c < 256:
    c = chr(c)
    try:
        self.keys[c]()
.......

(code from here: https://github.com/firecat53/tabview/bl … abview.py)

Good luck!
Scott

drcouzelis · 2014-03-31 19:08:41

ngoonee wrote:

Looking back at the code I've written (and refactored at least once), it looks
like junk. Its so... procedural.

Are you suggesting there's something wrong with procedural programming? Procedural, object oriented, functional, and all the other programming methodologies all have their place. None of them is better than the other. It all depends on what problem you're trying to solve.

The most important thing is that a program works, which you successfully accomplished.

Raynman · 2014-03-31 19:21:21

jakobcreutzfeldt wrote:

One thing that I notice is that you do string concatenation with the '+' operator. This is basically an unofficial quirk of CPython and shouldn't be expected to be supported on all systems.

What? Concatenation with + for strings (and other sequence types) is part of the language (but indeed it's often not the best method). Were you thinking of the following?

CPython implementation detail: If s and t are both strings, some Python implementations such as CPython can usually perform an in-place optimization for assignments of the form s = s + t or s += t. When applicable, this optimization makes quadratic run-time much less likely. This optimization is both version and implementation dependent. For performance sensitive code, it is preferable to use the str.join() method which assures consistent linear concatenation performance across versions and implementations.

jakobcreutzfeldt · 2014-03-31 19:43:53

Raynman wrote:

jakobcreutzfeldt wrote:
One thing that I notice is that you do string concatenation with the '+' operator. This is basically an unofficial quirk of CPython and shouldn't be expected to be supported on all systems.
What? Concatenation with + for strings (and other sequence types) is part of the language (but indeed it's often not the best method). Were you thinking of the following?
CPython implementation detail: If s and t are both strings, some Python implementations such as CPython can usually perform an in-place optimization for assignments of the form s = s + t or s += t. When applicable, this optimization makes quadratic run-time much less likely. This optimization is both version and implementation dependent. For performance sensitive code, it is preferable to use the str.join() method which assures consistent linear concatenation performance across versions and implementations.

From PEP8:

Code should be written in a way that does not disadvantage other implementations of Python (PyPy, Jython, IronPython, Cython, Psyco, and such).
For example, do not rely on CPython's efficient implementation of in-place string concatenation for statements in the form a += b or a = a + b. This optimization is fragile even in CPython (it only works for some types) and isn't present at all in implementations that don't use refcounting. In performance sensitive parts of the library, the ''.join() form should be used instead. This will ensure that concatenation occurs in linear time across various implementations.

Maybe I did misread the "...isn't present at all" to mean the implementation, not the optimization.

Last edited by jakobcreutzfeldt (2014-03-31 19:45:01)

Alber · 2014-04-03 15:31:33

I'm no a programer.
But your code looks clear for me.
I like Python because is more important clarity that performance.
If it works and two months later you can understand it at first view, it is OK. If you need a map, two coffees, four hours to understand it, it's bad.

For tricks.
More programming, more resourceful (by experience) you get.
As firecat53, when I found a lot of ifs I try to do a "case" with a dictionary, too.
Another rule I use is that if you write the same chunk of code two times, surely, you need a function.

ngoonee · 2014-04-11 01:02:23

Thanks all. I'll address some of the points brought up one by one.

jakobcreutzfeldt wrote:

On first glance, it's not really bad at all. I was
expecting to see one massive function or, worse, some 1000-line script all on
the top-level, which is what I tend to see in Python coming from people with no
programming experience.

Well, I DO have some programming experience, and I read a lot more than I
code...

jakobcreutzfeld wrote:

One thing that I notice is that you do string
concatenation with the '+' operator. This is basically an unofficial quirk of
CPython and shouldn't be expected to be supported on all systems.

Thanks, I'll definitely keep this in mind. By which I mean search-and-replacing
everything I can see.

jakobcreutzfeld wrote:

About the only other thing I might suggest would be to split things up further into modules. e.g. your songOS class (should be SongOS according to the usual Python style) can go in its own songos.py file. Of course, then you wouldn't have just two Python scripts but you would instead need a directory layout with __init__.py files and such. But if you were planning on distributing something, that would be the way to go. Anyway, I find that such compartmentalization helps me maintain larger programs: assume that anything that's not in the current file works like a black box; don't worry about the implementation details (out of site out of mind). But if something breaks, you know exactly where to find it, without having to scroll/search through a huge file. Of course, sometimes you just can't split a file up conveniently. Either way, this is a matter of taste.
I'm assuming you've read them but:
http://www.python.org/dev/peps/pep-0008/

I use a pep8 plugin in vim to maintain some basic pep8 correctness, though I
don't know the guidelines myself.

Splitting things up would be interesting. I haven't actually done (my own)
modules in python, seems I'd just need a directory structure and __init__.py
files. In any case, yes I'll look into that.

firecat53 wrote:

Two ways you could maybe try something different are:
1. Unit testing (standing in my glass house here throwing stones...)
2. If you have a lot of if/else statements, you can try a dictionary-type construct where you define a small function, then put the function and the key into a dictionary (ok, yes I see I have even more if/else inside each function).

Regarding 1, I've tried writing tests but encounter a problem with breaking
things down. Testing a single function requires suitable input (and wrong input,
to test test cases), and I'm lazy enough to figure that it would take so much
time creating said input that the net benefit would be zero. Based on reading
other people's projects, I'd say I'm not alone in that

Regarding 2, I'm not sure how that would benefit much. Is it supposed to reduce
lines of code or improve readability? Don't exactly see how it would do either,
could you explain more?

drcouzelis wrote:

Are you suggesting there's something wrong with procedural programming? Procedural, object oriented, functional, and all the other programming methodologies all have their place. None of them is better than the other. It all depends on what problem you're trying to solve.
The most important thing is that a program works, which you successfully accomplished.

Fair enough, but I see every code I write as an opportunity to improve. I picked
up python just to teach it to the undergrads in my Artificial Intelligence
subject (rather than having them use pirated software), and used it for this
project because I thought about this project at roughly the same time.

The 'wrong' thing I see with procedural programming is simply that after having
left the code alone for a few months, I find myself having difficulty reading
it. Maintainability etc. Now I want to expand this program and its taking me
ages just to read it, such that I'm thinking "surely there must be a better
way".

Stebalien · 2014-04-11 15:43:19

@jakobcreutzfeldt You also missed:

In performance sensitive parts of the library, the ''.join() form should be used instead.

For small string concatenations in none-performance critical sections (none of this code is performance critical), you should focus on making the code readable. If `a + b` is more readable, write `a + b`. Writing `''.join([a, b])` is a premature optimization (http://c2.com/cgi/wiki?PrematureOptimization) (which (a) is less readable, (b) easier to get wrong (because it's less readable), and (c) might even perform WORSE due to the construction of the extra list object (given small strings).

As for testing, I recommend doctests for small projects like this. To make it easier to test, I recommend passing around file objects instead of paths. This way, if you need a fake file for a test case, you can just pass in a StringIO object.

firecat53 · 2014-04-11 17:56:10

ngoonee wrote:

firecat53 wrote:
2. If you have a lot of if/else statements, you can try a dictionary-type construct where you define a small function, then put the function and the key into a dictionary (ok, yes I see I have even more if/else inside each function).
Regarding 2, I'm not sure how that would benefit much. Is it supposed to reduce
lines of code or improve readability? Don't exactly see how it would do either,
could you explain more?

In my opinion it improves the readability and modularity of the code. However there are a lot of opinions surrounding lack of a 'case' statement in python...so my opinion is only one of many

Scott

ngoonee · 2014-04-14 03:19:30

firecat53 wrote:

ngoonee wrote:
firecat53 wrote:
2. If you have a lot of if/else statements, you can try a dictionary-type construct where you define a small function, then put the function and the key into a dictionary (ok, yes I see I have even more if/else inside each function).
Regarding 2, I'm not sure how that would benefit much. Is it supposed to reduce
lines of code or improve readability? Don't exactly see how it would do either,
could you explain more?
In my opinion it improves the readability and modularity of the code. However there are a lot of opinions surrounding lack of a 'case' statement in python...so my opinion is only one of many
Scott

I see. Even in languages with a case function I find myself not using them much, simply because it causes multiple branches in my thought representation of the code rather than the binary branching schemes if/else does. If they saved a lot of vertical space I'd see the point, but in C for example I found it taking just about as much room.

drcouzelis · 2014-04-14 14:56:39

ngoonee wrote:

If they saved a lot of vertical space I'd see the point, but in C for example I found it taking just about as much room.

I think the idea was that, back in the old days, a C switch statement was faster than a block of if / else statements because it's able to jump directly to the condition it needs to run, instead of going line-by-line through every condition.

But of course, with modern processors and especially modern compilers, I have a feeling this really isn't an issue any more...

Trent · 2014-04-15 00:36:02

The difference between switch/case and if/elif/else is the lack of priority. A switch statement says, "I have this value and I may need to do one of a dozen things depending on what it is." An if-else chain is a different thing entirely; it says, "I'm going to check each of these conditions in turn and do the action associated with the first one that's true".

Switch: "This is a shirt, so put it in the shirts pile."

If-else: "This sock doesn't match sock A, and it doesn't match sock B, but it matches sock C, so fold it with C and put the pair away."

Modern compilers may be able to map one to the other reasonably well in the average case, but that doesn't free you from the obligation to use what makes the intent most clear. I use switches sparingly, but when you have to select between several things with no sense of priority, they're most often the right choice.

Arch Linux

#1 2014-03-31 14:15:51

Programming thinking? For a text utility in python

#2 2014-03-31 18:52:35

Re: Programming thinking? For a text utility in python

#3 2014-03-31 18:56:55

Re: Programming thinking? For a text utility in python

#4 2014-03-31 19:08:41

Re: Programming thinking? For a text utility in python

#5 2014-03-31 19:21:21

Re: Programming thinking? For a text utility in python

#6 2014-03-31 19:43:53

Re: Programming thinking? For a text utility in python

#7 2014-04-03 15:31:33

Re: Programming thinking? For a text utility in python

#8 2014-04-11 01:02:23

Re: Programming thinking? For a text utility in python

#9 2014-04-11 15:43:19

Re: Programming thinking? For a text utility in python

#10 2014-04-11 17:56:10

Re: Programming thinking? For a text utility in python

#11 2014-04-14 03:19:30

Re: Programming thinking? For a text utility in python

#12 2014-04-14 14:56:39

Re: Programming thinking? For a text utility in python

#13 2014-04-15 00:36:02

Re: Programming thinking? For a text utility in python

Board footer