Python as Bash Replacement: coreutils?

NewWorld · 2014-03-22 07:13:09

Hello,

I want to start replacing my Bash scripts with Python ones, partly due to Python's performance being much better. However, all the modules I found whose goal is to aid replacing shell-scripting with Python, interface with the GNU coreutils through `subprocess`. As we all know, forking the process to call external programs is why shell scripts are that much slower.

I was wondering if there's a Python module/library that replaces the behaviours of the coreutils, so that I don't have to write them myself, e.g. a `tail` behaviour written in Python. Or any other modules/libs that would aid in shell-scripting with Python without `subprocess`.

Last edited by NewWorld (2014-03-22 07:38:36)

Mr Green · 2014-03-22 07:17:37

Only a quick search

https://pypi.python.org/pypi/pycoreutils/

NewWorld · 2014-03-22 07:23:10

Mr Green wrote:

Only a quick search
https://pypi.python.org/pypi/pycoreutils/

Thanks, I overlooked that one because it had the same name as another, which used C-bindings. I'll take a look at this module and see if it helps me. Thanks, again.

UPDATE: It's alpha and it's buggy: look at the output of `ls /` I get: http://pastie.org/8958321 The dot-files shouldn't be there, and what's that core* stuff?!

If there are any other modules that would help with shell-scripting with Python, I would like to hear them.

Last edited by NewWorld (2014-03-22 07:38:58)

tomk · 2014-03-22 09:46:29

NewWorld wrote:

It's alpha and it's buggy

Maybe you could help make it beta and less buggy?

progandy · 2014-03-22 09:56:25

You might want to start fresh. Python 3.4 has a new pathlib module that allows you to handle paths and filesystem objects as python objects.

lykwydchykyn · 2014-03-23 04:17:53

Well, "os" covers a lot of ground, but you probably knew that already. I'm not sure how you'd expect python implementations of coreutils to be faster than the C versions that BASH is calling.

NewWorld · 2014-03-23 07:17:40

lykwydchykyn wrote:

Well, "os" covers a lot of ground, but you probably knew that already. I'm not sure how you'd expect python implementations of coreutils to be faster than the C versions that BASH is calling.

I have done research which concludes that shell scripts are a lot slower than Python. See this graph from this blog post, and this graph from this blog post.

This answer on SO gives an explanation for shell scripts being slower:

shell scripts are inherently slow, especially when they use a lot of external commands like yours. Biggest reason for this is because spawning external process is rather slow, and you do it a lot of times.
If you are really after high performance processing of your data, you should write Perl or Python script which would do what you need without ever spawning any external process: no dos2unix, no grep, no cut or anything like that.

@progandy, thanks a lot for the link. This module is also available for Python 2.7, which is what I'll be writing in.

tomk wrote:

NewWorld wrote:
It's alpha and it's buggy
Maybe you could help make it beta and less buggy?

As I mentioned in the original post, I don't want to be writing these tools myself; I'm already prone to RSI pain and that's just not how I want to spend my free time just to be able to write shell-scripts easier from within Python.

Scimmia · 2014-03-23 07:24:40

NewWorld wrote:

If you are really after high performance processing of your data, you should write Perl or Python script which would do what you need without ever spawning any external process: no dos2unix, no grep, no cut or anything like that.

If high performance data processing is really needed, you shouldn't be messing with scripts at all.

NewWorld · 2014-03-23 08:49:34

Scimmia wrote:

If high performance data processing is really needed, you shouldn't be messing with scripts at all.

I just want to run a script or two every minute with cron and don't want it spiking my CPU and turning the fans on, or, when my computer is under heavy load, these "minutely" scripts contributing to the lag. I also control my volume with a Bash script interfacing `amixer` that slides the volume up/down by tapping a keyboard combo repeatedly, and it could be faster and more responsive.

Last edited by NewWorld (2014-03-23 08:51:13)

karol · 2014-03-23 09:35:21

Maybe 'nice' and 'ionice' can help here somewhere?

progandy · 2014-03-23 09:37:49

NewWorld wrote:

Scimmia wrote:
If high performance data processing is really needed, you shouldn't be messing with scripts at all.
I just want to run a script or two every minute with cron and don't want it spiking my CPU and turning the fans on, or, when my computer is under heavy load, these "minutely" scripts contributing to the lag. I also control my volume with a Bash script interfacing `amixer` that slides the volume up/down by tapping a keyboard combo repeatedly, and it could be faster and more responsive.

Python startup is slower than bash, so I don't know if it will help anything. You could create a permanent daemon instead of restarting the script with cron, that might help. For your hotkeys, it is probably more responsive if you use a daemon like volumeicon, too.

[progandy@pamobile ~]$ time (for i in {1..100} ; do bash -c "echo 123" >/dev/null; done)

real	0m0.941s
user	0m0.613s
sys	0m0.090s
[progandy@pamobile ~]$ time (for i in {1..100} ; do python -c "print(123)" >/dev/null; done)

real	0m7.667s
user	0m6.767s
sys	0m0.683s
[progandy@pamobile ~]$ time (for i in {1..100} ; do python2 -c "print(123)" >/dev/null; done)

real	0m3.245s
user	0m2.537s
sys	0m0.513s

Last edited by progandy (2014-03-23 09:39:09)

NewWorld · 2014-03-23 20:11:04

progandy wrote:

Python startup is slower than bash, so I don't know if it will help anything. You could create a permanent daemon instead of restarting the script with cron, that might help. For your hotkeys, it is probably more responsive if you use a daemon like volumeicon, too.

Thanks for that info. I think I will first try to write the scripts in Python, and, if the startup is noticeable slow I will daemonise the program with something like daemonize or python-daemon. Python has spoiled me and now I feel filthy when writing shell scripts; I now don't like this language for anything but tiny scripts/wrappers and one-liners.

karol wrote:

Maybe 'nice' and 'ionice' can help here somewhere?

I didn't know about `ionice`! Great to know about that.

I appreciate all the responses to this thread; thanks for the help and ideas.

progandy · 2014-03-23 20:23:45

NewWorld wrote:

progandy wrote:
Python startup is slower than bash, so I don't know if it will help anything. You could create a permanent daemon instead of restarting the script with cron, that might help. For your hotkeys, it is probably more responsive if you use a daemon like volumeicon, too.
Thanks for that info. I think I will first try to write the scripts in Python, and, if the startup is noticeable slow I will daemonise the program with something like daemonize or python-daemon. Python has spoiled me and now I feel filthy when writing shell scripts; I now don't like this language for anything but tiny scripts/wrappers and one-liners.

Perl is faster, or try lua lua(jit) is my choice for fast background scripts if i need more than shellscripts can deliver (e.g. gobject introspection, hotkeys, ...)

[progandy@pamobile tmp]$ time (for i in {1..100} ; do perl -e "print(123)" >/dev/null; done)

real	0m0.427s
user	0m0.170s
sys	0m0.197s
[progandy@pamobile tmp]$ time (for i in {1..100} ; do lua -e "print(123)" >/dev/null; done)

real	0m0.288s
user	0m0.010s
sys	0m0.023s

flying sheep · 2014-03-25 13:28:05

btw, if you need access to the shell after all, try this!

http://amoffat.github.io/sh/

from sh import adduser
adduser("amoffat", system=True, shell="/bin/bash", no_create_home=True)

Last edited by flying sheep (2014-03-25 13:33:25)

Arch Linux

#1 2014-03-22 07:13:09

Python as Bash Replacement: coreutils?

#2 2014-03-22 07:17:37

Re: Python as Bash Replacement: coreutils?

#3 2014-03-22 07:23:10

Re: Python as Bash Replacement: coreutils?

#4 2014-03-22 09:46:29

Re: Python as Bash Replacement: coreutils?

#5 2014-03-22 09:56:25

Re: Python as Bash Replacement: coreutils?

#6 2014-03-23 04:17:53

Re: Python as Bash Replacement: coreutils?

#7 2014-03-23 07:17:40

Re: Python as Bash Replacement: coreutils?

#8 2014-03-23 07:24:40

Re: Python as Bash Replacement: coreutils?

#9 2014-03-23 08:49:34

Re: Python as Bash Replacement: coreutils?

#10 2014-03-23 09:35:21

Re: Python as Bash Replacement: coreutils?

#11 2014-03-23 09:37:49

Re: Python as Bash Replacement: coreutils?

#12 2014-03-23 20:11:04

Re: Python as Bash Replacement: coreutils?

#13 2014-03-23 20:23:45

Re: Python as Bash Replacement: coreutils?

#14 2014-03-25 13:28:05

Re: Python as Bash Replacement: coreutils?

Board footer