The most preferred way to send commands to bash in a python script

huzo11 · 2018-11-12 05:54:58

Hello,

So I wanted to write a python script to automate the installation of certain types of AUR packages. However, I am unsure about the most preferred, if not the best way to send commands to bash using python such as downloading AUR packages from the website and installing them. And as a relatively non-experienced person, I would like to hear your suggestions before starting this mini-project.

Thanks!

ewaller · 2018-11-12 06:44:17

IMHO, subprocess. Unless 'preferred' means 'most simple'. If it does, then use os.system

bulletmark · 2018-11-12 07:31:49

Python evolved over time with a number of unrelated and ad-hoc methods to run external programs like os.system, backquote, os.spawn etc. They were generally adopted from other languages for alternative use cases. Around python 2.4 all these were depreciated when the subprocess module was introduced to replace them in a consolidated manner. Even then, subprocess itself evolved further until subprocess.run() was introduced in 3.5 and that is what you should now prefer, except for some advanced cases which require subprocess.Popen(). I have found it interesting to watch this particular API design converge over the many years it took the Python devs to get to where they are now.

Trilby · 2018-11-12 14:39:38

Why would you spawn a bash process from python? Just do the shell work in python. The above mentioned `subprocess` is also my recommendation for starting some other process - but there is no case in which that other process should be BASH. You may run makepkg, or pacman via subprocess, but what do you think you need BASH for?

My own AUR tool written in python uses subprocess.run for spawning git commands to clone and subprocess.check_output to get the results from pacman for local packages. I don't recall the details, but I believe there has been some back-and-forth on whether check_output of Popen would be better for the latter use, but for my purposes the former works well.

Last edited by Trilby (2018-11-12 14:44:38)

huzo11 · 2018-11-12 15:35:57

Trilby wrote:

Why would you spawn a bash process from python? Just do the shell work in python. The above mentioned `subprocess` is also my recommendation for starting some other process - but there is no case in which that other process should be BASH. You may run makepkg, or pacman via subprocess, but what do you think you need BASH for?
My own AUR tool written in python uses subprocess.run for spawning git commands to clone and subprocess.check_output to get the results from pacman for local packages. I don't recall the details, but I believe there has been some back-and-forth on whether check_output of Popen would be better for the latter use, but for my purposes the former works well.

Thank you for your suggestion. I would like to state that I am not a linux expert. Therefore, I could not totally understand what you meant. Wouldn't running pacman, makepkg via subprocess be considered sending commands to bash? Aren't such commands bash commands themselves?

Thanks!

Trilby · 2018-11-12 15:48:08

huzo11 wrote:

Wouldn't running pacman, makepkg via subprocess be considered sending commands to bash? Aren't such commands bash commands themselves?

No to both.

Git and makepkg are binary executables: they can be executed directly. You can type them on the command line of an interactive shell (bash, ash, ksh, etc) and the shell will spawn git or makepkg as a subprocess of itself, but this is pointless if your goal is just to run git or makepkg: why add a pointless (bloated) middleman?

The only reason you'd need bash is if you wanted to run a loop, conditional, or variable expansion in the command. For example, one could theoretically set an environment variable 'pkgURL' and then run a command `git clone $pkgURL` which would require a shell to expand the variable, so in python that might be something like:

subprocess.run(['bash' , '-c', '"git clone $pkgURL"'])

But if you have the pkgURL in a variable in python, there is no need to export it to the environment, launch a bash shell to launch git and interpret the variable, instead, run git directly:

subprocess.run(['git', 'clone', pkgURL])

This does not start any shell to run git, it runs git directly.

Part of this is just terminological issues, but I think there is a bit more to it than that. Binary executables do not need a shell to run. But even if it is just terminology, your search results will be much better if you google (or ddg) for something like 'run git from python' or 'start other process from python' rather than 'send bash commands from python' as BASH here is a red herring.

For added background, there is a linux system call `exec` (there's actually a family of exec functions in C like execv execl execvp ...). This `exec` call starts a new process running the binary executable passed to this system call. When you run a bash shell and you type a line and hit enter, much of the time that line is chopped up (tokenized) by bash, then just sent to the exec system call. Python's subprocess is the python interface to this same `exec` system call (more or less). So python can `exec` git directly, there is no need to `exec` bash just to ask bash to `exec` git.

Last edited by Trilby (2018-11-12 15:51:06)

huzo11 · 2018-11-12 15:54:12

Trilby wrote:

Git and makepkg are binary executables: they can be executed directly. You can type them on the command line of an interactive shell (bash, ash, ksh, etc) and the shell will spawn git or makepkg as a subprocess of itself, but this is pointless if your goal is just to run git or makepkg: why add a pointless (bloated) middleman?

That was a very intuitive explanation. Thank you for elaborating. Can we say that every package's bash commands are basically binary executables and do not necessarily need a "middleman"? Or is it not always the case?

Trilby · 2018-11-12 16:04:21

It's not *always* the case.

If you look under /usr/bin/ you will find most of the executable stuff on your system. If you run `file` on any of those, you can see what they are:

$ file /usr/bin/ls
/usr/bin/ls: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=0dd69c924e36ce85c93964d2364e645509dc82ea, stripped
$ file /usr/bin/git
/usr/bin/git: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=93847465fbc376bb7bc4ba23590c5504efa27def, stripped

The two above confirm they are binary executables. But other items are scripts ... in fact, I'll highlight how I kinda' lied in my previous post:

$ file /usr/bin/makepkg
/usr/bin/makepkg: Bourne-Again shell script, ASCII text executable

makepkg is actually a bash script. The first line has a "shebang" (#!/usr/bin/bash), and the file has executable permission. So when this file is `exec`ed the system see's the shebang and starts bash as an interpreter for the script. So bash is needed here regardless. But if you use the longer `subprocess` command above from python, you'd launch bash to exec makepkg which in turn would launch a new bash process. So you'd add an *additional* bash process which is not needed: you'd have bash->bash->makepkg as the process tree rather than just bash->makepkg.

There are, of course, other interpreters other than bash:

$ file /usr/bin/gtester-report
/usr/bin/gtester-report: a /usr/bin/env python3 script, ASCII text executable

Gtester-report is a python script. So if you `exec` it, the system will start a python process to interpret gtester-report. But here, there is no bash needed at all. If you start bash to run gtester-report you'll have bash->python->gtester-report as the process tree rather than just python->gtester-report which is all that is needed.

huzo11 · 2018-11-12 21:43:57

Trilby wrote:

It's not *always* the case.
If you look under /usr/bin/ you will find most of the executable stuff on your system. If you run `file` on any of those, you can see what they are:
$ file /usr/bin/ls
/usr/bin/ls: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=0dd69c924e36ce85c93964d2364e645509dc82ea, stripped
$ file /usr/bin/git
/usr/bin/git: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=93847465fbc376bb7bc4ba23590c5504efa27def, stripped
The two above confirm they are binary executables. But other items are scripts ... in fact, I'll highlight how I kinda' lied in my previous post:
$ file /usr/bin/makepkg
/usr/bin/makepkg: Bourne-Again shell script, ASCII text executable
makepkg is actually a bash script. The first line has a "shebang" (#!/usr/bin/bash), and the file has executable permission. So when this file is `exec`ed the system see's the shebang and starts bash as an interpreter for the script. So bash is needed here regardless. But if you use the longer `subprocess` command above from python, you'd launch bash to exec makepkg which in turn would launch a new bash process. So you'd add an *additional* bash process which is not needed: you'd have bash->bash->makepkg as the process tree rather than just bash->makepkg.
There are, of course, other interpreters other than bash:
$ file /usr/bin/gtester-report
/usr/bin/gtester-report: a /usr/bin/env python3 script, ASCII text executable
Gtester-report is a python script. So if you `exec` it, the system will start a python process to interpret gtester-report. But here, there is no bash needed at all. If you start bash to run gtester-report you'll have bash->python->gtester-report as the process tree rather than just python->gtester-report which is all that is needed.

Ahh I see. Everything is crystal clear now. Thanks a billion!

Arch Linux

#1 2018-11-12 05:54:58

The most preferred way to send commands to bash in a python script

#2 2018-11-12 06:44:17

Re: The most preferred way to send commands to bash in a python script

#3 2018-11-12 07:31:49

Re: The most preferred way to send commands to bash in a python script

#4 2018-11-12 14:39:38

Re: The most preferred way to send commands to bash in a python script

#5 2018-11-12 15:35:57

Re: The most preferred way to send commands to bash in a python script

#6 2018-11-12 15:48:08

Re: The most preferred way to send commands to bash in a python script

#7 2018-11-12 15:54:12

Re: The most preferred way to send commands to bash in a python script

#8 2018-11-12 16:04:21

Re: The most preferred way to send commands to bash in a python script

#9 2018-11-12 21:43:57

Re: The most preferred way to send commands to bash in a python script

Board footer