How to implement common bash idioms in Python? [closed]

Question

Any shell has several sets of features.

• The Essential Linux/Unix commands. All of these are available through the subprocess library. This isn't always the best first choice for doing all external commands. Look also at shutil for some commands that are separate Linux commands, but you could probably implement directly in your Python scripts. Another huge batch of Linux commands are in the os library; you can do these more simply in Python.

  And -- bonus! -- more quickly. Each separate Linux command in the shell (with a few exceptions) forks a subprocess. By using Python shutil and os modules, you don't fork a subprocess.

• The shell environment features. This includes stuff that sets a command's environment (current directory and environment variables and what-not). You can easily manage this from Python directly.

• The shell programming features. This is all the process status code checking, the various logic commands (if, while, for, etc.) the test command and all of it's relatives. The function definition stuff. This is all much, much easier in Python. This is one of the huge victories in getting rid of bash and doing it in Python.

• Interaction features. This includes command history and what-not. You don't need this for writing shell scripts. This is only for human interaction, and not for script-writing.

• The shell file management features. This includes redirection and pipelines. This is trickier. Much of this can be done with subprocess. But some things that are easy in the shell are unpleasant in Python. Specifically stuff like (a | b; c ) | something >result. This runs two processes in parallel (with output of a as input to b), followed by a third process. The output from that sequence is run in parallel with something and the output is collected into a file named result. That's just complex to express in any other language.

Specific programs (awk, sed, grep, etc.) can often be rewritten as Python modules. Don't go overboard. Replace what you need and evolve your "grep" module. Don't start out writing a Python module that replaces "grep".

The best thing is that you can do this in steps.

1. Replace AWK and PERL with Python. Leave everything else alone.
2. Look at replacing GREP with Python. This can be a bit more complex, but your version of GREP can be tailored to your processing needs.
3. Look at replacing FIND with Python loops that use os.walk. This is a big win because you don't spawn as many processes.
4. Look at replacing common shell logic (loops, decisions, etc.) with Python scripts.

Yes, of course :)

Take a look at these libraries which help you Never write shell scripts again (Plumbum's motto).

• Plumbum
• Sarge
• sh

Also, if you want to replace awk, sed and grep with something Python based then I recommend pyp -

"The Pyed Piper", or pyp, is a linux command line text manipulation tool similar to awk or sed, but which uses standard python string and list methods as well as custom functions evolved to generate fast results in an intense production environment.

I just discovered how to combine the best parts of bash and ipython. Up to now this seems more comfortable to me than using subprocess and so on. You can easily copy big parts of existing bash scripts and e.g. add error handling in the python way :) And here is my result:

#!/usr/bin/env ipython3

# *** How to have the most comfort scripting experience of your life ***
# ######################################################################
#
# … by using ipython for scripting combined with subcommands from bash!
#
# 1. echo "#!/usr/bin/env ipython3" > scriptname.ipy    # creates new ipy-file
#
# 2. chmod +x scriptname.ipy                            # make in executable
#
# 3. starting with line 2, write normal python or do some of
#    the ! magic of ipython, so that you can use unix commands
#    within python and even assign their output to a variable via
#    var = !cmd1 | cmd2 | cmd3                          # enjoy ;)
#
# 4. run via ./scriptname.ipy - if it fails with recognizing % and !
#    but parses raw python fine, please check again for the .ipy suffix

# ugly example, please go and find more in the wild
files = !ls *.* | grep "y"
for file in files:
  !echo $file | grep "p"
# sorry for this nonsense example ;)

See IPython docs on system shell commands and using it as a system shell.

As of 2015 and Python 3.4's release, there's now a reasonably complete user-interactive shell available at: http://xon.sh/ or https://github.com/scopatz/xonsh

The demonstration video does not show pipes being used, but they ARE supported when in the default shell mode.

Xonsh ('conch') tries very hard to emulate bash, so things you've already gained muscle memory for, like

env | uniq | sort -r | grep PATH

or

my-web-server 2>&1 | my-log-sorter

will still work fine.

The tutorial is quite lengthy and seems to cover a significant amount of the functionality someone would generally expect at a ash or bash prompt:

• Compiles, Evaluates, & Executes!
• Command History and Tab Completion
• Help & Superhelp with ? & ??
• Aliases & Customized Prompts
• Executes Commands and/or *.xsh Scripts which can also be imported
• Environment Variables including Lookup with ${}
• Input/Output Redirection and Combining
• Background Jobs & Job Control
• Nesting Subprocesses, Pipes, and Coprocesses
• Subprocess-mode when a command exists, Python-mode otherwise
• Captured Subprocess with $(), Uncaptured Subprocess with $[], Python Evaluation with @()
• Filename Globbing with * or Regular Expression Filename Globbing with Backticks