Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Run programs in background and redirect their outputs to file in real time

I want to run several python scripts simultaneously in one bash session and to inspect their outputs respectively in real time. To fulfil this task, I wrote a simple bash script which is shown below:

#!/bin/bash
python 1.py > 1.output &
python 2.py > 2.output &
python 3.py > 3.output &

When I use cat 1.output command to check what have been printed in the middle of executing, however, nothing can be seen.

After thinking a while, I realise that the 1.output must be filled when 1.py finishes executing. In other words, the method which I used here is not a real time fashion.

You may propse that to wait for these python scripts' finish. The fact, unfortunately, is that all there python scripts are actually long run programs, they maybe finish after days or months, and that is why I want to inspect their outputs in real time.

Also, you may suggest me to modify the python scripts to print message to file directly instead of stdout. Sorry, the scripts here are too complex to modify, there are chucks of print function in them.

what can I do now?

like image 580
Douglas Su Avatar asked Jul 10 '15 16:07

Douglas Su


People also ask

How do I run a command in the background Linux and output to a file?

The easiest way to run a Linux background command is to add an Ampersand (&) symbol after the command. For example, if you start the gedit text editor from your terminal, you can not use the shell until you close the editor. However, when you add an extra & to your command, you'll be able to use the shell immediately.

What happens in the background when you run a Python file?

It will directly put the output in the file you have selected.

How do I keep a code running in the background?

The easiest way of running a python script to run in the background is to use cronjob feature (in macOS and Linux). In windows, we can use Windows Task Scheduler. You can then give the path of your python script file to run at a specific time by giving the time particulars.


1 Answers

The -u switch and the equivalent PYTHONUNBUFFERED environment variable forces stdout to be unbuffered. Try this:

#!/bin/bash
python -u 1.py > 1.output &
python -u 2.py > 2.output &
python -u 3.py > 3.output &

or

#!/bin/bash
export PYTHONUNBUFFERED=yes
python 1.py > 1.output &
python 2.py > 2.output &
python 3.py > 3.output &

Note that -u has side effects: read the doc to learn more.

Reference:

  • https://docs.python.org/2/using/cmdline.html#cmdoption-u
  • https://docs.python.org/2/using/cmdline.html#envvar-PYTHONUNBUFFERED
like image 190
Robᵩ Avatar answered Oct 02 '22 10:10

Robᵩ