python: read csv, execute command and write results to new vertical column

Question

I am totally new to python, I read python's csv module is great for what Id like to do. Ive spent some time trying several different methods but have not yet been able to even create an array using the fourth (vertical) column.

I have a four column csv file with hundreds of rows. Before I go on I should probably verify python can even accomplish all that Id like to do.

1. read a csv FILE,

2. executes COMMAND on fourth (vertical) column of FILE

3. the COMMAND prints

4. read each line for HEALTHY (from COMMAND)

5. write HEALTHY on new fifth column to NEW_FILE with all five columns

6. loop until first empty row of FILE

example FILE (comma delimited in cell view)

  HOST                    PLATFORM        ARCH               COMMAND
  server1                 win             x86_64             python '/root/server1.py'
  server2                 linux           x86_64             python '/root/server2.py'
  server3                 linux           x86_64             python '/root/server3.py'

example COMMAND

  # python '/root/server1.py'
  --------------------
  Error: Could not open /root/server1.py


  # python '/root/server2.py'
  --------------------
  server2 p1 (NTFS)       output1:100  output:200    HEALTHY:Yes
  --------------------


  # python 'root/server3.py'
  --------------------
  server3 p1 (linux)       output1:100  output:200    HEALTHY:No
  server3 p2 (linux)       output1:100  output:200    HEALTHY:Yes
  server3 p3 (swap)       output1:100  output:200    HEALTHY:No
  --------------------

if multiple lines of HEALTHY and all do not equal Yes, HEALTHY equals "No"

if HEALTHY is not found on any lines, HEALTHY equals "Error Scanning"

This is what I have so far

  #!/usr/bin/python
  #

  import csv
  import subprocess

  # read csv file
  csv_file = open("my_list.csv", "rb")
  my_csv_reader = csv.reader(csv_file, delimiter=",")
  my_data_list = []
  for row in my_csv_reader:
          print row
          my_data_list.append(row)
  csv_file.close()

  # write csv file
  csv_file = open("new_data.csv", "wb")
  my_csv_writer = csv.writer(csv_file, delimiter=",")
  for row in my_data_list:
          my_csv_writer.writerow(row)
  csv_file.close()

  # running commands, getting output
  # run COMMAND column from csv_file, use "python 'my_script.py'" for now
  # my_script.py only for now: print "HEALTHY:Yes"
  p = subprocess.Popen("python '/root/my_script.py'",stdout=subprocess.PIPE,stderr=subprocess.PIPE)
  output, errors = p.communicate()
  print output
  print errors

Executing the above:

  # python '/root/this_script.py'
  ['HOST', 'PLATFORM', 'ARCH', 'COMMAND']
  ['server1', 'win', 'x86_64', "python '/root/server1.py'"]
  ['server2', 'linux', 'x86_64', "python '/root/server2.py'"]
  ['server3', 'linux', 'x86_64', "python '/root/server3.py'"]
  Traceback (most recent call last): 
     File "thisscript.py", line 24, in ? 
       p = subprocess.Popen('python myscript1.py',stdout=subprocess.PIPE,stderr=subprocess.PIPE) 
     File "/usr/lib64/python2.4/subprocess.py", line 550, in __init__ 
       errread, errwrite) 
     File "/usr/lib64/python2.4/subprocess.py", line 993, in _execute_child 
       raise child_exception 
     OSError: [Errno 2] No such file or directory

Bonus:
If I wanted to search the stdout/command output for something also (such as linux, swap, NTFS, etc, --third example command in question above) and append it to row[5], or next after it has already searched for [i]Healthy[/i]... Ive tried starting a new if statement but it appears to only append row[4], or the same row as when it does for [i]Healthy[/i].

I also cant figure out how to to use an OR statement. Where

 if 'Linux' OR 'swap' OR 'LVM' in stdout:  
     writer.writerow(row + ['Linux']) # for multiple lines/partitions.

 elif 'BSD' in stdout:  
     writer.writerow(row + ['BSD'])

 elif 'NTFS' in stdout:
     writer.writerow(row + ['Windows'])

 else:
     writer.writerow(row + ['Error Scanning'])

Last I have changed the COMMAND column to the PATH and modified the command to execute the PATH. Which is working. I'd like to execute a second command to fetch the filesize of PATH. Ive tried a couple methods.

Thank you for your time. I hope this can all be done.

Answer 1

You're not using subprocess.Popen correctly, which is leading to the immediate problem (OSError: [Errno 2] No such file or directory).

In general, the first argument to Popen should be a sequence, not a string unless you also pass the shell=True keyword parameter. If the first argument is a string and shell=False (the default), Popen will attempt to execute the file named for the value of the string. There is no file named "python '/root/my_script.py'" (the whole string), therefore you get an OSError.

So,

p = subprocess.Popen(
    "python '/root/my_script.py'", 
    stdout=subprocess.PIPE, stderr=subprocess.PIPE
)

should probably become something like ...

p = subprocess.Popen(
    ["python", "'/root/my_script.py'"], 
    stdout=subprocess.PIPE, stderr=subprocess.PIPE
)

or (essentially equivalent)

p = subprocess.Popen(
    "python '/root/my_script.py'".split(), 
     stdout=subprocess.PIPE, stderr=subprocess.PIPE
)

or (see warning)

p = subprocess.Popen(
    "python '/root/my_script.py'", shell=True,
    stdout=subprocess.PIPE, stderr=subprocess.PIPE
)

---

Update: The answer to your question is Yes. Python can help you accomplish all that you'd like to do. Here's a breakdown of you list.

SPOILER ALERT! Do not read beyond this line if you want to figure things out for yourself.

• read a csv FILE

What you've done is fine. Another way ...

with open('my_list.csv', 'rb') as fp:
    my_data_list = [row for row in csv.reader(fp)]

... which introduces some potentially new concepts, the with statement, and list comprehensions. But you don't really need an intermediate list to act upon, you can read and write in the same loop (see below)

• executes COMMAND on fourth (vertical) column of FILE
• the COMMAND prints
• loop until first empty row of FILE

I've assumed you want to print the output, or result, of running the command.

for row in my_data_list:
    command = row[3] #<- 4th column is index 3, 1st is 0
    p = Popen(command.split(), stdout=PIPE, stderr=STDOUT) #<- stderr to stdout
    stdout, empty = p.communicate()
    print stdout

• read each line for HEALTHY (from COMMAND)
  • if multiple lines of HEALTHY and all do not equal Yes, HEALTHY equals "No"
  • if HEALTHY is not found on any lines, HEALTHY equals "Error Scanning"

• write HEALTHY on new fifth column to NEW_FILE with all five columns

  if 'HEALTHY:No' in stdout:
      writer.writerow(row + ['No'])
  elif 'HEATHLY:Yes' in stdout:
      writer.writerow(row + ['Yes'])
  else: 
      writer.writerow(row + ['Error Scanning'])
  

And putting it all together (untested) ...

import csv
from subprocess import Popen, PIPE, STDOUT

with open('my_list.csv', 'rb') as incsv:
    with open('new_data.csv', 'wb') as outcsv:
        reader = csv.reader(incsv)
        writer = csv.writer(outcsv)

        for row in reader:
            p = Popen(row[3].split(), stdout=PIPE, stderr=STDOUT)
            stdout, empty = p.communicate()

            print 'Command: %s\nOutput: %s\n' % (row[3], stdout)

            if 'HEALTHY:No' in stdout:
                writer.writerow(row + ['No'])
            elif 'HEATHLY:Yes' in stdout:
                writer.writerow(row + ['Yes'])
            else: 
                writer.writerow(row + ['Error Scanning'])

Update: fixed poor naming choice of csv reader and writer file objects

Update: Python 2.5 introduced the from __future__ import with_statement directive. For versions of python older than 2.5, the with statement is unavailable. In this case, the common approach is to wrap file operations in a try finally. As in,

import csv
from subprocess import Popen, PIPE, STDOUT

incsv = open('my_list.csv', 'rb')
try:
    reader = csv.reader(incsv)
    outcsv = open('new_data.csv', 'wb')
    try:    
        writer = csv.writer(outcsv)

        for row in reader:
            p = Popen(row[3].split(), stdout=PIPE, stderr=STDOUT)
            stdout, empty = p.communicate()

            print 'Command: %s\nOutput: %s\n' % (row[3], stdout)

            if 'HEALTHY:No' in stdout:
                writer.writerow(row + ['No'])
            elif 'HEATHLY:Yes' in stdout:
                writer.writerow(row + ['Yes'])
            else: 
                writer.writerow(row + ['Error Scanning'])
    finally:
        outcsv.close()
finally:
    incsv.close()

HTH!

Answer 2

In your "bonus" section:

If you want to search for multiple things, the simplest and most straightforward way is to search for each separately and then connect with or:

if 'Linux' in stdout or 'swap' in stdout or 'LVM' in stdout:
    writer.writerow(row + ['Linux'])

If you find this inelegant or need to search for more things, you can use the any function and a generator expression:

if any(x in stdout for x in ('Linux', 'swap', 'LVM')):
    writer.writerow(row + ['Linux'])

Finally, if this is still too inelegant, or if stdout becomes much larger and you don't want to search it multiple times, you can use regular expressions via the re module.