I am trying to use Python to interact with another program via the command line. The main problem I am having is a specific call that has multiple follow-up prompts. Initially the command line call asks for the name of a project and then proceeds to ask if I would like to view any of the subfolders of the project. I need to answer y/n to each of these in order and the answer to each is unfortunately not all y or n. Additionally, I cannot know the answer to the question without reading the individual prompts so I am incapable of sending a block of 'y's or 'n's all at once.
This is the command line call:
si viewproject
After entering the command, the command line prompts:
Enter the project name:
And an example response would be:
Enter the project name: c:/test.pj
After entering the project, it prompts the following:
Do you want to recurse into the subproject test_subprj.pj? [ynYN](n)
At which point I need to respond with either a y or n depending on if I need that subproject. Again, the response to this question is dependent on the subproject. I need to be able to read the subproject in this prompt in order to respond to it with a 'y' or 'n'
Currently I need to manually enter in the project and each of the y's and n's respectively. My goal is to automate this process using Python.
Is there a way to respond to these command line prompts automatically?
project_path = "c:/test.pj"
with Popen(["si", "viewproject", "--project=" + project_path],
stdin=PIPE, stdout=PIPE, universal_newlines=True) as p:
for line in p.stdout:
if line.startswith("Do you want"):
answer = 'n'
else:
continue # skip it
print(answer, file=p.stdin) # provide answer
p.stdin.flush()
This method is hanging after the with Popen statement. It never errors, but it never enters or exits the for statement and never completes. Currently I am defaulting all answers to "n", but that will be replaced with logic later.
import re
import sys
from functools import partial
import winpexpect
project_path = "c:/test.pj"
p = winpexpect.winspawn('si viewproject --project=' + project_path)
p.logfile = sys.stdout
patterns = [re.compile('ynYN'), winpexpect.EOF]
for found in iter(partial(p.expect, patterns), 1): # until EOF
if found == 0:
answer = 'n'
p.sendline(answer)
Returns the following error message:
Traceback (most recent call last):
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\winpexpect.py", line 541, in read_nonblocking
handle, status, data = self.child_output.get(timeout=timeout)
File "C:\Python33\lib\queue.py", line 175, in get
raise Empty
queue.Empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1378, in expect_loop
c = self.read_nonblocking (self.maxread, timeout)
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\winpexpect.py", line 543, in read_nonblocking
raise TIMEOUT('Timeout exceeded in read_nonblocking().')
pexpect.TIMEOUT: Timeout exceeded in read_nonblocking().
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "K:\eclipse_3.6.0\plugins\org.python.pydev_2.6.0.2012062818\pysrc\pydev_runfiles.py", line 432, in __get_module_from_str
mod = __import__(modname)
File "C:\workspace\Test_prj\Test_prj.py", line 19, in <module>
for found in iter(partial(p.expect, patterns), 1): # until EOF
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1311, in expect
return self.expect_list(compiled_pattern_list, timeout, searchwindowsize)
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1325, in expect_list
return self.expect_loop(searcher_re(pattern_list), timeout, searchwindowsize)
File "C:\Python33\lib\site-packages\winpexpect-1.5-py3.3.egg\pexpect.py", line 1409, in expect_loop
raise TIMEOUT (str(e) + '\n' + str(self))
pexpect.TIMEOUT: Timeout exceeded in read_nonblocking().
<winpexpect.winspawn object at 0x0144AE50>
version: 2.3 ($Revision: 399 $)
command: si
args: ['si', 'viewproject', '--project=c:/test.pj']
searcher: searcher_re:
0: re.compile("ynYN")
1: EOF
buffer (last 100 chars):
before (last 100 chars):
after: <class 'pexpect.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 6448
child_fd: 4
closed: False
timeout: 30
delimiter: <class 'pexpect.EOF'>
logfile: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='Cp1252'>
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
ERROR: Module: Test_prj could not be imported (file: C:\workspace\Test_prj\Test_prj.py).
Lazy Persons Way
Install Distribute
Do This
Install PyWin32
Install Winpexpect
Optional: Install Nose
Optional: Install Pip
First-World Problems
Python is a new language for me, and I had never installed a package before for Python. Additionally, Python 3.x is a little different than the other versions of Python making installing modules a little bit more of an adventure.
So, to help others get some sweet sweet module action (and to help those who are more knowledgeable see if I did anything wrong) here's a soon to be success story (hopefully) documenting how I got and installed my first module.
Setup
Python allows third-party groups to develop and distribute modules that extend the abilities of the programming language. Naturally, there is a standard way to help third-party developers make modules as easily available to the end-user as possible.
For Python 3.x, that standard for distributing modules is called Distutils.
Here is how a developer uses Distutils: Distributing Python Modules
And here is how the end-user uses Distutils: Installing Python Modules
Normally, navigating to the folder of your downloaded module in the command line and running "setup.py install" will be enough.
BUT
Sometimes life isn't so easy and you may still have problems with your installation. You may, in fact, need something else. For example, you may get the following error:
"ImportError “No Module named Setuptools”"
Luckily, there is a solution for that: Python 3: ImportError "No Module named Setuptools"
As it turns out, not everything uses distutils. Some packages use setuptools. Unfortunately, there is no setuptools for Python 3.x. Rather, Python 3.x uses distribute which is a branch of setuptools.
So for those who use Python 3.x, here is Distribute: Distribute
For those using Python 2.x, here is Setuptools: Setuptools
In the Installation Instructions for Distribute, it says the following:
"Download
distribute_setup.py <http://python-distribute.org/distribute_setup.py>
_
and execute it, using the Python interpreter of your choice."
It also says: "Notice this file is also provided in the source release."
So I downloaded Distribute and saved it to the computer. Once it was saved to the computer, I ran distribute_setup.py from the source release and got the following error:
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.36.tar.gz
Traceback (most recent call last):
File "C:\Python33\lib\urllib\request.py", line 1252, in do_open
h.request(req.get_method(), req.selector, req.data, headers) File "C:\Python33\lib\http\client.py", line 1049, in request
self._send_request(method, url, body, headers)
File "C:\Python33\lib\http\client.py", line 1087, in _send_request
self.endheaders(body)
File "C:\Python33\lib\http\client.py", line 1045, in endheaders
self._send_output(message_body)
File "C:\Python33\lib\http\client.py", line 890, in _send_output
self.send(msg)
File "C:\Python33\lib\http\client.py", line 828, in send
self.connect()
File "C:\Python33\lib\http\client.py", line 806, in connect
self.timeout, self.source_address)
File "C:\Python33\lib\socket.py", line 406, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno 11001] getaddrinfo failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\workspace\PythonTest\distribute_setup.py", line 553, in <module>
sys.exit(main())
File "C:\workspace\PythonTest\distribute_setup.py", line 549, in main
tarball = download_setuptools(download_base=options.download_base)
File "C:\workspace\PythonTest\distribute_setup.py", line 204, in download_setuptools
src = urlopen(url)
File "C:\Python33\lib\urllib\request.py", line 160, in urlopen
return opener.open(url, data, timeout)
File "C:\Python33\lib\urllib\request.py", line 473, in open
response = self._open(req, data)
File "C:\Python33\lib\urllib\request.py", line 491, in _open
'_open', req)
File "C:\Python33\lib\urllib\request.py", line 451, in _call_chain
result = func(*args)
File "C:\Python33\lib\urllib\request.py", line 1272, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "C:\Python33\lib\urllib\request.py", line 1255, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 11001] getaddrinfo failed>
Well that is no good! I honestly still do not know where that error is coming from or why it happened.
Regardless, then I found the following site that ran a .exe to install distribute as well as pip.
Install Distribute
Install Pip
So I got those installed and then used the following site to setup my computer to more easily use easy_install: Setting Up Easy Install Made Easy
Once I got this working I then installed nose:Nose
The reason I got nose was because the Winpexpect website says: "WinPexpect includes unit tests. To run the tests, you need nose. Use the following command to run the tests:
$ python setup.py test"
Well that sounds nice :). Now I just wished I knew where to run that test. I know that if you install manually you use the setup.py install command so there will most definitely be a setup.py in the zipped directory online. To see if this was correct, I downloaded and saved the winpexpect file, extracted the information, navigated to it via command-line, and ran setup.py test.
Here was the following result:
running test
running build_py
running egg_info
creating c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info
writing c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\PKG-INFO
writing dependency_links to c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\dependency_links.txt
writing top-level names to c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\top_level.txt
writing requirements to c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\requires.txt
writing manifest file 'c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\SOURCES.txt'
reading manifest file 'c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\SOURCES.txt'
writing manifest file 'c:\documents and settings\slz1fh\desktop\winpexpect\geertj-winpexpect-76df3cfcb143\build\lib\winpexpect.egg-info\SOURCES.txt'
running build_ext
Traceback (most recent call last):
File "C:\Documents and Settings\SLZ1FH\Desktop\winpexpect\geertj-winpexpect-76df3cfcb143\setup.py", line 35, in <module>
use_2to3 = True
File "C:\Python33\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "C:\Python33\lib\distutils\dist.py", line 917, in run_commands
self.run_command(cmd)
File "C:\Python33\lib\distutils\dist.py", line 936, in run_command
cmd_obj.run()
File "C:\Python33\lib\site-packages\distribute-0.6.36-py3.3.egg\setuptools\command\test.py", line 138, in run
self.with_project_on_sys_path(self.run_tests)
File "C:\Python33\lib\site-packages\distribute-0.6.36-py3.3.egg\setuptools\command\test.py", line 118, in with_project_on_sys_path
func()
File "C:\Python33\lib\site-packages\distribute-0.6.36-py3.3.egg\setuptools\command\test.py", line 164, in run_tests
testLoader = cks
File "C:\Python33\lib\unittest\main.py", line 124, in __init__
self.parseArgs(argv)
File "C:\Python33\lib\unittest\main.py", line 168, in parseArgs
self.createTests()
File "C:\Python33\lib\unittest\main.py", line 175, in createTests
self.module)
File "C:\Python33\lib\unittest\loader.py", line 137, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "C:\Python33\lib\unittest\loader.py", line 137, in <listcomp>
suites = [self.loadTestsFromName(name, module) for name in names]
File "C:\Python33\lib\unittest\loader.py", line 96, in loadTestsFromName
module = __import__('.'.join(parts_copy))
File "C:\Python33\lib\site-packages\nose-1.3.0-py3.3.egg\nose\__init__.py", line 1, in <module>
from nose.core import collector, main, run, run_exit, runmodule
File "C:\Python33\lib\site-packages\nose-1.3.0-py3.3.egg\nose\core.py", line 143
print "%s version %s" % (os.path.basename(sys.argv[0]), __version__)
^
SyntaxError: invalid syntax
Ok, so the Python 3.3 version of Nose contains invalid syntax for Python 3.3?
print "%s version %s" % (os.path.basename(sys.argv[0]), version)...
should definitely have parenthesis around it... This makes me question if nose will actually work here as it clearly looks to be made for earlier versions of Python.
In the comments you mentioned that xx viewproject < answers.txt > output.txt
works but you can't use it because answers depend on the output from the subprocess.
In general pexpect
-like modules such as winpexpect
(for Windows) could be used. Something like:
import re
import sys
from functools import partial
from winpexpect import EOF, winspawn as spawn
p = spawn('xx viewproject')
p.logfile = sys.stdout
patterns = ['the project:', re.escape('? [ynYN](n)'), EOF]
for found in iter(partial(p.expect, patterns), 2): # until EOF
if found == 0:
p.sendline(project_name)
elif found == 1:
filename = get_filename_from_prompt(p.before) # a regex could be used
answer = yes_or_no_from_subproject.get(filename, 'no') # a dict
p.sendline(answer)
If the prompts are terminated with a newline (and the subprocess doesn't buffer them); you could read line by line using subprocess
module directly:
from subprocess import Popen, PIPE
with Popen(["xx", "viewproject"], stdin=PIPE, stdout=PIPE,
universal_newlines=True) as p:
for line in p.stdout:
if line.startswith("Please enter the name of the project"):
answer = project_name
elif line.startswith("Would you like to recurse into the subproject"):
filename = get_filename_from_prompt(line) # a regex could be used
answer = yes_or_no_from_subproject.get(filename, 'n') # a dict
else:
continue # skip it
print(answer, file=p.stdin) # provide answer
p.stdin.flush()
To test that you can read something from the xx
using subprocess
:
from subprocess import Popen, PIPE, STDOUT
with Popen(["xx", "viewproject"], bufsize=0,
stdin=PIPE, stdout=PIPE, stderr=STDOUT) as p:
print(repr(p.stdout.read(1)))
Yes, first of all you may create subprocess as an object by:
p = subprocess.Popen('xx viewproject', shell=True, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, universal_newlines=True)
Then you'll have methods like communicate()
available, for instance:
newline = os.linesep # [1]
commands = ['y', 'n', 'y', 'n', 'y']
p.communicate( newline.join( commands))
1 - os.linesep
Which will send all the answers at once (and hopefully it'll be enough) relying on the same order of question every time.
You may also try parsing p.stdout
and then writing to p.stdin
, but this may cause deadlock when one buffer will get full while waiting for another, so be careful with this. Luckily there are some complex examples on google.
Simple version would be:
p = Popen(...)
line = p.stdout.readline() # At this point, if child process will wait for stdin
# you have a deadlock on your hands
parse_line( line)
p.stdin.write( newline.join( commands).encode( 'utf-8'))
I would also consider rewriting:
p = subprocess.Popen('si viewproject --project=d:/Projects/test.pj', shell=True,
stdin=subprocess.PIPE, stdout=subprocess.PIPE)
To:
p = subprocess.Popen( ['si', 'viewproject', '--project=d:/Projects/test.pj'],
shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
Unless you explicitly need Shell invocation.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With