IPython uses a different $PATH environment than the shell

Question

I had some problems with calling pip within the IPython REPL, and after a while I noticed that IPython doesn't use the same $PATH environment as my shell.

$ echo $PATH
/Users/jimmy/dev/anaconda/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/texbin

$ ipython
In [1]: !echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/texbin:/Users/jimmy/dev/anaconda/bin

It does seem to scramble the path order, and I'm not quite sure what's wrong.

I'm using ZSH installed through oh-my-zsh as shell, if it is of help.

This is what sys.path() returns:

['',
 '/Users/jimmy/dev/anaconda/bin',
 '/Users/jimmy/dev/anaconda/lib/python2.7/site-packages/sparsesvd-0.2.2-py2.7-macosx-10.5-x86_64.egg',
 '/Users/jimmy/dev/projects/pyresult',
 '/Users/jimmy/dev/work/gavagai/userdata',
 '/Users/jimmy/dev/work/gavagai/gavapi',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python27.zip',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python2.7',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python2.7/plat-darwin',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python2.7/plat-mac',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python2.7/plat-mac/lib-scriptpackages',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python2.7/lib-tk',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python2.7/lib-old',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python2.7/lib-dynload',
 '/Users/jimmy/dev/anaconda/lib/python2.7/site-packages/runipy-0.1.0-py2.7.egg',
 '/Users/jimmy/dev/anaconda/lib/python2.7/site-packages/setuptools-3.6-py2.7.egg',
 '/Users/jimmy/dev/anaconda/lib/python2.7/site-packages',
 '/Users/jimmy/dev/anaconda/lib/python2.7/site-packages/PIL',
 '/Users/jimmy/dev/anaconda/lib/python2.7/site-packages/IPython/extensions',
 '/Users/jimmy/.ipython']

And this is what os.environ['PATH'] returns:

'/Users/jimmy/dev/anaconda/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/texbin'

Both seem to place it in the correct order.

Grateful for all help.

Answer 1

To summarize and complement @holdenweb's helpful answer, particularly with respect to OS X:

• A subshell started from IPython with ! is a non-interactive non-login instance of the user's default shell - even if IPython was started from a different shell.
  • Behind the scenes, the subshell is started with path/to/default/shell -c ...
  • To see specifics, run !ps -p $$ && :

• echo $SHELL always tells you the default shell - even when run from a different shell.

• Initialization files sourced in non-interactive non-login shells:

  • zsh:: /etc/zshenv and ~/.zshenv
  • bash: a script pointed to in the $BASH_ENV variable, if defined.

• As pointed out, shells load different/additional initialization files depending on whether:

  • the shell is a login shell or not
  • an interactive shell or not

  Note that a login shell can be interactive or not, and an interactive shell can be a login shell or not.

Thus, in the case at hand, potentially two additional initialization files were loaded in the interactive shell, explaining the difference in behavior between the interactive shell and the subshell created by IPython:

• ~/.zprofile - if the shell is a login shell - which will be the case if zsh is the default shell (on OS X, all interactive instances of the default shell created in a terminal such as Terminal.app are login shells).
• ~/.zshrc

---

Finally, on a related note, note that on OS X the default $PATH for NON-shell processes is:

/usr/bin:/bin:/usr/sbin:/sbin  # Note the absence of /usr/local/bin.

and only shells add /usr/local/bin (by default; extensible) to that, via system-wide initialization files (that call /usr/libexec/path_helper):

• zsh:
  • /etc/zshenv
  • Note: takes effect for ALL zsh instances.
• bash (also when invoked as sh), ksh:
  • /etc/profile
  • Note: takes effect for LOGIN shells only.

Upshot:

• Non-shell apps see only the default $PATH (notably, without /usr/local/bin and other additions made via shell initialization files).
• bash/sh and ksh non-interactive non-login shells not launched from a login shell also see only the default $PATH. This happens, for instance, when GUI command-line launchers such as Alfred create shell instances.
• zsh is not affected, because /etc/zshenv is read by every zsh instance.