"No module named _scproxy" on OSX

Question

I'm on OSX 10.6 with pre-installed python 2.6 and would like to install python packages via easy_install or setup.py (in a downloaded package). In my case I'm trying to install MySQLdb. In both cases I get a stack trace which ends like so:

...
File  "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/setuptools/command/easy_install.py", line 21, in <module>
    from setuptools.package_index import PackageIndex, parse_bdist_wininst
File  "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/setuptools/package_index.py", line 2, in <module>
   import sys, os.path, re, urlparse, urllib2, shutil, random, socket, cStringIO
File  "/System/Library/Frameworks/Python/framework/Versions/2.6/lib/python2.6/urllib2.py", line 111, in <module>
   from urllib import (unwrap, unquote, splittype, splithost, quote,
File  "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib.py", line 1335, in <module>
   from _scproxy import _get_proxy_settings, _get_proxies
ImportError: No module named _scproxy

The python installation is the unmodified pre-installed version 2.6.1 except that I added the source files to the lib folder. A "find /System/Library/Frameworks/Python.framework/ -name scproxy" does not yield any results.

How can I install the missing module?

Answer 1

Background

_scproxy is a Mac-specific urllib helper interfacing with OS-specific libraries to do HTTP requests. It seems to be missing on my system too (10.6.7). Preliminary, I think it looks like a problem with the system Python build (I can't find anything that looks like it under /System/Libraries).

Hack-o-rama solution

It's (sort of) possible to install the missing module. But first a bit of advice:

You Shouldn't^TM mess too much with your system Python installation. Do yourself a favour by learning to use virtualenv, and apply potentially dangerous operations on new, fresh virtualenv's. That way you're system wont be affected by installation of problematic packages.

Anyway: stock Python on Snow Leopard is 2.6.1. I did my experiments with the most recent 2.6, 2.6.6, a marginably safer way would be to download that instead. My experience is however that varying dot releases work just fine together.

Anyway, I downloaded 2.6.6 in my ~/src directory like this:

~/src/ext/python$ wget http://www.python.org/ftp/python/2.6.6/Python-2.6.6.tgz
~/src/ext/python$ tar zxf Python-2.6.6.tgz
~/src/ext/python$ cd Python-2.6.6
~/src/ext/python$ ./configure

console spams like crazy

~/src/ext/python$ make sharedmods

hopefully no errors, more console spamming

Locate the newly built _scproxy.so:

~/src/ext/python/Python-2.6.6$ find . -name '_scproxy.so'
./build/lib.macosx-10.4-x86_64-2.6/_scproxy.so # <- exact path may vary

You can now copy your _scproxy.so to /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynlo ad (and then remember that there's a home-built somewhat alien module in you system Python). Or, much better, add it to the lib/python2.6/ subdirectory of a virtualenv. After doing these things, I could import _scproxy in the manner indicated in your traceback:

Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from _scproxy import _get_proxy_settings, _get_proxies
>>>

This is a strong indication that installing packages via a method requiring urllib requests utlizing _scproxy will work. From there on you have continue on your own, though, since I don't want to test-install MySQL itself.