Why i can't do some things without sudo using Python and pip?

Question

When I use pip it usually doesn't work without sudo. I often see people use pip without sudo, so what am I doing wrong?

I read that it is not recommended to install pip packages with sudo. I know that with virtualenv I can use pip without sudo, but to install virtualenv I have to use sudo first.

When I try to install pip without sudo, I get:

PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.4/dist-packages/pip'

When trying to install flask with pip3 install flask:

PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.4/dist-packages/werkzeug'

Answer 1

The reason is that your regular user doesn't have the permissions needed to modify the system directories. Just like in this message:

PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.4/dist-packages/pip'

Here is a brief rundown of what you need to know:

Your system has python installed, in order to modify the system's python you have to use sudo or be the root user.

You can install python libraries in your home directory without using sudo, but only you (not other users of the system) will be able to use it. Do this with pip install --user package-name like gongzhitaao mentioned.

You can also create unique installations of python in a directory of your choosing, like The Laughing Man mentioned. This is called virtualenv, and I think this is the most-preferred way to work.

Answer 2

sudo is used in Unix/Linux systems to perform tasks as another user, using their permissions, such as the ability to write to certain directories. When you don't specify the user to emulate, such as when running

sudo pip install flask

you are attempting to run the command as the system administrator, known as root in many environments. You'll be asked for the administrator password (which can be your own, if you've given your user admin privileges), then the specified command runs as that user, meaning that it has read/write access to essentially every file and directory on the system (there are some exceptions, but they're mostly corner cases and not very important here). This means that you need to be extra careful when using sudo, as an error as small as a single space can really mess things up: compare

sudo rm -rf /usr/local/lib/python3.4/dist-packages/numpy*

with

sudo rm -rf /usr /local/lib/python3.4/dist-packages/numpy*

See that space between /usr and local/? You just started deleting the entire /usr folder, which contains a large portion of the vital files and programs on the system. Hope you have a backup! Now, this doesn't mean you need to be scared to death of sudo, but you do need to have a healthy respect for it.

Python installations tend to be system-level (yes, I know there are exceptions), which means you need to use sudo to modify them, such as when installing 3rd-party modules with pip. If you run

ls -l /usr/local/lib/python3.4 

you'll see something along the lines of

drwxrwsr-x 125 root 4096 Nov  3 00:40 dist-packages

showing that the directory you're trying to install to with pip is owned by root, so the use of sudo is necessary.

Now, there are a couple of ways around this. If you're comfortable with it, and don't mind modifying the system's global packages, go ahead and use sudo with pip (in fact, you may need to use sudo -H ... if you get a little message in yellow at the beginning about permissions in your home directory). All of your modules will be installed to /usr/local/lib/python3.4/dist-packages and be available to all users on the system.

The second option is to use pip's --user option, which will create a lib/python3.4/site-packages hierarchy in your home directory (~) and store any installed modules there, along with scripts in ~/bin (which you should add to your $PATH. The advantage of this method is that you don't need to use sudo, so you won't accidentally overwrite system-dependent modules where specific versions are required for other programs to run. The disadvantage is that the installed modules are only available to you, so you may run into issues if, for example, your web server is trying to run Flask as itself, and can't read the source files. However, that's nothing that a little config file-editing can't fix. This is my recommended solution for most users.

The third option is to use virtual environments like virtualenv. This will create a custom Python installation in the location of your choosing, with a separate python executable and site-packages hierarchy (there are options as to whether or not you want to link to or use the system's dist-packages repository). You can pip install packages directly into the virtualenv, and make as many environments as your little heart desires, each with slightly different versions of various dependencies, for example, so you can more robustly test your programs. You can turn on and off the virtual environments, so for example you could have a couple running in different tabs of Terminal, for example, testing things in parallel. This is a close second-place recommendation of mine, because there's (slightly) more work involved in activating and using the environments, and you may get confused about which one you're working in if you're not very good about naming them. Downsides include the lack of system-wide availability, like the second option, and the fact that the virtual environment needs to be manually activated before use.

So, take a look at the options, and see which is best for your system and your particular situation. Good luck!