nginx permission denied while reading upstream - even when run as root

Question

I have a flask app running under uWSGI behind nginx.

*1 readv() failed (13: Permission denied) while reading upstream, client: 10.0.3.1, server: , request: "GET /some/path/constants.js HTTP/1.1", upstream: "uwsgi://unix:/var/uwsgi.sock:", host: "dev.myhost.com"

The permissions on the socket are okay (666, and set to the same user as nginx), in fact, even when I run nginx as root I still get this error.

The flask app/uwsgi is sending the request properly. But it's just not being read by Nginx. This is on Ubuntu Utopic Unicorn.

Any idea where the permission might be getting denied if the nginx process has full access to the socket?

As a complicating factor this server is running in a container that has Ubuntu 14.04 installed in it. And this setup used to work... but I recently upgraded the host to 14.10... I can fully understand that this could be the cause of the problem. But before I downgrade the host or upgrade the container I want to understand why.

When I run strace on a worker that's generating this error I see the call it's making is something like this:

readv(14, 0x7fffb3d16a80, 1)            = -1 EACCES (Permission denied)

14 seems to be the file descriptor created by this system call

socket(PF_LOCAL, SOCK_STREAM, 0)        = 14

So it can't read from a local socket that it has just created?

Answer 1

Okay! So the problem was, I think, related to this bug. It seems that even though apparmor wasn't configured to prevent access to sockets inside the containers it was actually doing something to prevent reading from them (though not creation...) so turning off apparmor for the container (following these instructions) worked to fix it.

The two relevant lines were:

sudo apparmor_parser -R /etc/apparmor.d/usr.bin.lxc-start

sudo ln -s /etc/apparmor.d/usr.bin.lxc-start /etc/apparmor.d/disabled/

and adding

lxc.aa_profile = unconfined 

To the containers config file.

NB: These errors were not recorded in any apparmor logs.

Answer 2

This problem was probably introduced in kernel 3.16, because it does not reproduce on 14.04 with 3.13 kernel. Strange apparmor bug was indeed responsible for that.

Unfortunately @aychedee's solution didn't work for me. In my case I had to add the following parameter to docker run command to get rid of the issue:

docker run --security-opt apparmor:unconfined ...

If someone's aware what is the current state of the issue, please consider adding comment under this answer :)