Find bottleneck of flask application

Question

I wrote a flask application. I found it very slow when I deployed it in a remote server. So, I did some profiling practices with it. Please take a look at the pictures below:

The code I use to profiling is:

#coding: utf-8
from werkzeug.contrib.profiler import ProfilerMiddleware
from app import app

app.config['PROFILE'] = True
app.wsgi_app = ProfilerMiddleware(app.wsgi_app, restrictions = [30])
app.run(debug = True)

Picture 1

profiling in the remote server. Maybe the bottleneck is _socket.getaddrinfo

Picture 2

profiling in the local machine. Nothing found bottleneck.

Picture 3

Sometimes， even in the remote server, there are no bottleneck found. No _socket.getaddrinfo found. Weird!

I did profiling in remote server python shell, too, with cProfile. Take a look at this:

In [10]: cProfile.run("socket.getaddrinfo('easylib.gdufslib.org', 80, 0, 0, socket.SOL_TCP)")
         3 function calls in 8.014 CPU seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    8.014    8.014 :1()
        1    8.014    8.014    8.014    8.014 {_socket.getaddrinfo}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}



In [11]: cProfile.run("socket.getaddrinfo('easylib.gdufslib.org', 80, 0, 0, socket.SOL_TCP)")
         3 function calls in 8.009 CPU seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    8.009    8.009 :1()
        1    8.009    8.009    8.009    8.009 {_socket.getaddrinfo}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Maybe there is a fact that it takes much time to do some dns resolve job, and I can't change this myself.

Can any one tell me: why _socket.getaddrinfo is called and why sometimes not called? How to prevent the _socket.getaddrinfo being called? Because it slow down my website which let me down saddly.

Answer 1

I just ran into this myself on a Flask app running on a dedicated box from Digital Ocean, so I'll post the solution in case someone else hits this in the future.

I noticed a few days ago that API requests to GitHub were insanely slow, sometimes taking between 10 and 20 seconds. But running my app locally didn't have any issues. I profiled my app, and socket.getaddrinfo was indeed the culprit:

1 15058.431 15058.4310 15058.431 15058.4310 {_socket.getaddrinfo}
1 26.545 26.5450 26.545 26.5450 {_ssl.sslwrap}
1 23.246 23.2460 23.246 23.2460 {built-in method do_handshake}
4 22.387 5.5968 22.387 5.5968 {built-in method read}
1 7.632 7.6320 7.632 7.6320 {method 'connect' of '_socket.socket' objects}
103 4.995 0.0485 7.131 0.0692 <s/werkzeug/urls.py:374(url_quote)>
2 2.459 1.2295 2.578 1.2890 <ssl.py:294(close)>
36 1.495 0.0415 10.548 0.2930 <s/werkzeug/routing.py:707(build)>
859 1.442 0.0017 1.693 0.0020 {isinstance}
.... etc.

Working with Digital Ocean support, and suspecting it was somehow a DNS issue, the working solution was to change (in /etc/resolv.conf)

nameserver 4.2.2.2
nameserver 8.8.8.8

to

nameserver 8.8.4.4
nameserver 8.8.8.8

For whatever reason, 4.2.2.2 (run by Level3) decided it hated me but for the time being me and Google's DNS are cool.

Update: My colleague Karl suggested I go ahead and set up a local DNS caching server with bind to prevent Google's DNS from hating me as well. This link was super helpful.

Answer 2

I reckon this is a caused by either your remote host is not caching it's DNS lookups or getaddrinfo is slow because of an issue with IPv6.

Try this (a few times) to test if your nameservers seem to be caching:

$ time host easylib.gdufslib.org

and to test whether the lookups is faster when forcing IPv4-only:

import socket
socket.getaddrinfo("easylib.gdufslib.org", 80, socket.AF_INET, 0, socket.SOL_TCP)

If it's caused by the first one you can install a local caching nameserver or just fix so that the existing nameserver caches. If the latter is the cause you can either try to fix it in your code and libs, or disable IPv6 on your host if you don't use it.

Answer 3

I had the same issue in the near past. To see what causes the slowness I decided to see which ones of my endpoints are more being requested and which ones are the bottleneck. In this purpose I have created a tool to analyze requests. it may help you too, have a look at it https://github.com/muatik/flask-profiler