Why is Python's requests 10x faster than C's libcurl?

Question

Python's requests library seems to be ~10x faster than C's libcurl (the C API, the CLI application, and the Python API) for a 1.6 MB request (requests takes 800ms, while curl/libcurl sometimes takes as much as 7 seconds).

• Why is this?

• How can I get curl in C to run as fast as requests in Python?

libcurl seems to be getting its replies in 16KB chunks, while requests seems to get the whole thing at once, but I'm not sure this is it... I tried curl_easy_setopt(curl_get, CURLOPT_BUFFERSIZE, 1<<19) but it only seems good to get the buffer size to be smaller.

I've tried looking at the source code for requests, and I think it uses urllib3 as its HTTP "backend"... but using urllib3 directly results in the same (disappointing) results as using curl.

Here are some examples.

/*
gcc-8 test.c -o test -lcurl  &&  t ./test
*/
#include <curl/curl.h>

int main(){
  CURLcode curl_st;
  curl_global_init(CURL_GLOBAL_ALL);

  CURL* curl_get = curl_easy_init();
  curl_easy_setopt(curl_get, CURLOPT_URL,           "https://api.binance.com/api/v3/exchangeInfo");
  curl_easy_setopt(curl_get, CURLOPT_BUFFERSIZE, 1<<19);
  curl_st=curl_easy_perform(curl_get);  if(curl_st!=CURLE_OK) printf("\x1b[91mFAIL  \x1b[37m%s\x1b[0m\n", curl_easy_strerror(curl_st));

  curl_easy_cleanup(curl_get);
  curl_global_cleanup();
}

'''FAST'''
import requests
reply = requests.get('https://api.binance.com/api/v3/exchangeInfo')
print(reply.text)

'''SLOW'''
import urllib3
pool = urllib3.PoolManager()  # conn = pool.connection_from_url('https://api.binance.com/api/v3/exchangeInfo')
reply = pool.request('GET', 'https://api.binance.com/api/v3/exchangeInfo')
print(reply.data)
print(len(reply.data))

'''SLOW!'''
import urllib.request
with urllib.request.urlopen('https://api.binance.com/api/v3/exchangeInfo') as response:
  html = response.read()

'''SLOW!'''
import pycurl
from io import BytesIO
buf  = BytesIO()
curl = pycurl.Curl()
curl.setopt(curl.URL, 'https://api.binance.com/api/v3/exchangeInfo')
curl.setopt(curl.WRITEDATA, buf)
curl.perform()
curl.close()
body = buf.getvalue()  # Body is a byte string. We have to know the encoding in order to print it to a text file such as standard output.
print(body.decode('iso-8859-1'))

curl https://api.binance.com/api/v3/exchangeInfo

Answer 1

One way of speeding up the transfer of web content is to use HTTP compression. This works by compressing data on the fly before it is sent between the server and client so it takes less time to transmit.

Although HTTP compression is supported by libcurl, it is disabled by default:. From the CURLOPT_ACCEPT_ENCODING documentation:

Set CURLOPT_ACCEPT_ENCODING to NULL to explicitly disable it, which makes libcurl not send an Accept-Encoding: header and not decompress received contents automatically.

The default value of this parameter is NULL, so unless you specifically enable HTTP compression, you won't get it.