how to remove the "User-Agent" header when send request in python

Question

I'm using python requests library, I need send a request without a user-agent header. I found this question, but it's for Urllib2.

I'm simulating an Android app which does this when calling a private API.

I try to set User-Agent to None as in the following code, but it doesn't work. It still sends User-Agent: python-requests/2.27.1.

headers = requests.utils.default_headers()
headers['User-Agent'] = None
requests.post(url, *args, headers=headers, **kwargs)

Answer 1

The requests library is built on top of the urllib3 library. So, when you pass None User-Agent header to the requests's post method, the urllib3 set their own default User-Agent

import requests

r = requests.post("https://httpbin.org/post", headers={
    "User-Agent": None,
})

print(r.json()["headers"]["User-Agent"])

Output

python-urllib3/1.26.7

Here the urllib3 source of connection.py

class HTTPConnection(_HTTPConnection, object):
    ...

    def request(self, method, url, body=None, headers=None):
        if headers is None:
            headers = {}
        else:
            # Avoid modifying the headers passed into .request()
            headers = headers.copy()
        if "user-agent" not in (six.ensure_str(k.lower()) for k in headers):
            headers["User-Agent"] = _get_default_user_agent()
        super(HTTPConnection, self).request(method, url, body=body, headers=headers) 

So, you can monkey patch it to disable default User-Agent header

import requests
from urllib3 import connection


def request(self, method, url, body=None, headers=None):
    if headers is None:
        headers = {}
    else:
        # Avoid modifying the headers passed into .request()
        headers = headers.copy()
    super(connection.HTTPConnection, self).request(method, url, body=body, headers=headers)

connection.HTTPConnection.request = request


r = requests.post("https://httpbin.org/post", headers={
    "User-Agent": None,
})

print(r.json()["headers"])

Output

{
'Accept': '*/*', 
'Accept-Encoding': 'gzip, deflate', 
'Content-Length': '0', 
'Host': 'httpbin.org', 
'X-Amzn-Trace-Id': 'Root=1-61f7b53b-26c4c8f6498c86a24ff05940'
}

Also, consider to provide browser-like User-Agent like this Mozilla/5.0 (Macintosh; Intel Mac OS X 12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36. Maybe it solves your task with less effort

Answer 2

For urllib3 >= 1.26.0 (aka, requests >= 2.25.0), you can make use of urllib3.util.SKIP_HEADER to suppress User-Agent, Accept-Encoding and Host header. See https://github.com/psf/requests/issues/5671#issuecomment-1006735307 for an example and https://github.com/urllib3/urllib3/pull/2018 for upstream changes.

PS: You may want to use urllib3 >= 1.26.3 to avoid a minor issue spamming connection.py: BytesWarning: Comparison between bytes and string. See https://github.com/urllib3/urllib3/issues/2071.