Bing Congitive Web Search API with Python 3

Question

Hi Stackoverflow Community

I am attempting to access the new Bing Cognitive Search API through a Python 3 script. I was able to find threats on using Bing Search 2.0 (since depreciated) but wasn't able to identify an example for the new API with Python 3. I used the following code:

import urllib.parse
import urllib.request
import json
import base64


def bing_search(query):
    key = 'mysubscription_key'
    query = urllib.parse.quote(query)

    #Create credentials for authentication
    user_agent = 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; FDM; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 1.1.4322)'
    encoded = base64.b64encode(bytes(':%s' % key, 'utf-8'))
    credentials = encoded[:-1]   # the "-1" is to remove the trailing "\n" which encode adds
    print(credentials)
    auth = 'Basic %s' % credentials
    print(auth)
    url = 'https://api.cognitive.microsoft.com/bing/v5.0/search?q=' + query + '&mkt=en-us'
    print(url)

    #Create the API request
    urlrequest = urllib.request.urlopen(url)        # in Python3 urllib.request(...) becomes urllib.request.open(...)
    urlrequest.add_header('Authentication', auth)
    urlrequest.add_header('User Agent', user_agent)
    request_opener = urllib.request.build_opener()

    # Handle the response
    response = request_opener.open(urlrequest)
    results = json.load(response)

    result_list = results['webPages']['values']
    print(result_list)

bing_search('good news')

Unfortunately I am receiving the following 'Access denied' error. Would anyone be able to point me in the right direction?

C:\Users\Admin\AppData\Local\Programs\Python\Python35-32\python.exe C:/Users/Admin/PycharmProjects/momely/placementarchitect/bingtest.py
b'OmZhNTlmNTBlMmFmMjQyZjhhYmE5MTZlNmZkYThhMDM'
Basic b'OmZhNTlmNTBlMmFmMjQyZjhhYmE5MTZlNmZkYThhMDM'
https://api.cognitive.microsoft.com/bing/v5.0/search?q=good%20news&mkt=en-us
Traceback (most recent call last):
  File "C:/Users/Admin/PycharmProjects/momely/placementarchitect/bingtest.py", line 34, in <module>
    bing_search('good news')
  File "C:/Users/Admin/PycharmProjects/momely/placementarchitect/bingtest.py", line 22, in bing_search
    urlrequest = urllib.request.urlopen(url)        # in Python3 urllib.request(...) becomes urllib.request.open(...)
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python35-32\lib\urllib\request.py", line 162, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python35-32\lib\urllib\request.py", line 471, in open
    response = meth(req, response)
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python35-32\lib\urllib\request.py", line 581, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python35-32\lib\urllib\request.py", line 509, in error
    return self._call_chain(*args)
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python35-32\lib\urllib\request.py", line 443, in _call_chain
    result = func(*args)
  File "C:\Users\Admin\AppData\Local\Programs\Python\Python35-32\lib\urllib\request.py", line 589, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Access Denied

Thank you kindly M.

Answer 1

The authentication is not through basic auth but through custom header Ocp-Apim-Subscription-Key:

Subscription key which provides access to this API.

Which can be found in your subscriptions

See full details at https://www.microsoft.com/cognitive-services/en-us/bing-web-search-api

Assuming the key variable holds the subscription key, what you need to do is simply urlrequest.add_header('Ocp-Apim-Subscription-Key', key)

As a minor suggestion, your code can be greatly simplified by using python-requests. A minimum example can be:

import requests

def bing_search(query):
    url = 'https://api.cognitive.microsoft.com/bing/v5.0/search'
    # query string parameters
    payload = {'q': query}
    # custom headers
    headers = {'Ocp-Apim-Subscription-Key': 'xxxxxxxxxxxxx'}
    # make GET request
    r = requests.get(url, params=payload, headers=headers)
    # get JSON response
    return r.json()

j = bing_search('good news')
print(j.get('webPages', {}).get('value', {}))

Answer 2

You should also check out py-ms-cognitive package for Python. It's a wrapper for the Cognitive Services API, currently supporting web search. I'm also looking for feedback / collaborator, so feel free to leave an issue, make PR etc.

To install:

pip install py-ms-cognitive

To use:

from py_ms_cognitive import PyMsCognitiveWebSearch
web_bing = PyMsCognitiveWebSearch('api_key', "xbox")
first_fity_results = web_bing.search(limit=50)
second_fity_results = web_bing.search(limit=50)
print second_fifty_results[0].url