Google search issue in Python

Question

I have implemented a program in python which performs the Google search and captures top ten links from the search results. I am using 'pygoogle' library for search, when I am implementing my program for the first two or three times, it is getting proper hits and the entire project is working very fine. But afterward, after certain links got downloaded, it's giving an error as follows. (gui_two.py is my program name)

Exception in Tkinter callback

Traceback (most recent call last):

  File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 1413, in __call__
    return self.func(*args)
  File "gui_two.py", line 113, in action
    result = uc.utilcorpus(self.fn1,"")
  File "/home/ci/Desktop/work/corpus/corpus.py", line 125, in utilcorpus
    for url in g1.get_urls():   #this is key sentence based search loop
  File "/home/ci/Desktop/work/corpus/pygoogle.py", line 132, in get_urls
    for result in  data['responseData']['results']:
  TypeError: 'NoneType' object has no attribute '__getitem__'

I know this is most familiar error in python, but I am not able to do anything since it is a library. I wonder my program is spamming the Google or I need custom Google search API's or may be the other reason. Please give me precise information for performing search without any issue. I will be so grateful for your help.

Thanks.

Edited: Actually my code is very huge, here is a small piece of code, where problem arises.

  g1 = pygoogle(query)  
  g1.pages = 1
  for url in g1.get_urls(): #error is in this line
      print "URL : ",url

It may work if we simply copy it in a simple .py file, but if we execute it many times, program gives an error.

Answer 1

Here's the culprit code from pygoogle.py (from http://pygoogle.googlecode.com/svn/trunk/pygoogle.py)

def get_urls(self):
    """Returns list of result URLs"""
    results = []
    search_results = self.__search__()
    if not search_results:
        self.logger.info('No results returned')
        return results
    for data in search_results:
        if data and data.has_key('responseData') and data['responseData']['results']:
            for result in  data['responseData']['results']:
                if result:
                    results.append(urllib.unquote(result['unescapedUrl']))
    return results

Unlike every other place where data['responseData']['results'] is used, they're not both being checked for existence using has_key().

I suspect that your responseData is missing results, hence the for loop fails.

Since you have the source, you can edit this yourself.

Also make an issue for the project - very similar to this one in fact.