Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to get data from all pages in Github API with Python?

I'm trying to export a repo list and it always returns me information about the 1rst page. I could extend the number of items per page using URL+"?per_page=100" but it's not enough to get the whole list. I need to know how can I get the list extracting data from page 1, 2,...,N. I'm using Requests module, like this:

while i <= 2:
      r = requests.get('https://api.github.com/orgs/xxxxxxx/repos?page{0}&per_page=100'.format(i), auth=('My_user', 'My_passwd'))
      repo = r.json()
      j = 0
      while j < len(repo):
            print repo[j][u'full_name']
            j = j+1
      i = i + 1

I use that while condition 'cause I know there are 2 pages, and I try to increase it in that waym but It doesn't work

like image 742
skiel95 Avatar asked Nov 23 '15 18:11

skiel95


1 Answers

import requests

url = "https://api.github.com/XXXX?simple=yes&per_page=100&page=1"
res=requests.get(url,headers={"Authorization": git_token})
repos=res.json()
while 'next' in res.links.keys():
  res=requests.get(res.links['next']['url'],headers={"Authorization": git_token})
  repos.extend(res.json())

If you aren't making a full blown app use a "Personal Access Token"

https://github.com/settings/tokens

like image 52
gotit Avatar answered Sep 22 '22 15:09

gotit