GitHub v3 API: Get full commit list for large comparison

Question

I'm trying to use the GitHub v3 API to get the full list of commits between two SHAs, using the comparison API (/repos/:owner/:repo/compare/:base...:head), but it only returns the first 250 commits and I need to get all of them.

I found the API pagination docs, but the compare API doesn't appear to support either the page or per_page parameters, either with counts or SHAs (EDIT: the last_sha parameter doesn't work either). And unlike the commits API, the compare API doesn't seem to return a Link HTTP header.

Is there any way to either increase the commit count limit on the compare API or to fetch a second page of commits?

Answer 1

Try using the parameter sha, for example:

https://api.github.com/repos/junit-team/junit/commits?sha=XXX, where the XXX is the SHA of the last returned commit in the current round of the query. Then iterate this process until you reach the ending SHA.

Sample python code:

startSHA = ''
endSHA = ''
while True:
    url = 'https://api.github.com/repos/junit-team/junit/commits?sha=' + startSHA
    r = requests.get(url)
    data = json.loads(r.text)
    for i in range(len(data)):
        commit = data[i]['sha']
        if commit == endSHA:
            #reach the ending SHA, stop here
        startSHA = commit

Answer 2

It's relatively easy. Here is an example:

import requests
next_url = 'https://api.github.com/repos/pydanny/django-admin2/commits'
while next_url:
    response = requests.get(next_url)
    # DO something with response
    # ...
    # ...
    if 'next' in response.links:
        next_url = response.links['next']['url']
    else:
        next_url = ''

UPDATE:

takie in mind that next urls are different than initial ex: Initial url:

https://api.github.com/repos/pydanny/django-admin2/commits

next url:

https://api.github.com/repositories/10054295/commits?top=develop&last_sha=eb204104bd40d2eaaf983a5a556e38dc9134f74e

So it's totally new url structure.

Answer 3

I tried solving this again. My notes:

• Compare (or pull request commits) list only shows 250 entries. For the pull request one, you can paginate, but you will only get a maximum of 250 commits, no matter what you do.

• Commit list API can traverse the entire commit chain with paging all the way to the beginning of the repository.

• For a pull request, the "base" commit is not necessarily in the history reachable from the pull request "head" commit. This is the same for comparison, the "base_commit" is not necessarily a part of the history of the current head.

• The "merge_base_commit" is, however, a part of the history, so the correct approach is to start from the "head" commit, and iterate commit list queries until you reach the "merge_base_commit". For a pull request, this means that it is mandatory to make a compare between "head" and "base" of the pull separately.

• Alternative approach is to use "total_commits" returned by compare, and just iterate backwards until reaching the desired number of commits. This seems to work, however I am not 100% certain that this is correct in all corner cases with merges and such.

So, commit list API, pagination and "merge_base_commit" solves this dilemma.