Accessing an item beyond start_index=1000 in a YouTube user upload feed

Question

I am currently trying to pull data about videos from a YouTube user upload feed. This feed contains all of the videos uploaded by a certain user, and is accessed from the API by a request to:

http://gdata.youtube.com/feeds/api/users/USERNAME/uploads

Where USERNAME is the name of the YouTube user who owns the feed.

However, I have encountered problems when trying to access feeds which are longer than 1000 videos. Since each request to the API can return 50 items, I am iterating through the feed using max_length and start_index as follows:

http://gdata.youtube.com/feeds/api/users/USERNAME/uploads?start-index=1&max-results=50&orderby=published
http://gdata.youtube.com/feeds/api/users/USERNAME/uploads?start-index=51&max-results=50&orderby=published

And so on, incrementing start_index by 50 on each call. This works perfectly up until:

http://gdata.youtube.com/feeds/api/users/USERNAME/uploads?start-index=1001&max-results=50&orderby=published

At which point I receive a 400 error informing me that 'You cannot request beyond item 1000.' This confused me as I assumed that the query would have only returned 50 videos: 1001-1051 in the order of most recently published. Having looked through the documentation, I discovered this:

Limits on result counts and accessible results

...

For any given query, you will not be able to retrieve more than 1,000 results even if there are more than that. The API will return an error if you try to retrieve greater than 1,000 results. Thus, the API will return an error if you set the start-index query parameter to a value of 1001 or greater. It will also return an error if the sum of the start-index and max-results parameters is greater than 1,001.

For example, if you set the start-index parameter value to 1000, then you must set the max-results parameter value to 1, and if you set the start-index parameter value to 980, then you must set the max-results parameter value to 21 or less.

I am at a loss about how to access a generic user's 1001st last uploaded video and beyond in a consistent fashion, since they cannot be indexed using only max-results and start-index. Does anyone have any useful suggestions for how to avoid this problem? I hope that I've outlined the difficulty clearly!

Answer 1

Getting all the videos for a given account is supported, but you need to make sure that your request for the uploads feed is going against the backend database and not the search index. Because you're including orderby=published in your request URL, you're going against the search index. Search index feeds are limited to 1000 entries.

Get rid of the orderby=published and you'll get the data you're looking for. The default ordering of the uploads feed is reverse-chronological anyway.

This is a particularly easy mistake to make, and we have a blog post up explaining it in more detail:

http://apiblog.youtube.com/2012/03/keeping-things-fresh.html

The nice thing is that this is something that will no longer be a problem in version 3 of the API.