PRAW 6: Get all submission of a subreddit

Question

I'm trying to iterate over submissions of a certain subreddit from the newest to the oldest using PRAW. I used to do it like this:

subreddit = reddit.subreddit('LandscapePhotography')
for submission in subreddit.submissions(None, time.time()):
    print("Submission Title: {}".format(submission.title))

However, when I try to do it now I get the following error:

AttributeError: 'Subreddit' object has no attribute 'submissions'

From looking at the docs I can't seem to figure out how to do this. The best I can do is:

for submission in subreddit.new(limit=None):
    print("Submission Title: {}".format(submission.title))

However, this is limited to the first 1000 submissions only.

Is there a way to do this with all submissions and not just the first 1000 ?

Answer 1

Unfortunately, Reddit removed this function from their API.

Check out the PRAW changelog. One of the changes in version 6.0.0 is:

Removed

• Subreddit.submissions as the API endpoint backing the method is no more. See https://www.reddit.com/r/changelog/comments/7tus5f/update_to_search_api/.

The linked post says that Reddit is disabling Cloudsearch for all users:

Starting March 15, 2018 we’ll begin to gradually move API users over to the new search system. By end of March we expect to have moved everyone off and finally turn down the old system.

PRAW's Subreddit.sumbissions() used Cloudsearch to search for posts between the given timestamps. Since Cloudsearch has been removed and the search that replaced it doesn't support timestamp search, it is no longer possible to perform a search based on timestamp with PRAW or any other Reddit API client. This includes trying to get all posts from a subreddit.

For more information, see this thread from /r/redditdev posted by the maintainer of PRAW.

---

Alternatives

Since Reddit limits all listings to ~1000 entries, it is currently impossible to get all posts in a subreddit using their API. However, third-party datasets with APIs exist, such as pushshift.io. As /u/kungming2 said on Reddit:

You can use Pushshift.io to still return data from defined time periods by using their API:

https://api.pushshift.io/reddit/submission/search/?after=1334426439&before=1339696839&sort_type=score&sort=desc&subreddit=translator

This, for example, allows you to parse submissions to r/translator between 2012-04-14 and 2012-06-2014.

Answer 2

You can retrieve all the data from pushshift.io using an iterative loop. Just set the start date as the current epoch date, and get 1000 items, then put the created_utc of the last items in the list as the before parameter to get the next 1000 items and keeps going until it stops returning.

Below is a useful link for further information: https://www.reddit.com/r/pushshift/comments/b7onr6/max_number_of_results_returned_per_query/enter link description here