Grabbing tagged instagram photos in real time

Question

I'm trying to download photos posted with specific tag in real time. I found real time api pretty useless so I'm using long polling strategy. Below is pseudocode with comments of sublte bugs in it

newMediaCount = getMediaCount();
delta = newMediaCount - mediaCount;
if (delta > 0) {
    // if mediaCount changed by now, realDelta > delta, so realDelta - delta photos won't be grabbed and on next poll if mediaCount didn't change again realDelta - delta would be duplicated else ...
    // if photo posted from private account last photo will be duplicated as counter changes but nothing is added to recent
    recentMedia = getRecentMedia(delta);
    // persist recentMedia
    mediaCount = newMediaCount;
}

Second issue can be addressed with Set of some sort I gueess. But first really bothers me. I've moved two calls to instagram api as close as possible but is this enough?

Edit

As Amir suggested I've rewritten the code with use of min/max_tag_ids. But it still skips photos. I couldn't find better way to test this than save images on disk for some time and compare result to instagram.com/explore/tags/.

public class LousyInstagramApiTest {

   @Test
    public void testFeedContinuity() throws Exception {
        Instagram instagram = new Instagram(Settings.getClientId());
        final String TAG_NAME = "portrait";
        String id = instagram.getRecentMediaTags(TAG_NAME).getPagination().getMinTagId();
        HashtagEndpoint endpoint = new HashtagEndpoint(instagram, TAG_NAME, id);

        for (int i = 0; i < 10; i++) {
            Thread.sleep(3000);
            endpoint.recentFeed().forEach(d -> {
                try {
                    URL url = new URL(d.getImages().getLowResolution().getImageUrl());
                    BufferedImage img = ImageIO.read(url);
                    ImageIO.write(img, "png", new File("D:\\tmp\\" + d.getId() + ".png"));
                } catch (Exception e) {
                    e.printStackTrace();
                }
            });
        }
    }
}

class HashtagEndpoint {
    private final Instagram instagram;
    private final String hashtag;
    private String minTagId;

    public HashtagEndpoint(Instagram instagram, String hashtag, String minTagId) {
        this.instagram = instagram;
        this.hashtag = hashtag;
        this.minTagId = minTagId;
    }

    public List<MediaFeedData> recentFeed() throws InstagramException {
        TagMediaFeed feed = instagram.getRecentMediaTags(hashtag, minTagId, null);
        List<MediaFeedData> dataList = feed.getData();
        if (dataList.size() == 0) return Collections.emptyList();

        String maxTagId = feed.getPagination().getNextMaxTagId();
        if (maxTagId != null && maxTagId.compareTo(minTagId) > 0) dataList.addAll(paginateFeed(maxTagId));
        Collections.reverse(dataList);
//        dataList.removeIf(d -> d.getId().compareTo(minTagId) < 0);

        minTagId = feed.getPagination().getMinTagId();
        return dataList;
    }

    private Collection<? extends MediaFeedData> paginateFeed(String maxTagId) throws InstagramException {
        System.out.println("pagination required");

        List<MediaFeedData> dataList = new ArrayList<>();
        do {
            TagMediaFeed feed = instagram.getRecentMediaTags(hashtag, null, maxTagId);
            maxTagId = feed.getPagination().getNextMaxTagId();
            dataList.addAll(feed.getData());
        } while (maxTagId.compareTo(minTagId) > 0);
        return dataList;
    }

}

Answer 1

Using the Tag endpoints to get the recent media with a desired tag, it returns a min_tag_id in its pagination info, which is tied to the most recently tagged media at the time of your call. As the API also accepts a min_tag_id parameter, you can pass that number from your last query to only receive those media that are tagged after your last query.

So based on whatever polling mechanism you have, you just call the API to get the new recent media if any based on last received min_tag_id.

You will also need to pass a large count parameter and follow the pagination of the response to receive all data without losing anything when the speed of tagging is faster than your polling.

Update:
Based on your updated code:

public List<MediaFeedData> recentFeed() throws InstagramException {
    TagMediaFeed feed = instagram.getRecentMediaTags(hashtag, minTagId, null, 100000);
    List<MediaFeedData> dataList = feed.getData();
    if (dataList.size() == 0) return Collections.emptyList();

    // follow the pagination
    MediaFeed recentMediaNextPage = instagram.getRecentMediaNextPage(feed.getPagination());
    while (recentMediaNextPage.getPagination() != null) {
        dataList.addAll(recentMediaNextPage.getData());
        recentMediaNextPage = instagram.getRecentMediaNextPage(recentMediaNextPage.getPagination());
    }

    Collections.reverse(dataList);

    minTagId = feed.getPagination().getMinTagId();
    return dataList;
}