How to get data json from 9GAG

Question

maybe you think it's a silly question but I hope you can give me some advises.

My problem: When I view source of 9gag.com, I realize that they have a some lines code to load more content:

<div class="loading">
    <a class="btn badge-load-more-post" href="/?id=ag0jnqK%2CaNoj2Nv%2CaEGZb39&c=10" data-loading-text="Loading more posts..." data-load-count-max="30">Load more posts</a>
</div>

Then, I try to get data by using PHP-CURL from http://9gag.com/?id=ag0jnqK%2CaNoj2Nv%2CaEGZb39&c=10, I hope I can get data application/json but it return is a text/html ( full html).

I tried to use Network Inspector of Google Chrome, I realize that the data return when Chrome get from link http://9gag.com/?id=ag0jnqK%2CaNoj2Nv%2CaEGZb39&c=10

It is possible IF there is a way to help me get data application/json above like Google Chrome.

Answer 1

9gag supports JSON requests ! There is an example of URL : https://9gag.com/v1/tag-posts/tag/samantha/type/fresh

The only thing you have to do is to specify a User-Agent called Mediapartners-Google that you can find at https://9gag.com/robots.txt to bypass the Cloudflare captcha.

EDIT : It seems that 9gag removed their Cloudflare protection from their API, so you may not have to get a specific user agent. It a guess, but if you ge tin trouble, you still can do the user agent method.