Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Axios and fetch responded with different status code from same external endpoint

I'm trying to send a request to this endpoint: https://9gag.com/v1/group-posts/group/default/type/trending to get 9gag posts data.

Works fine on postman and chrome, but when I tried using axios from Node.js, it responded with 403 (and reading the returned DOM, it believe that it's asking me to enter captcha), and when I tried to send a request using node-fetch, it responded with 200 with posts data (which is the correct one).

Question is, how come node-fetch works fine but axios doesn't?

Here's the code I used for testing:

import axios from "axios";
import fetch from "node-fetch";

const URL = "https://9gag.com/v1/group-posts/group/default/type/trending";

// Inside async function
await fetch(URL); // Responded with 200 and json data
await axios(URL); // Responded with 403 and HTML DOM, axios.get() also gives the same result
like image 716
Owl Avatar asked Nov 16 '22 04:11

Owl


1 Answers

If you take a look at https://9gag.com/robots.txt, you might notice Mediapartners-Google. It's a User-Agent you have to specify in headers to bypass the Cloudflare captcha.

like image 82
Ximaz Avatar answered Dec 05 '22 18:12

Ximaz