Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to get webcontent that is loaded by JavaScript using cURL?

I am trying to grab the score table from a website. The problem is that the site is using javascript that apparently loads on document.ready.

For that reason I don't seem to succeed getting the table and all I get is the buttons at the top of that page.

Any suggestions or ideas on how to solve this?

like image 384
m33ts4k0z Avatar asked Dec 12 '13 20:12

m33ts4k0z


People also ask

How do I get page content on curl?

To get this to work, just urlencode($strSearch) in the $url , and remove urlencode() from the CURLOPT_URL line. Show activity on this post.

Does curl load Javascript?

cURL will only get you the markup of the page. It won't load any additional resources or process the page. You probably want to look at something like PhantomJS for this. PhantomJS is a headless WebKit browser.

What does curl do in Javascript?

Curl is a popular command line tool often used for making HTTP requests. Curl supports a wide variety of other protocols, but, as a Node. js developer, you'll most likely use it to make HTTP requests to RESTful APIs.

How does curl use data to get from a website?

To make a GET request using Curl, run the curl command followed by the target URL. Curl automatically selects the HTTP GET request method unless you use the -X, --request, or -d command-line option.


2 Answers

cURL will only get you the markup of the page. It won't load any additional resources or process the page. You probably want to look at something like PhantomJS for this. PhantomJS is a headless WebKit browser. It has its own API that lets you "script" behavior. So you can tell PhantomJS to load the page and dump out the data you need.

like image 173
Vivin Paliath Avatar answered Oct 11 '22 03:10

Vivin Paliath


Depending on what exactly you're trying to do, you could just fetch the JSON data that makes the table:

http://www.opap.gr/web/services/rs/betting/availableBetGames/sport/program/4100/0/sport-1.json?localeId=el_GR

like image 42
Lasse Avatar answered Oct 11 '22 01:10

Lasse