Hidden parts in html source code while scraping (python)

Question

So I want to scrape the 'Buy price' integer from this url: https://rsbuddy.com/exchange?id=5502

But when I look at the source code, I can't reach those prices. Neither does BeautifulSoup scraper. This is the ouput from BeautifulSoup:

<div class="col-md-7" id="buy-price">
    ---
</div>

But when I 'inspect element' using chrome, i actually am able to see that price:

<div id="buy-price" class="col-md-7">29,990 gp</div>

Why is that part of the code 'hidden'? Is it simply because they don't want people to scrape from their website? Is there a way to get around this?

Thanks in advance

EDIT: I found the answer by tracking the javascript traffic using the chrome tools. Apparently even though api.rsbuddy.com doesn't give you anything, it does use the api: https://api.rsbuddy.com/grandExchange?a=guidePrice&i=5502

Answer 1

If certain parts of the page are inserted via JavaScript your best bet is to use something like selenium with PhantomJS as the driver.

The Python binding's are quite easy to use and this will allow the JavaScript to execute in the browser and you can grab the prices from there.

Let me know if you want any more information and I'd be happy to help.