Trouble getting the name of a product from a webpage

Question

I've written a script in php to scrape the title of a product located at the top right corner in a webpage. The title is visible as Gucci.

when I execute my below script, it gives me an error Notice: Trying to get property 'plaintext' of non-object in C:\xampp\htdocs\runcode\testfile.php on line 16.

How can I get only the name Gucci from that webpage?

Link to the url

I've written so far:

<?php
include "simple_html_dom.php";
$link = "https://www.farfetch.com//bd/shopping/men/gucci-rhyton-web-print-leather-sneaker-item-12964878.aspx"; 

function get_content($url)
    {
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, $url);
        curl_setopt($ch, CURLOPT_HTTPHEADER, array('User-Agent: Mozilla/5.0',));
        curl_setopt($ch, CURLOPT_BINARYTRANSFER, 1);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        $htmlContent = curl_exec($ch);
        curl_close($ch);
        $dom = new simple_html_dom();
        $dom->load($htmlContent);
        $itemTitle = $dom->find('#bannerComponents-Container [itemprop="name"]', 0)->plaintext;
        echo "{$itemTitle}";
    }
get_content($link);
?>

Btw, the selector I've used within the script is flawless.

To clear the confusion I've copied a chunk of html elements from the page source which neither generats dynamically nor javascript encrypted so I don't find any reason for curl not to be able to handle that:

<div class="cdb2b6" id="bannerComponents-Container">
    <p class="_41db0e _527bd9 eda00d" data-tstid="merchandiseTag">New Season</p>
    <div class="_1c3e57">
        <h1 class="_61cb2e" itemProp="brand" itemscope="" itemType="http://schema.org/Brand">
            <a href="/bd/shopping/men/gucci/items.aspx" class="fd9e8e e484bf _4a941d f140b0" data-trk="pp_infobrd" data-tstid="cardInfo-title" itemProp="url" aria-label="Gucci">
                <span itemProp="name">Gucci</span>
            </a>
        </h1>
    </div>
</div>

Post script: It's very pathetic that I had to show a real life example from another language to make sure the name Gucci is not dynamically generated as few comments and an answer have already indicated that

The following script is written in python (using requests module which can't handle dynamic content):

import requests
from bs4 import BeautifulSoup

url = "https://www.farfetch.com//bd/shopping/men/gucci-rhyton-web-print-leather-sneaker-item-12964878.aspx"

with requests.Session() as s:
    s.headers["User-Agent"] = "Mozilla/5.0"
    res = s.get(url)
    soup = BeautifulSoup(res.text,"lxml")
    item = soup.select_one('#bannerComponents-Container [itemprop="name"]').text
    print(item)

Output It produces:

Gucci

Now, it's clear that the content I look for is static.

Please check out the below image to recognize the title which I've already marked by a pencil.

Answer 1

@t.m.adam already solved the problem, i just want to add that there's no good reason to use simple_html_dom today, seems unmaintained, development stopped in 2014, there's lots of unresolved bugreports, and most importantly, DOMDocument & DOMXPath can do just about everything simple_html_dom can, and is maintained, and is an integrated part of PHP, which means there's nothing to include/bundle with your script. parsing it with DOMDocument & DOMXPath would look like:

$htmlContent = curl_exec($ch);
curl_close($ch);
fclose($cookieFileh); // thanks to tmpfile(), this also deletes the cookie file.
$dom = @DOMDocument::loadHTML($htmlContent);
$xp=new DOMXPath($dom);
$itemTitle = $xp->query('//*[@id="bannerComponents-Container"]//*[@itemprop="name"]')->item(0)->textContent;
echo $itemTitle;

Answer 2

Your selector works in a browser indeed, but your selector is not present when you use curl to get the page source.

Try saving the curled page in terminal and you'll see that the page structure is different from what you see in the browser.

This is true for most modern websites because they use Javascript heavily and curl does not run javascript for you.

I saved the curl results into a file, the brand info looks like this:

<a itemprop="brand" class="generic" data-tstid="Label_ItemBrand" href="/bd/shopping/men/gucci/items.aspx" dir="ltr">Gucci</a>

Answer 3

The main difference between your successful Python script and your PHP script is the use of session. Your PHP script doesn't use cookies, and that triggers a differend response from the server.

We have two options:

1. Change the selector. As mentioned in Mark's answer, the item is still on the html, but in a different tag. We could get it with this selector:

   'a[itemprop="brand"]'
   

2. Use cookies. We can get the same response as your Python script if we use CURLOPT_COOKIESESSION and a temporary file to write/read the cookies.

   function get_content($url) {
       $cookieFileh = tmpfile();
       $cookieFile=stream_get_meta_data($cookieFileh)['uri'];
       $ch = curl_init();
       curl_setopt($ch, CURLOPT_URL, $url);
       curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0');
       curl_setopt($ch, CURLOPT_COOKIESESSION, true);
       curl_setopt($ch, CURLOPT_COOKIEJAR, $cookieFile);
       curl_setopt($ch, CURLOPT_COOKIEFILE, $cookieFile); 
       curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); //
       curl_setopt($ch, CURLOPT_ENCODING, "gzip");
       curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
       curl_exec($ch);
       $htmlContent = curl_exec($ch);
       curl_close($ch);
       fclose($cookieFileh); // thanks to tmpfile(), this also deletes the cookie file.
       $dom = new simple_html_dom();
       $dom->load($htmlContent);
       $itemTitle = $dom->find('#bannerComponents-Container [itemprop="name"]', 0)->plaintext;
       echo "{$itemTitle}";
   }
   
   $link = "https://www.farfetch.com/bd/shopping/men/gucci-rhyton-web-print-leather-sneaker-item-12964878.aspx"; 
   get_content($link);
   //Gucci
   

   This script performs two requests; the first request writes the cookies to file, the second reads and uses them.

   In this case the server returns a compressed response, so I've used CURLOPT_ENCODING to unzip the contents.

   Since you use headers only to set a user-agent, it's best to use the CURLOPT_USERAGENT option.

   I've set CURLOPT_SSL_VERIFYPEER to false because I haven't set a certificate, and CURL fails to use HTTPS. If you can communicate with HTTPS sites it's best not to use this option for security reasons. If not, you could set a certifcate with CURLOPT_CAINFO.