Get all visible plain text and find out which HTML tag or DOM element each piece of text belongs to

Question

I know how to get all visible plain text on a page:

const text = await page.$eval('*', el => el.innerText);

But I also need to know which element of the page each piece of text belongs to, and I can't find a way to do that.

Aankhen · Accepted Answer

On the client side, you can do this in a way that preserves order using TreeWalker. Here’s an example with sample content from Web Scraper Testing Ground:

const IGNORE = ["style", "script"];

const walker = document.createTreeWalker(document.body, NodeFilter.SHOW_TEXT);

const pairs = [];

let node;

while ((node = walker.nextNode()) !== null) {
  const parent = node.parentNode.tagName;

  if (IGNORE.includes(parent)) {
    continue;
  }

  const value = node.nodeValue.trim();

  if (value.length === 0) {
    continue;
  }

  pairs.push([parent.toLowerCase(), value]);
}

console.log(pairs);

<div id="topbar"></div>
		<a href="/" style="text-decoration: none">
		    <div id="title">WEB SCRAPER TESTING GROUND</div>
		    <div id="logo"></div>
		</a>
		<div id="content">
<h1>BLOCKS: Price List </h1>
<div id="caseinfo">In this test, the web scraper needs to scrape a price list organized in a block layout. Specifically, it has to:
	<ol>
		<li>Extract all the products (their names, descriptions and prices), while skipping advertisements</li>
		<li>Scrape discounted products only</li>
		<li>Scrape products with red prices only</li>
	</ol>
<p>
</p><p>There is a <b>ver</b> parameter (which varies from 1 to 5) to show different table versions (with different product numbers, best price and advertisement positions).</p>
<p>Also there are two tables presented:
	</p><ul>
		<li><b>Case 1</b> (simple one, with products and prices placed into the same block)
		</li><li><b>Case 2</b> (complicated one, with products and prices placed into separate blocks)</li>
	</ul>
<p></p>
<p>For testing, you may use the following sample links. The scraper should sufficiently scrape all data from a certain case using the same project:
</p><ul>
	<li><a href="/blocks?ver=1">Price list 1</a></li>
	<li><a href="/blocks?ver=2">Price list 2</a></li>
	<li><a href="/blocks?ver=3">Price list 3</a></li>
	<li><a href="/blocks?ver=4">Price list 4</a></li>
	<li><a href="/blocks?ver=5">Price list 5</a></li>
</ul>
<p></p>
</div>

<div id="case_blocks">

<h2>Case 1</h2>
<div id="case1">
<div class="prod2"><span style="float: left"><div class="name">Dell Latitude D610-1.73 Laptop Wireless Computer</div>2 GHz Intel Pentium M, 1 GB DDR2 SDRAM, 40 GB, Microsoft Windows XP Professional</span><span style="float: right">$239.95</span></div><div class="prod1"><span style="float: left"><div class="name">Samsung Chromebook (Wi-Fi, 11.6-Inch)</div>1.7 GHz, 2 GB DDR3 SDRAM, 16 GB, Chrome</span><span style="float: right" class="best">$249.00</span><span style="float: right;margin-right:10px" class="best">BEST<br>PRICE!</span></div><div class="ads">ADVERTISEMENT</div><div class="prod2"><span style="float: left"><div class="name">Apple MacBook Pro MD101LL/A 13.3-Inch Laptop (NEWEST VERSION)</div>2.5 GHz Intel Core i5, 4 GB DDR3 SDRAM, 500 GB Serial ATA, Mac OS X v10.7 Lion</span><span style="float: right">$1,099.99</span></div><div class="prod1"><span style="float: left"><div class="name">Acer Aspire AS5750Z-4835 15.6-Inch Laptop (Black)</div>2 GHz Pentium B940, 4 GB SDRAM, 500 GB, Windows 7 Home Premium 64-bit</span><span style="float: right" class="best">$385.72</span><span style="float: right;margin-right:10px" class="best">BEST<br>PRICE!</span></div><div class="ads">ADVERTISEMENT</div><div class="prod2"><span style="float: left"><div class="name">HP Pavilion g7-2010nr 17.3-Inch Laptop (Black)</div>2.3 GHz Core i3-2350M, 6 GB SDRAM, 640 GB, Windows 7 Home Premium 64-bit</span><span style="float: right">$549.99<div class="disc">discount 7%</div></span></div><div class="prod1"><span style="float: left"><div class="name">ASUS A53Z-AS61 15.6-Inch Laptop (Mocha)</div>1.4 GHz A-Series Quad-Core A6-3420M, 4 GB DIMM, 750 GB, Windows 7 Home Premium 64-bit</span><span style="float: right">$399.99</span></div></div>

<h2 style="margin-top: 50px">Case 2</h2>
<div id="case2">
<div class="left"><div class="prod2"><div class="name">Dell Latitude D610-1.73 Laptop Wireless Computer</div>2 GHz Intel Pentium M, 1 GB DDR2 SDRAM, 40 GB, Microsoft Windows XP Professional</div><div class="prod1"><div class="name">Samsung Chromebook (Wi-Fi, 11.6-Inch)</div>1.7 GHz, 2 GB DDR3 SDRAM, 16 GB, Chrome</div><div class="ads">ADVERTISEMENT</div><div class="prod2"><div class="name">Apple MacBook Pro MD101LL/A 13.3-Inch Laptop (NEWEST VERSION)</div>2.5 GHz Intel Core i5, 4 GB DDR3 SDRAM, 500 GB Serial ATA, Mac OS X v10.7 Lion</div><div class="prod1"><div class="name">Acer Aspire AS5750Z-4835 15.6-Inch Laptop (Black)</div>2 GHz Pentium B940, 4 GB SDRAM, 500 GB, Windows 7 Home Premium 64-bit</div></div><div class="right"><div class="price2">$239.95</div><div class="price1 best">$249.00</div><div class="ads"></div><div class="price2">$1,099.99</div><div class="price1 best">$385.72</div></div><div class="ads" style="clear: both">ADVERTISEMENT</div><div class="left"><div class="prod2"><div class="name">HP Pavilion g7-2010nr 17.3-Inch Laptop (Black)</div>2.3 GHz Core i3-2350M, 6 GB SDRAM, 640 GB, Windows 7 Home Premium 64-bit</div><div class="prod1"><div class="name">ASUS A53Z-AS61 15.6-Inch Laptop (Mocha)</div>1.4 GHz A-Series Quad-Core A6-3420M, 4 GB DIMM, 750 GB, Windows 7 Home Premium 64-bit</div></div><div class="right"><div class="price2">$549.99<div class="disc">discount 7%</div></div><div class="price1">$399.99</div></div></div>

</div>
<br><br><br>
		</div>

Use evaluate to call this in Puppeteer, per Grant Miller’s answer:

const pairs = await page.evaluate(() => {
  const IGNORE = ["style", "script"];
  const NONWHITESPACE_RE = /\S/;

  const result = document.evaluate(
    "//*[child::text()]",
    document,
    null,
    XPathResult.ORDERED_NODE_SNAPSHOT_TYPE,
    null
  );

  const pairs = [];

  for (let i = 0, j = result.snapshotLength; i < j; i++) {
    const element = result.snapshotItem(i);

    if (IGNORE.includes(element.tagName.toLowerCase())) {
      continue;
    }

    const nodes = [...element.childNodes];

    for (const node of nodes) {
      if (node.nodeType !== document.TEXT_NODE) {
        continue;
      }

      if (node.nodeValue.search(NONWHITESPACE_RE) === -1) {
        continue;
      }

      pairs.push({
        tag: element.tagName.toLowerCase(),
        text: node.nodeValue.trim()
      });
    }
  }

  return pairs;
});

console.log(pairs);

Here is the original version of the client-side function, which uses XPath but always puts the direct children of a node before its indirect children:

const IGNORE = ["style", "script"];
const NONWHITESPACE_RE = /\S/;

// get all text nodes in the document
const result = document.evaluate(
  // matches any node in the document that has at least one direct
  // text node child, including whitespace-only nodes
  "//*[child::text()]",
  document,
  null,
  XPathResult.ORDERED_NODE_SNAPSHOT_TYPE,
  null
);

// the result doesn't use the JavaScript iterator protocol, so we have
// to manually iterate over the elements
const pairs = [];

for (let i = 0, j = result.snapshotLength; i < j; i++) {
  const element = result.snapshotItem(i);

  if (IGNORE.includes(element.tagName.toLowerCase())) {
    continue;
  }

  const nodes = [...element.childNodes];

  for (const node of nodes) {
    if (node.nodeType !== document.TEXT_NODE) {
      continue;
    }

    // filter out whitespace-only nodes
    if (node.nodeValue.search(NONWHITESPACE_RE) === -1) {
      continue;
    }

    pairs.push({
      tag: element.tagName.toLowerCase(),
      // remove the `.trim()` to preserve leading & trailing whitespace
      text: node.nodeValue.trim()
    });
  }
}

console.log(pairs);

        <div id="topbar"></div>
		<a href="/" style="text-decoration: none">
		    <div id="title">WEB SCRAPER TESTING GROUND</div>
		    <div id="logo"></div>
		</a>
		<div id="content">
<h1>BLOCKS: Price List </h1>
<div id="caseinfo">In this test, the web scraper needs to scrape a price list organized in a block layout. Specifically, it has to:
	<ol>
		<li>Extract all the products (their names, descriptions and prices), while skipping advertisements</li>
		<li>Scrape discounted products only</li>
		<li>Scrape products with red prices only</li>
	</ol>
<p>
</p><p>There is a <b>ver</b> parameter (which varies from 1 to 5) to show different table versions (with different product numbers, best price and advertisement positions).</p>
<p>Also there are two tables presented:
	</p><ul>
		<li><b>Case 1</b> (simple one, with products and prices placed into the same block)
		</li><li><b>Case 2</b> (complicated one, with products and prices placed into separate blocks)</li>
	</ul>
<p></p>
<p>For testing, you may use the following sample links. The scraper should sufficiently scrape all data from a certain case using the same project:
</p><ul>
	<li><a href="/blocks?ver=1">Price list 1</a></li>
	<li><a href="/blocks?ver=2">Price list 2</a></li>
	<li><a href="/blocks?ver=3">Price list 3</a></li>
	<li><a href="/blocks?ver=4">Price list 4</a></li>
	<li><a href="/blocks?ver=5">Price list 5</a></li>
</ul>
<p></p>
</div>

<div id="case_blocks">

<h2>Case 1</h2>
<div id="case1">
<div class="prod2"><span style="float: left"><div class="name">Dell Latitude D610-1.73 Laptop Wireless Computer</div>2 GHz Intel Pentium M, 1 GB DDR2 SDRAM, 40 GB, Microsoft Windows XP Professional</span><span style="float: right">$239.95</span></div><div class="prod1"><span style="float: left"><div class="name">Samsung Chromebook (Wi-Fi, 11.6-Inch)</div>1.7 GHz, 2 GB DDR3 SDRAM, 16 GB, Chrome</span><span style="float: right" class="best">$249.00</span><span style="float: right;margin-right:10px" class="best">BEST<br>PRICE!</span></div><div class="ads">ADVERTISEMENT</div><div class="prod2"><span style="float: left"><div class="name">Apple MacBook Pro MD101LL/A 13.3-Inch Laptop (NEWEST VERSION)</div>2.5 GHz Intel Core i5, 4 GB DDR3 SDRAM, 500 GB Serial ATA, Mac OS X v10.7 Lion</span><span style="float: right">$1,099.99</span></div><div class="prod1"><span style="float: left"><div class="name">Acer Aspire AS5750Z-4835 15.6-Inch Laptop (Black)</div>2 GHz Pentium B940, 4 GB SDRAM, 500 GB, Windows 7 Home Premium 64-bit</span><span style="float: right" class="best">$385.72</span><span style="float: right;margin-right:10px" class="best">BEST<br>PRICE!</span></div><div class="ads">ADVERTISEMENT</div><div class="prod2"><span style="float: left"><div class="name">HP Pavilion g7-2010nr 17.3-Inch Laptop (Black)</div>2.3 GHz Core i3-2350M, 6 GB SDRAM, 640 GB, Windows 7 Home Premium 64-bit</span><span style="float: right">$549.99<div class="disc">discount 7%</div></span></div><div class="prod1"><span style="float: left"><div class="name">ASUS A53Z-AS61 15.6-Inch Laptop (Mocha)</div>1.4 GHz A-Series Quad-Core A6-3420M, 4 GB DIMM, 750 GB, Windows 7 Home Premium 64-bit</span><span style="float: right">$399.99</span></div></div>

<h2 style="margin-top: 50px">Case 2</h2>
<div id="case2">
<div class="left"><div class="prod2"><div class="name">Dell Latitude D610-1.73 Laptop Wireless Computer</div>2 GHz Intel Pentium M, 1 GB DDR2 SDRAM, 40 GB, Microsoft Windows XP Professional</div><div class="prod1"><div class="name">Samsung Chromebook (Wi-Fi, 11.6-Inch)</div>1.7 GHz, 2 GB DDR3 SDRAM, 16 GB, Chrome</div><div class="ads">ADVERTISEMENT</div><div class="prod2"><div class="name">Apple MacBook Pro MD101LL/A 13.3-Inch Laptop (NEWEST VERSION)</div>2.5 GHz Intel Core i5, 4 GB DDR3 SDRAM, 500 GB Serial ATA, Mac OS X v10.7 Lion</div><div class="prod1"><div class="name">Acer Aspire AS5750Z-4835 15.6-Inch Laptop (Black)</div>2 GHz Pentium B940, 4 GB SDRAM, 500 GB, Windows 7 Home Premium 64-bit</div></div><div class="right"><div class="price2">$239.95</div><div class="price1 best">$249.00</div><div class="ads"></div><div class="price2">$1,099.99</div><div class="price1 best">$385.72</div></div><div class="ads" style="clear: both">ADVERTISEMENT</div><div class="left"><div class="prod2"><div class="name">HP Pavilion g7-2010nr 17.3-Inch Laptop (Black)</div>2.3 GHz Core i3-2350M, 6 GB SDRAM, 640 GB, Windows 7 Home Premium 64-bit</div><div class="prod1"><div class="name">ASUS A53Z-AS61 15.6-Inch Laptop (Mocha)</div>1.4 GHz A-Series Quad-Core A6-3420M, 4 GB DIMM, 750 GB, Windows 7 Home Premium 64-bit</div></div><div class="right"><div class="price2">$549.99<div class="disc">discount 7%</div></div><div class="price1">$399.99</div></div></div>

</div>
<br><br><br>
		</div>

Grant Miller · Answer

You can use the following solution to obtain an array of objects that contain tag names with their associated text:

const example = await page.evaluate(() => {
  return Array.from(document.body.getElementsByTagName('*'), e => [...e.childNodes].filter(e =>
       e.nodeType === 3
    && !['SCRIPT','STYLE'].includes(e.parentNode.tagName)
    && e.textContent.trim().length
  )).flat().map(e => ({
    tag_name: e.parentNode.tagName,
    text_content: e.textContent.trim(),
  }));
});

console.log(example[0].tag_name);     // DIV
console.log(example[0].text_content); // Hello, world!

Get all visible plain text and find out which HTML tag or DOM element each piece of text belongs to

Tags:

puppeteer

Violet Giraffe

2 Answers

Aankhen

Grant Miller

Recent Activity

Donate For Us

Get all visible plain text and find out which HTML tag or DOM element each piece of text belongs to

Tags:

puppeteer

Violet Giraffe

2 Answers

Aankhen

Grant Miller

Related questions

Recent Activity

Donate For Us