Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Does Google's crawler index asynchronously loaded elements?

I've built some widget for websites which is asynchronously loaded after the page is loaded:

<html>
    <head>...</head>
    <body>
        <div>...</div>

        <script type="text/javascript">
            (function(){
                var ns = document.createElement("script");
                ns.type = "text/javascript"; 
                ns.async = true;
                ns.src = "http://mydomain.com/myjavascript.js";
                var s = document.getElementsByTagName("script")[0];
                s.parentNode.insertBefore(ns, s);
            })();
        </script>
    </body>
</html>

Is there anyway to notify Google's crawler to index the page only after the page is fully loaded (after the async JavaScript modified the HTML)?

like image 707
vovafeldman Avatar asked Oct 03 '11 19:10

vovafeldman


People also ask

How does Google's Web crawler work?

Crawling: Google downloads text, images, and videos from pages it found on the internet with automated programs called crawlers. Indexing: Google analyzes the text, images, and video files on the page, and stores the information in the Google index, which is a large database.

What is Google crawling indexing?

Crawling is the discovery of pages and links that lead to more pages. Indexing is storing, analyzing, and organizing the content and connections between pages. There are parts of indexing that help inform how a search engine crawls.

Does Google index JavaScript generated content?

We ran a series of tests that verified Google is able to execute and index JavaScript with a multitude of implementations. We also confirmed Google is able to render the entire page and read the DOM, thereby indexing dynamically generated content.

Does Google crawl content?

Most of our Search index is built through the work of software known as crawlers. These automatically visit publicly accessible webpages and follow links on those pages, much like you would if you were browsing content on the web.


2 Answers

No. You have to set up static mirror pages for asynchronous content. See here: http://code.google.com/web/ajaxcrawling/docs/getting-started.html

like image 194
Darth Egregious Avatar answered Nov 15 '22 00:11

Darth Egregious


Things have evolved since then:

  • Google crawls and indexes all content that was injected by javascript.
  • Google even shows results in the SERP that are based on asynchronously injected content.
  • Google can handle content from httpRequest().

(...)

  • Dynamically updated meta elements get crawled and indexed, too.

Source: http://www.centrical.com/test/google-json-ld-and-javascript-crawling-and-indexing-test.html

like image 42
Teodor Sandu Avatar answered Nov 14 '22 22:11

Teodor Sandu