Indexing ajax loaded contents

Question

Is there a widely used standard way on how to index ajax loaded content (for search engines)?

For example, indexing HTML content that would dynamically be inserted into a page.

Thanks

Answer 1

You may want to consider using some sort of sitemap generator that aggregates all the content you normally load through AJAX.

Sitemaps are particularly beneficial on websites where:

• Some areas of the website are not available through the browsable interface, or
• Webmasters use rich Ajax, Silverlight, or Flash content that is not normally processed by search engines.

From Wikipedia - Sitemaps

Remember that:

Because most web crawlers do not execute JavaScript code, publicly indexable web applications should provide an alternative means of accessing the content that would normally be retrieved with Ajax, to allow search engines to index it.

From Wikipedia - AJAX Drawbacks

In addition you may be interested in checking out the following articles:

• Official Google Webmaster Central Blog - A proposal for making AJAX crawlable
• SoftwareDeveloper.com - How to: Get Google and AJAX to Play Nice
• Crawling Ajax-driven Web 2.0 Applications

Answer 2

One way of doing this is using JS fallbacks for dialog boxes like thickbox: A link would point to the dialog box loading Ajax content, and the fallback href='...' would point to a search engine-readable representation of that content (i.e. the HTML snippet that the AJAX function would load, but surrounded by the necessary HTML body basics).

Example (I pulled rel='box' out of my arse, this is supposed to be the anchor for the box plugin, like rel=thickbox):

<a href='/encyclopedia/definition/mushroom.html' rel='box'>Definition of Mushroom</a>

• Clicking on the link in a Ajax/JS enabled browser will open a nice dialog box with the article
• Clicking on the link without JS (or as a search engine) will lead to a new page containing the article (which needs some server side intelligence to detect which channel the request came from).

That's all that comes to my mind in this direction. Ajax and search engines is a widely uncharted field otherwise.