C# httpwebrequest and javascript

Question

I am using C# HttpWebRequest to get some data of a webpage. The problem is that some of the data is updated using javascript/ajax after the page is loaded and I am not getting it in the response string. Is there a way to have the webrequest wait untill all the scripts in the page have finished executing?

Thanks

Amit

Answer 1

Just an idea but there is a way to have .net load a webpage as if it were in a browser: using System.Windows.Forms

you could Load the webpage into a WebBrowser control

WebBrowser wb = new WebBrowser(); wb.ScrollBarsEnabled = false; wb.ScriptErrorsSuppressed = true; wb.Navigate(url); while (wb.ReadyState != WebBrowserReadyState.Complete) { Application.DoEvents(); } wb.Document.DomDocument.ToString() 

This will probably give you the pre ajax DOM but maybe there is a way to let it run the ajax first.