HttpWebRequest.BeginGetRequestStream() best practice

Question

I'm working on an async Http crawler that gathers data from various services, and at the moment, I'm working with threadpools that do serial HttpWebRequest calls to post/get data from the services.

I want to transition over to the async web calls (BeginGetRequestStream and BeginGetResponse), I need some way get the response data and POST stats (% completed with the write, when complete (when complete more important), etc). I currently have an event that is called from the object that spawns/contains the thread, signaling HTTP data has been received. Is there an event in the WebRequests I can attach to to call the already implemented event? That would be the most seamless for the transition.

Thanks for any help!!

Answer 1

The following code I just copy/pasted (and edited) from this article about asynchronous Web requests. It shows a basic pattern of how you can write asynchronous code in a somewhat organized fashion, while keeping track of what responses go with what requests, etc. When you're finished with the response, just fire an event that notifies the UI that a response finished.

private void ScanSites ()
{
  // for each URL in the collection...
  WebRequest request = HttpWebRequest.Create(uri);

  // RequestState is a custom class to pass info
  RequestState state = new RequestState(request, data);

  IAsyncResult result = request.BeginGetResponse(
    new AsyncCallback(UpdateItem),state);
}

private void UpdateItem (IAsyncResult result)
{
  // grab the custom state object
  RequestState state = (RequestState)result.AsyncState;
  WebRequest request = (WebRequest)state.request;
  // get the Response
  HttpWebResponse response =
    (HttpWebResponse )request.EndGetResponse(result);

  // fire the event that notifies the UI that data has been retrieved...
}

Note you can replace the RequestState object with any sort of object you want that will help you keep track of things.

You are probably already doing this, but if not, I believe this is a perfectly acceptable and clean way to tackle the problem. If this isn't what you were looking for, let me know.