Second call to HttpWebRequest.GetRequestStream() throws timeout exception

Question

Help... I can't figure out why HttpWebRequest.GetRequestStream() is throwing a timeout exception... it ALWAYS occurs when trying to establish the second (and subsequent) connections using POST. Basically, I am trying to create the connection every 30 seconds. Here is the relevent code:

My Main Loop:

for( int i = 0; i < students.Count; i++ ) {

    Log( "Processing: " + students[i].DbID + ";" + students[i].Username + ";" + students[i].Password );

    UserConnection uc = new UserConnection( students[i] );
    uc.ParseAll();


    Log( "Done Processing: " + students[i].DbID + ";" + students[i].Username + ";" + students[i].Password );
}

From User Connection:

public UserConnection( Student student )
{
    this.student = student;

    this.cookies = new CookieContainer();
    this.InitCookies();

    this.courses = new List<Course>();
}


private void InitCookies()
{
    HttpWebRequest req = (HttpWebRequest)WebRequest.Create( baseUri );
    req.Method = "GET";
    req.CookieContainer = this.cookies;

    req.GetResponse();


}

public void ParseAll()
{
    ParseCourseInfo(); //get info for each class
    .
    .
    .
}

private void ParseCourseInfo()
{
    HtmlDocument page = GetPage( "POST", homeUri, "username=" + student.Username + "&password=" + student.Password + "&testcookies=1" );
    .
    .
    .
}

The following exception is occuring in GetPage:

29/07/2012 1:04:22 PM : Exception: System.Net.WebException
Message: The operation has timed out
Source: System
Stack Trace:    at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
   at System.Net.HttpWebRequest.GetRequestStream()
   at URCoursesParserV2.UserConnection.GetPage(String method, Uri pageUri, String queryString) in UserConnection.cs:line 167

Here is the GetPage code where the problem is:

private HtmlDocument GetPage( string method, Uri pageUri, String queryString = "" )
{
    Stream data = null;
    HttpWebResponse res = null;
    HttpWebRequest req = null;
    try {

        req = (HttpWebRequest)WebRequest.Create( pageUri );
        req.CookieContainer = this.cookies;
        req.Timeout = 1000 * 10; //10 seconds - I've also tried leaving it as default
        req.KeepAlive = false; ////I've also tried leaving it as true.

        if( method.ToUpper() == "POST" ) {
            req.Method = "POST";

            if( queryString != "" ) {
                byte[] postBytes = Encoding.UTF8.GetBytes( queryString );
                req.ContentType = "application/x-www-form-urlencoded";
                req.ContentLength = postBytes.Length;

                using( data = req.GetRequestStream() ) { //////////////EXCEPTION HERE ON SECOND ITERATION
                    data.Write( postBytes, 0, postBytes.Length );
                    data.Close();
                }
            }

        } else if( method.ToUpper() == "GET" ) {
            req.Method = "GET";

            if( queryString != "" ) {
                pageUri = new Uri( pageUri.ToString() + '?' + queryString );
            }

        } else {
            return null;
        }

        HtmlDocument page = null;
        using( res = (HttpWebResponse)req.GetResponse() ) {
            using( data = res.GetResponseStream() ) {
                page = new HtmlDocument();
                page.Load( data );
            }
        }
        return page;

    } catch(WebException e) {
        URCoursesParser.Log( e );
        return null;

    } catch( Exception e ) {
        URCoursesParser.Log( e );
        return null;

    } finally { ///data and res probably don't need to be checked here now because of 'using' blocks
        if( data != null ) {
            data.Close();
            data = null;
        }
        if( res != null ) {
            res.Close();
            res = null;
        }
        if( req != null ) {
            req.Abort();
            req = null;
        }
    }
}

Any ideas!?! The first iteration is always fine. If I terminate the program and restart the first iteration is again fine... It's always the second and subsequent iterations.

Answer 1

This could be part of the problem:

private void InitCookies()
{
    HttpWebRequest req = (HttpWebRequest)WebRequest.Create( baseUri );
    req.Method = "GET";
    req.CookieContainer = this.cookies;
    req.GetResponse();
}

Here you're not disposing of the response... so it's not returning the connection to the pool.

It's not really clear what this method is meant to do, but you should dispose of the response:

using (req.GetResponse()) {}

Answer 2

I had a similar problem, but I explicitly wanted to exit the method before waiting for the response. I set the Timeout of the request to 100 milliseconds, and that seemed to fix it for me.