F# WebRequest throws WebException ("protocol error") instead of 404 status code

Question

I'm implementing a F#-ish web request function which looks like this:

let request 
        (httpMethod:string) 
        (url:string) 
        (headers:Header list) 
        (content:Content option)=

    let groupHeaders (headers:WebHeaderCollection) =
        let d = new Dictionary<string, string list> ()
        let addToDict h =
            let oldValue = if d.ContainsKey(h) then d.Item(h) else []
            d.Item(h) <- oldValue @ [headers.Get h]

        headers.AllKeys |> Array.iter addToDict

        d |> Seq.map (|KeyValue|) |> Map.ofSeq

    async {
        let rq = WebRequest.Create(url) :?> HttpWebRequest
        rq.Method <- httpMethod

        headers 
        |> List.iter (fun (key, value) -> rq.Headers.Add(key, value) |> ignore ) 

        match content with
        | Some (contentType, bytes) ->
            rq.ContentType <- contentType
            do! rq.GetRequestStream().AsyncWrite(bytes) 
        | None -> ()

        try
            use! response = rq.AsyncGetResponse()
            let webResponse = response :?> HttpWebResponse

            let responseCode = webResponse.StatusCode

            let stream = webResponse.GetResponseStream()
            let length = webResponse.ContentLength |> int32 //TODO what if the data is bigger then 4GB?

            let! bytes = stream.AsyncRead(length)

            let respHeaders = groupHeaders webResponse.Headers
            return
                match webResponse.StatusCode with 
                | HttpStatusCode.OK -> OK (respHeaders, bytes)
                | HttpStatusCode.NotFound -> NotFound
                | otherCode -> Error <| otherCode.ToString()
        with 
            | :? WebException as ex -> 
                return Error <| ex.Status.ToString()
    }

The problem is that if I try to get a page (try to get a riak value) which returns 404 the result is a WebException with the message The remote server returned an error: (404) Not Found. instead of a response with HttpStatusCode.NotFound

Doing curl GET for the same URL gives me the following result:

$ curl -v http://localhost:18098/riak/user/asfasfd?returnbody=true
* About to connect() to localhost port 18098 (#0)
*   Trying ::1...
* connected
* Connected to localhost (::1) port 18098 (#0)
> GET /riak/user/asfasfd?returnbody=true HTTP/1.1
> User-Agent: curl/7.27.0
> Host: localhost:18098
> Accept: */*
>
* additional stuff not fine /usr/src/ports/curl/curl-7.27.0-1/src/curl-7.27.0/lib/transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 404 Object Not Found
< Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
< Date: Sun, 28 Oct 2012 05:30:12 GMT
< Content-Type: text/plain
< Content-Length: 10
<
not found
* Connection #0 to host localhost left intact
* Closing connection #0

Any ideas why the exception WebException is thrown? If I want to identify the 404-page not found situation, what options do I have instead of parsing the exception message?

Answer 1

@desco is correct, but elaborating on his answer -- here's how you can use pattern matching to solve your problem:

try
    // Do stuff here
    //
with
| :? WebException as webEx when (webEx.Response :? HttpWebResponse) ->
    /// The exception's Response, as an HttpWebResponse.
    /// From this we can get the HTTP status code of the response.
    let httpWebResponse = webEx.Response :?> HttpWebResponse

    // Return an error message based on the HTTP status code.
    match httpWebResponse.StatusCode with
    | HttpStatusCode.NotFound ->
        return NotFound

    | otherCode ->
        return Error <| otherCode.ToString()

| :? WebException as webEx ->
        return Error <| webEx.Status.ToString()

Also, the reason you get the WebException is because that's how HttpWebResponse handles 4xx and 5xx response codes. The match statement in your code where you check webResponse.StatusCode won't ever reach the HttpStatusCode.NotFound case because the exception will be thrown instead. You should keep the match there though, to handle any non-error codes which are not HttpStatusCode.OK (e.g., a 301 redirect).

Answer 2

WebException has 2 properties that should help you:

1. Status of type WebExceptionStatus. 404 should be indicated by ProtocolError
2. Response exposes response returned by remote host. Downcast response to HttpWebResponse and its use Status property