F# Continue on Async.Catch

Question

I'm doing many async web requests and using Async.Parallel. Something like:

xs                          
|> Seq.map (fun u -> downloadAsync u.Url)
|> Async.Parallel
|> Async.Catch

Some request may throw exceptions, I want to log them and continue with the rest of urls. I found the Async.Catch function, but this stop the computation when the first exception is thrown. I know I can use a try...with expression within the async expression in order to compute the entire list, but, i think, this implies passing a log function to my downloadAsync function changing his type. Is there any other way to catch the exceptions, log them and continue with the rest of urls?

Answer 1

The 'trick' is to move the catch into the map such that catching is parallelized as well:

open System
open System.IO
open System.Net

type T = { Url : string }

let xs = [
    { Url = "http://microsoft.com" }
    { Url = "thisDoesNotExists" } // throws when constructing Uri, before downloading
    { Url = "https://thisDotNotExist.Either" }
    { Url = "http://google.com" }
]

let isAllowedInFileName c =
    not <| Seq.contains c (Path.GetInvalidFileNameChars())

let downloadAsync url =
    async {
        use client = new WebClient()
        let fn =
            [|
                __SOURCE_DIRECTORY__
                url |> Seq.filter isAllowedInFileName |> String.Concat
            |]
            |> Path.Combine
        printfn "Downloading %s to %s" url fn
        return! client.AsyncDownloadFile(Uri(url), fn)
    }

xs
|> Seq.map (fun u -> downloadAsync u.Url |> Async.Catch)
|> Async.Parallel
|> Async.RunSynchronously
|> Seq.iter (function
    | Choice1Of2 () -> printfn "Succeeded"
    | Choice2Of2 exn -> printfn "Failed with %s" exn.Message)

(*
Downloading http://microsoft.com to httpmicrosoft.com
Downloading thisDoesNotExists to thisDoesNotExists
Downloading http://google.com to httpgoogle.com
Downloading https://thisDotNotExist.Either to httpsthisDotNotExist.Either
Succeeded
Failed with Invalid URI: The format of the URI could not be determined.
Failed with The remote name could not be resolved: 'thisdotnotexist.either'
Succeeded
*)

Here I wrapped the download into another async to capture the Uri construction exception.