Can Haskell's Control.Concurrent.Async.mapConcurrently have a limit?

Question

I'm attempting to run multiple downloads in parallel in Haskell, which I would normally just use the Control.Concurrent.Async.mapConcurrently function for. However, doing so opens ~3000 connections, which causes the web server to reject them all. Is it possible to accomplish the same task as mapConcurrently, but only have a limited number of connections open at a time (i.e. only 2 or 4 at a time)?

Answer 1

This is really easy to do using the Control.Concurrent.Spawn library:

import Control.Concurrent.Spawn

type URL      = String
type Response = String    

numMaxConcurrentThreads = 4

getURLs :: [URL] -> IO [Response]
getURLs urlList = do
   wrap <- pool numMaxConcurrentThreads
   parMapIO (wrap . fetchURL) urlList

fetchURL :: URL -> IO Response