Http-Conduit frequent connection failures

Question

I am writing application which will download some files by HTTP. Up to some point I was using following code snippet to download page body:

import network.HTTP
simpleHTTP (getRequest "http://www.haskell.org/") >>= getResponseBody

It was working fine but it could not establish connection by HTTPS protocol. So to fix this I have switched to HTTP-Conduit and now I am using following code:

simpleHttp' :: Manager -> String -> IO (C.Response LBS.ByteString)
simpleHttp' manager url = do
     request <- parseUrl url
     runResourceT $ httpLbs request manager

It can connect to HTTPS but new frustrating problem appeared. About every fifth connection fails with exception:

getpics.hs: FailedConnectionException "i.imgur.com" 80

I am convinced that this is HTTP-Conduit problem because network.HTTP was working fine on same set of pages (excluding https pages).

Have anybody met such problem and know solution or better (and simple because this is simple task which should not take more than few lines of code) alternative to Conduit library?

Answer 1

One simple alternative would be to use the curl package. It supports HTTP, HTTPS and a bunch of other alternative protocols, as well as many options to customize its behavior. The price is introducing an external dependency on libcurl, required to build the package.

Example:

import Network.Curl

main :: IO ()
main = do
  let addr = "https://google.com/" 
  -- Explicit type annotation is required for calls to curlGetresponse_.
  -- Use ByteString instead of String for higher performance:
  r <- curlGetResponse_ addr [] :: IO (CurlResponse_ [(String,String)] String)

  print $ respHeaders r
  putStr $ respBody r

---

Update: I tried to replicate your problem, but everything works for me. Could you post a Short, Self Contained, Compilable, Example that demonstrates the problem? My code:

import Control.Monad
import qualified Data.Conduit as C
import qualified Data.ByteString.Lazy as LBS
import Network.HTTP.Conduit

simpleHttp'' :: String -> Manager -> C.ResourceT IO (Response LBS.ByteString)
simpleHttp'' url manager = do
     request <- parseUrl url
     httpLbs request manager

main :: IO ()
main = do
  let url = "http://i.imgur.com/"
      count = 100
  rs <- withManager $ \m -> replicateM count (simpleHttp'' url m)
  mapM_ (print . responseStatus) $ rs