I'm trying to scrape data from a site using HTTPS. I managed to make basic requests using Network.HTTP.Conduit successfully (posting credentials, etc.), but failed at extracting cookie information from the response headers (Set-Cookie). It looks like http-conduit has its own mechanism for dealing with cookies, which I failed to understand.
Network.HTTP.Conduit.Browser seems to deal with cookies automatically (which is fine by me), but I couldn't get it to work due to lacking documentation.
Could someone with more experience dealing with the http-conduit browser module show me how to:
To me it looks like the abstraction level of Network.HTTP.Conduit.Browser is more suited for my application compared to Network.HTTP.Conduit, so I would like to make the switch even if I could deal with cookies manually using the latter.
I've never used Browser, but I have used http-conduit. I read the source code to answer these questions, I apologize if I make any mistakes.
Do the same thing you're doing. When you've created the Manager
with the right managerCheckCerts
, pass that along to browse :: Manager -> BrowserAction a -> ResourceT IO a
.
makeRequest :: Request IO -> BrowserAction (Response (Source IO BS.ByteString))
takes a Request IO
; use urlEncodedBody
like before to create a POST request with parameters in the body and pass it to makeRequest
. Set redirectCount
to 0 to disable redirect following, I believe.
I believe you just need to use getCookieJar :: BrowserAction CookieJar
; the BrowserAction
comes from getBrowserState :: BrowserAction BrowserState
.
The way http-conduit manages cookies outside the Browser module is that it doesn't. Cookies are returned in the HTTP response; what you can do is parse the response and store the cookies in a cookie jar. That's actually all Browser really does.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With