how to make curl accept secure cookies overt http connection

Question

I am using curl to connect to an http server which sends back a secure flagged cookie, and I found out that curl doesn't handle such cookies (secure cookies received over http connection), in other words : even using -c cookieFile switch, such cookies are not saved.

A workaround is to use -D switch to save all headers then manually (externally to curl) read the cookie from the file and set it in the curl command to send it back to server.

I want to know if there is a possibility (may be I am missing some curl options) to make curl support such cookies ? I tried to look into curl manual but nothing useful to my use case.

Thanks in advance,

Answer 1

TL;DR: With recent versions of cURL it is no longer possible to save cookies with the secure attribute in conjunction with cookie related switches.

According the documentation cURL removed the ability to save cookies with the secure attribute in order to satisfy the RFC draft draft-ietf-httpbis-cookie-alone-01. This RFC draft mandates that secure cookies are only supposed to be handled, saved or overwritten by an HTTP client if said cookie was transferred over HTTPS.

I just stumbled over the exactly the same problem, so I can offer two alternatives:

1. use a cURL version before the feature was implemented
   • cURL < v7.46.0
   • see respective Github issue which led to this behaviour
2. dump the headers manually with curl -i or curl -D and extract the cookies
   • example to save all secure cookies and save them in a file cookies.txt

    curl -i http://server.com | grep "Set-Cookie: " | sed 's/Set-Cookie: //g' > cookies.txt
   

Now, a cookie jar would be useless if you would not use the cookies inside. Especially regarding the second alternative, it may be necessary to remove the Secure attribute in order to make cURL send the saved cookies back to the web server.

Answer 2

Another alternative to add on to Philipp's:

Modify the cookiejar file to allow the https-only cookies to be used in http.

curl saves the cookies in the Netscape cookie file format, which is a TAB-separated file with the following columns:

Index	Type	Example	Description
0	string	example.com	the domain name
1	boolean	FALSE	include subdomains
2	string	/foobar/	path
3	boolean	TRUE	send/receive over HTTPS only
4	number	1462299217	expires at - seconds since Jan 1st 1970, or 0
5	string	person	name of the cookie
6	string	daniel	value of the cookie

If the cookies you're getting back from your first request are marked as HTTPS-only (i.e. TRUE for column index 3), curl won't set those cookies on HTTP requests using the same cookiejar file (Philipp pointed this out).

To get around this, you could modify the cookiejar file to have FALSE instead of TRUE for HTTPS-only. You can use sed to do this:

sed -i 's/\(.*\t.*\t.*\t\)TRUE/\1FALSE/' cookieJarFile

Or if you're on macOS, you might need an extra argument:

sed -i '' 's/\(.*\t.*\t.*\t\)TRUE/\1FALSE/' cookieJarFile

Of course, you should probably not do this, since the cookies were marked HTTPS-only for a reason.