How to correctly send binary data over HTTPS POST?

Question

I send binary data from client (Debian 6.0.3) to server (Windows Server 2003). To bypass most firewalls I use HTTPS POST. Client and server are implemented using Boost.Asio and OpenSSL. First I implemented the simplest possible version and it worked fine.

HTTP Header:

POST / HTTP/1.1
User-Agent: my custom client v.1

[binary data]

([binary data] is not base64 encoded if this matters)

Then, on another client machine it failed (connected to the same server machine). The behavior is not stable. Connection always is established fine (port 443). Most time I pass SSL handshake fine but server receive no data (almost no data, sometimes a packet or two are actually received). Sometimes I receive SSL handshake error "short read". Sometimes I receive invalid data.

Client connects to server, handshakes, sends HTTP POST header and then infinitely sends binary data until something wrong hapenned. For test I use custom generated SSL certificate.

Server code:

namespace ssl = boost::asio::ssl;
ssl::context context(io_service, ssl::context::sslv23);
context.set_options(ssl::context::default_workarounds | ssl::context::no_sslv2);
context.use_certificate_chain_file("server.pem");
context.use_private_key_file("server.pem", boost::asio::ssl::context::pem);

ssl::stream<tcp::socket> socket(io_service, context);

// standard connection accepting

socket.async_handshake(ssl::stream_base::server, ...);
...
boost::asio::async_read_until(socket, POST_header, "\r\n\r\n", ...);
...

Client code:

ssl::context context(io_service, ssl::context::sslv23);
context.load_verify_file("server.crt");
socket.reset(new ssl::stream<tcp::socket>(io_service, context));
socket->set_verify_mode(ssl::verify_none);

// standard connection

socket.async_handshake(ssl::stream_base::client, ...);
...

(error handling is omitted along with not relevant code)

As you can see, it's the simplest possible SSL connection. What is wrong? Can the reason be a firewall?

I tried simple TCP w/o SSL over the same 443 port, this works fine.

EDIT:

Tried adding "Content-Type: application/octet-stream", doesn't help.

EDIT 2:

Usually I receive HTTP POST header fine. Then I send data chunks as chunk-size(4 bytes)chunk(chunk-size bytes).... Server receives chunk-size fine, but then nothing. Client doesn't notify server problems (no errors) and continue to send data. Sometimes server can receive chunk or two, sometimes it receives invalid chunk-size, but most time just nothing.

EDIT 3:

Compared captured traffic on client and server, didn't find any differences.

Solution

I was misled from the start with this problem. Narrowed it down to surprising details:

Sending over SSL socket fails if I use Boost.Asio multi-buffers in Boost v.1.48 (the most recent one at this moment). Example:

// data to send, protocol is [packet size: 4 bytes][packet: packet_size bytes]
std::vector<char> packet = ...;
uint32_t packet_size = packet.size();
// prepare buffers
boost::array<boost::asio::const_buffer, 2> bufs = {{boost::asio::buffer(&packet_size, sizeof(packet_size)), boost::asio::buffer(packet)}};
// send multi buffers by single call
boost::asio::async_write(socket, bufs, ...);

Sending separately packet_size and packet in this example works around the problem. I'm far from calling any suspicious behavior as a bug, especially if it's related with Boost libraries. But this one really looks like a bug. Tried on Boost v.1.47 - works fine. Tried with usual TCP socket (not SSL one) - works fine. The same on both Linux and Windows.

I'm going to find any reports about this problem in Asio mailing list and will report it if nothing found.

Answer 1

If you don't have to operate in front of web server, you don't have to use HTTPS protocol. From the firewall point of view HTTPS looks like yet another SSL connection and it has no idea what going through. So if the only thing you need is just to pass the data - not to actual web server, use just SSL connection over 443 port.

So just troubleshoot your SSL connection the problem has nothing to do with HTTP.

---

If you want to use HTTP web server and not custom client:

Two points:

1. You need to specify Content-Length.
2. If you are using HTTP/1.1 you need to specify Host header.

The simplest would be

POST /url HTTP/1.0
User-Agent: my custom client v.1
Content-Type: application/octet-stream
Content-Length: NNN

Actual Content

Or for HTTP/1.1

POST /url HTTP/1.1
Host: www.example.com
User-Agent: my custom client v.1
Content-Type: application/octet-stream
Content-Length: NNN

Actual Content

Note: you can't send infinite data. HTTP protocol requires fixed content-lenght and most of the time web servers would load the data first before passing it to the backend.

So you will have to transfer data by chunks.

Answer 2

I was misled from the start with this problem. Narrowed it down to surprising details:

Sending over SSL socket fails if I use Boost.Asio multi-buffers in Boost v.1.48 (the most recent one at this moment). Example:

// data to send, protocol is [packet size: 4 bytes][packet: packet_size bytes]
std::vector<char> packet = ...;
uint32_t packet_size = packet.size();
// prepare buffers
boost::array<boost::asio::const_buffer, 2> bufs = {{boost::asio::buffer(&packet_size, sizeof(packet_size)), boost::asio::buffer(packet)}};
// send multi buffers by single call
boost::asio::async_write(socket, bufs, ...);

Sending separately packet_size and packet in this example works around the problem. I'm far from calling any suspicious behavior as a bug, especially if it's related with Boost libraries. But this one really looks like a bug. Tried on Boost v.1.47 - works fine. Tried with usual TCP socket (not SSL one) - works fine. The same on both Linux and Windows.

I'm going to find any reports about this problem in Asio mailing list and will report it if nothing found.

Answer 3

(EDIT: I had originally deleted this because I had realised it wasn't using HTTP really. Following a comment where you think you might have a MITM proxy and should use proper HTTP, I'm undeleting/editing.)

POST / HTTP/1.1
User-Agent: my custom client v.1

[binary data]

Whether it's binary data or not, in plain HTTP or with SSL/TLS, you'll need a Content-Length header or to use chunked transfer encoding. This this section of the HTTP spec. A Content-Type header would be useful too.

Chunked transfer encoding is for when you don't necessarily know the length of the stream. (You always need some form of delimiters when sending data, if only to detect reliably when it ends.)

This being said, you should be able to find out whether you're behind a MITM proxy that looks into the application layer on top of SSL/TLS if you get a certificate that's not your servers. If you do still get a successful handshake with your won server cert, there isn't such a proxy. Even an HTTP proxy would use CONNECT and relay everything, without altering the SSL/TLS connection (and thus without altering your original pseudo-HTTP on top).