Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

OkHttp buffers ~800kb before uploading data despite flush()

Im trying to publish upload progress for images, uploading them using okhttp as my client and mimecraft to package up the multipart file.

I have added logs to write the bytecount when data gets written to the socket (in chunks of 4kb if i can tell correctly) and then uploaded.

The problem is that, while i call flush on the outputstream every time a chunk of data is written, nothing seems to get uploaded until ~800kb has been written to the outputstream. Once it hits that point, it seems to upload ~100kb, and the app then writes another 100kb to the outputstream (like its freed up space and can write a bit more) and continues on.

The result of this is that (over a 1.2mb file) the first 800kb gets written/reported almost instantly, and then it starts uploading (tracked through charles network software) and will then start a read/write/upload of 100kb over the next few seconds, then once ive written the last bytes to the outputstream, the app reports 100% of the upload is written. This sint true though as the network client is still uploading the last 800kb that still is in the network buffer, and it sits there for another 5-10 seconds uploading that, then finishes the request.

Has anyone had this experience, or knows if this is a common problem in okhttp?

Cheers

EDIT: if i upload a file less than 800kb, ive tested some 250kb and 500kb, and they all get written 100% to the stream instantly before being uploaded, but a 3mb image will still upload 800kb, and then tick away in roughly 100kb chunks, and the writing loop sits there writing another 100kb each time some gets uploaded.

like image 959
Glenn.nz Avatar asked May 12 '14 02:05

Glenn.nz


People also ask

What is okhttp used for?

OkHttp Overview OkHttp is an efficient HTTP & HTTP/2 client for Android and Java applications. It comes with advanced features such as connection pooling (if HTTP/2 isn’t available), transparent GZIP compression, and response caching to avoid the network completely for repeated requests.

How do I use timeouts in okhttp?

Use timeouts to fail a call when its peer is unreachable. Network failures can be due to client connectivity problems, server availability problems, or anything between. OkHttp supports connect, read, and write timeouts. In this example, we built our client with a readTimeout of 1 seconds, while the URL is served with 2 seconds of delay:

What happens to the internal buffer when a file is closed?

If there is data in the internal buffer when the file is closed, the operating system does not automatically write the contents of the buffer to the disk before closing the file. If the application does not force the operating system to write the buffer to disk before closing the file, the caching algorithm determines when the buffer is written.

How does flushfilebuffers work in Linux?

When an application writes to a file, the system usually buffers the data and writes the data to the disk on a regular basis. An application can force the operating system to write the contents of these data buffers to the disk by using the FlushFileBuffers function.


1 Answers

As the answer from @nmr said its because the send buffer's are too high as explained by the bug ticket here: https://github.com/square/okhttp/issues/1078

So I've created a socket factory that lets you override the send buffer size.

https://gist.github.com/slightfoot/00a26683ea68856ceb50e26c7d8a47d0

like image 148
Simon Avatar answered Sep 21 '22 12:09

Simon