OKHTTP 3 Tracking Multipart upload progress

Question

How can I track progress of upload in OkHttp 3 I can find answers for v2 but not v3, like this

A sample Multipart request from OkHttp recipes

private static final String IMGUR_CLIENT_ID = "...";
private static final MediaType MEDIA_TYPE_PNG = MediaType.parse("image/png");

private final OkHttpClient client = new OkHttpClient();

public void run() throws Exception {
    // Use the imgur image upload API as documented at https://api.imgur.com/endpoints/image
    RequestBody requestBody = new MultipartBody.Builder()
            .setType(MultipartBody.FORM)
            .addFormDataPart("title", "Square Logo")
            .addFormDataPart("image", "logo-square.png",
                    RequestBody.create(MEDIA_TYPE_PNG, new File("website/static/logo-square.png")))
            .build();

    Request request = new Request.Builder()
            .header("Authorization", "Client-ID " + IMGUR_CLIENT_ID)
            .url("https://api.imgur.com/3/image")
            .post(requestBody)
            .build();

    Response response = client.newCall(request).execute();
    if (!response.isSuccessful()) throw new IOException("Unexpected code " + response);

    System.out.println(response.body().string());
}

Answer 1

You can decorate your OkHttp request body to count the number of bytes written when writing it; in order to accomplish this task, wrap your MultiPart RequestBody in this RequestBody with an instance of Listener and Voila!

public class ProgressRequestBody extends RequestBody {

    protected RequestBody mDelegate;
    protected Listener mListener;
    protected CountingSink mCountingSink;

    public ProgressRequestBody(RequestBody delegate, Listener listener) {
        mDelegate = delegate;
        mListener = listener;
    }

    @Override
    public MediaType contentType() {
        return mDelegate.contentType();
    }

    @Override
    public long contentLength() {
        try {
            return mDelegate.contentLength();
        } catch (IOException e) {
            e.printStackTrace();
        }
        return -1;
    }

    @Override
    public void writeTo(BufferedSink sink) throws IOException {
        mCountingSink = new CountingSink(sink);
        BufferedSink bufferedSink = Okio.buffer(mCountingSink);
        mDelegate.writeTo(bufferedSink);
        bufferedSink.flush();
    }

    protected final class CountingSink extends ForwardingSink {
        private long bytesWritten = 0;
        public CountingSink(Sink delegate) {
            super(delegate);
        }
        @Override
        public void write(Buffer source, long byteCount) throws IOException {
            super.write(source, byteCount);
            bytesWritten += byteCount;
            mListener.onProgress((int) (100F * bytesWritten / contentLength()));
        }
    }

    public interface Listener {
        void onProgress(int progress);
    }
}

Check this link for more.

Answer 2

I was unable to get any of the answers to work for me. The issue was that the progress would run to 100% before the image was uploaded hinting that some buffer was getting filled prior to the data being sent over the wire. After some research, I found this was indeed the case and that buffer was the Socket send buffer. Providing a SocketFactory to the OkHttpClient finally worked. My Kotlin code is as follows...

First, As others, I have a CountingRequestBody which is used to wrap the MultipartBody.

class CountingRequestBody(var delegate: RequestBody, private var listener: (max: Long, value: Long) -> Unit): RequestBody() {

    override fun contentType(): MediaType? {
        return delegate.contentType()
    }

    override fun contentLength(): Long {
        try {
            return delegate.contentLength()
        } catch (e: IOException) {
            e.printStackTrace()
        }
        return -1
    }

    override fun writeTo(sink: BufferedSink) {
        val countingSink = CountingSink(sink)
        val bufferedSink = Okio.buffer(countingSink)
        delegate.writeTo(bufferedSink)
        bufferedSink.flush()
    }

    inner class CountingSink(delegate: Sink): ForwardingSink(delegate) {
        private var bytesWritten: Long = 0

        override fun write(source: Buffer, byteCount: Long) {
            super.write(source, byteCount)
            bytesWritten += byteCount
            listener(contentLength(), bytesWritten)
        }
    }
}

I'm using this in Retrofit2. A general usage would be something like this:

val builder = MultipartBody.Builder()
// Add stuff to the MultipartBody via the Builder

val body = CountingRequestBody(builder.build()) { max, value ->
      // Progress your progress, or send it somewhere else.
}

At this point, I was getting progress, but I would see 100% and then a long wait while the data was uploading. The key was that the socket was, by default in my setup, configured to buffer 3145728 bytes of send data. Well, my images were just under that and the progress was showing the progress of filling that socket send buffer. To mitigate that, create a SocketFactory for the OkHttpClient.

class ProgressFriendlySocketFactory(private val sendBufferSize: Int = DEFAULT_BUFFER_SIZE) : SocketFactory() {

    override fun createSocket(): Socket {
        return setSendBufferSize(Socket())
    }

    override fun createSocket(host: String, port: Int): Socket {
        return setSendBufferSize(Socket(host, port))
    }

    override fun createSocket(host: String, port: Int, localHost: InetAddress, localPort: Int): Socket {
        return setSendBufferSize(Socket(host, port, localHost, localPort))
    }

    override fun createSocket(host: InetAddress, port: Int): Socket {
        return setSendBufferSize(Socket(host, port))
    }

    override fun createSocket(address: InetAddress, port: Int, localAddress: InetAddress, localPort: Int): Socket {
        return setSendBufferSize(Socket(address, port, localAddress, localPort))
    }

    private fun setSendBufferSize(socket: Socket): Socket {
        socket.sendBufferSize = sendBufferSize
        return socket
    }

    companion object {
        const val DEFAULT_BUFFER_SIZE = 2048
    }
}

And during config, set it.

val clientBuilder = OkHttpClient.Builder()
    .socketFactory(ProgressFriendlySocketFactory())

As others have mentioned, Logging the request body may effect this and cause the data to be read more than once. Either don't log the body, or what I do is turn it off for CountingRequestBody. To do so, I wrote my own HttpLoggingInterceptor and it solves this and other issues (like logging MultipartBody). But that's beyond the scope fo this question.

if(requestBody is CountingRequestBody) {
  // don't log the body in production
}

The other issues was with MockWebServer. I have a flavor that uses MockWebServer and json files so my app can run without a network so I can test without that burden. For this code to work, the Dispatcher needs to read the body data. I created this Dispatcher to do just that. Then it forwards the dispatch to another Dispatcher, such as the default QueueDispatcher.

class BodyReadingDispatcher(val child: Dispatcher): Dispatcher() {

    override fun dispatch(request: RecordedRequest?): MockResponse {
        val body = request?.body
        if(body != null) {
            val sink = ByteArray(1024)
            while(body.read(sink) >= 0) {
                Thread.sleep(50) // change this time to work for you
            }
        }
        val response = child.dispatch(request)
        return response
    }
}

You can use this in the MockWebServer as:

var server = MockWebServer()
server.setDispatcher(BodyReadingDispatcher(QueueDispatcher()))

This is all working code in my project. I did pull it out of illustration purposes. If it does not work for you out of the box, I apologize.