How to programmatically limit the download speed?

Question

I use the following code to limit the download speed of a file in java:

package org;

import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URL;

class MainClass {

    public static void main(String[] args) {
        download("https://speed.hetzner.de/100MB.bin");
    }

    public static void download(String link) {
        try {
            URL url = new URL(link);
            HttpURLConnection con = (HttpURLConnection) url.openConnection();
            con.setConnectTimeout(5000);
            con.setReadTimeout(5000);
            InputStream is = con.getInputStream();
            CustomInputStream inputStream = new CustomInputStream(is);
            byte[] buffer = new byte[2024];
            int len;
            while ((len = inputStream.read(buffer)) != -1) {
                System.out.println("downloaded : " + len);
                //save file
            }
        } catch (IOException e) {
            e.printStackTrace();
        }

    }

    public static class CustomInputStream extends InputStream {

        private static final int MAX_SPEED = 8 * 1024;
        private final long ONE_SECOND = 1000;
        private long downloadedWhithinOneSecond = 0L;
        private long lastTime = System.currentTimeMillis();

        private InputStream inputStream;

        public CustomInputStream(InputStream inputStream) {
            this.inputStream = inputStream;
            lastTime = System.currentTimeMillis();
        }

        @Override
        public int read() throws IOException {
            long currentTime;
            if (downloadedWhithinOneSecond >= MAX_SPEED
                    && (((currentTime = System.currentTimeMillis()) - lastTime) < ONE_SECOND)) {
                try {
                    Thread.sleep(ONE_SECOND - (currentTime - lastTime));
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
                downloadedWhithinOneSecond = 0;
                lastTime = System.currentTimeMillis();
            }
            int res = inputStream.read();
            if (res >= 0) {
                downloadedWhithinOneSecond++;
            }
            return res;
        }

        @Override
        public int available() throws IOException {
            return inputStream.available();
        }

        @Override
        public void close() throws IOException {
            inputStream.close();
        }
    }

}

The download speed is successfully limited, but a new problem arises. When the download is in progress, and I disconnect from the internet, the download does not end and continues for a while. When i disconnect the internet connection, it takes more than 10 seconds to throw a java.net.SocketTimeoutException exception. I do not really understand what happens in the background.

Why does this problem arise?

Answer 1

Your rate limit doesn't actually work like you think it does, because the data is not actually sent byte-per-byte, but in packets. These packets are buffered, and what you observe (download continues without connection) is just your stream reading the buffer. Once it reaches the end of your buffer, it waits 5 seconds before the timeout is thrown (because that is what you configured).

You set the rate to 8 kB/s, and the normal packet size is normally around 1 kB and can go up to 64 kB, so there would be 8 seconds where you are still reading the same packet. Additionally it is possible that multiple packets were already sent and buffered. There exists also a receive buffer, this buffer can be as small as 8 - 32 kB up to several MB. So really you are just reading from the buffer.

[EDIT]

Just to clarify, you are doing the right thing. On average, the rate will be limited to what you specify. The server will send a bunch of data, then wait until the client has emptied its buffer enough to receive more data.

Answer 2

You apparently want to limit download speed on the client side, and you also want the client to respond immediately to the connection being closed.

AFAIK, this is not possible ... without some compromises.

The problem is that the only way that the client application can detect that the connection is closed is by performing a read operation. That read is going to deliver data. But if you have already reached your limit for the current period, then that read will push you over the limit.

Here are a couple of ideas:

• If you "integrate" the download rate over a short period (e.g. 1kbytes every second versus 10kbytes every 10 seconds) then you can reduce the length of time for the sleep calls.

• When you are close to your target download rate, you could fall back to doing tiny (e.g. 1 byte) reads and small sleeps.

Unfortunately, both of these will be inefficient on the client side (more syscalls), but this is the cost you must pay if you want your application to detect connection closure quickly.

---

In a comment you said:

I'd expect the connection to be reset as soon as the internet connection is disabled.

I don't think so. Normally, the client-side protocol stack will deliver any outstanding data received from the network before telling the application code that the connection it is reading has been closed.