Why speed of downloading files using Java is so low

Question

I wrote simple Java Downloader and I have some problems with speed.

At first, speed is OK - just like when I use my browser to download this file. But after a while speed decreases a lot and change every two seconds - from 42kb/s to 64kb/s and from 64kb/s to 42kb/s.

My code:

InputStream is = null;
FileOutputStream os = null;
os = new FileOutputStream(...);
URL u = new URL(...);
URLConnection uc = u.openConnection();
is = uc.getInputStream();
final byte[] buf = new byte[1024];
for(int count = is.read(buf);count != -1;count = is.read(buf)) {
    os.write(buf, 0, count);
}

What should I do to maximalise speed of download?

UPDATE

Sizes of files are various from 1 to about 100MB. I increased the buffer to 65536 to it is the same.

About measuring : I check every 3 second how many bytes was written, and then divide it by 3 and by 1024 - it gives me kb / s

Answer 1

To increase speed, up to the limit of your bandwidth and server capacity, an application should use several connections (more than only one) with multi-threaded code: each thread creates its own connection and queries for parts of the file.

An example of such an application is IBM download director which often uses three HTTP connections to retrieve large files. Most FTP clients are also able to work with multiple connections to increase throughput. In Java, Apache HttpClient may be used to write such a multi-threaded application.

You have not detailed the protocol used in your URL. If HTTP, a HEAD request returns the file length and GET with chunking support is used to query for file parts.

Probably you can get better performance even with a single connection if you directly use HttpURLConnection and set value for ChunkedStreamingMode.

If still unsatisfied, please provide additional details:

• what your "But after a while" means, do you download many files in sequence ?
• what is the protocol ? do you use a specific URLStreamHandler ?
• have you checked your JVM memory and garbage collection usage ?
• your workstation may be busy doing something else: CPU used, network bandwidth used by something else, anti-virus slowing down disk access ?
• do you pass through a proxy: some HTTP proxy may reduce bandwidth after some minutes on the same connection...