How can I buffer my Java input/output/file streams properly?

Question

I'm writing an application that needs to send a file over the network. I've only been taught how to use standard java.net and java.io classes so far (in my first year of college) so I have no experience with java.nio and netty and all those nice things. I've got a working server/client set up using Socket and ServerSocket classes along with BufferedInput/OutputStreams and BufferedFile streams, as follows:

The server:

    public class FiletestServer {

    static ServerSocket server;
    static BufferedInputStream in;
    static BufferedOutputStream out;

    public static void main(String[] args) throws Exception {
    server = new ServerSocket(12354);
    System.out.println("Waiting for client...");
    Socket s = server.accept();
    in = new BufferedInputStream(s.getInputStream(), 8192);
    out = new BufferedOutputStream(s.getOutputStream(), 8192);
    File f = new File("test.avi");
    BufferedInputStream fin = new BufferedInputStream(new FileInputStream(f), 8192);

    System.out.println("Sending to client...");
    byte[] b = new byte[8192];
    while (fin.read(b) != -1) {
        out.write(b);
    }
    fin.close();
    out.close();
    in.close();
    s.close();
    server.close();
    System.out.println("done!");
    }
}

And the client:

public class FiletestClient {

    public static void main(String[] args) throws Exception {
    System.out.println("Connecting to server...");
    Socket s;
    if (args.length < 1) {
        s = new Socket("", 12354);
    } else {
        s = new Socket(args[0], 12354);
    }
    System.out.println("Connected.");
    BufferedInputStream in = new BufferedInputStream(s.getInputStream(), 8192);
    BufferedOutputStream out = new BufferedOutputStream(s.getOutputStream(), 8192);
    File f = new File("test.avi");
    System.out.println("Receiving...");
    FileOutputStream fout = new FileOutputStream(f);
    byte[] b = new byte[8192];
    while (in.read(b) != -1) {
        fout.write(b);
    }
    fout.close();
    in.close();
    out.close();
    s.close();
    System.out.println("Done!");
    }
}

At first I was using no buffering, and writing each int from in.read(). That got me about 200kb/s transfer according to my network monitor gadget on windows 7. I then changed it as above but used 4096 byte buffers and got the same speed, but the file received was usually a couple kilobytes bigger than the source file, and that is what my problem is. I changed the buffer size to 8192 and I now get about 3.7-4.5mb/sec transfer over wireless to my laptop, which is plenty fast enough for now, but I still have the problem of the file getting slightly bigger (which would cause it to fail an md5/sha hash test) when it is received.

So my question is what is the proper way of buffering to get decent speeds and end up with exactly the same file on the other side? Getting it to go a bit faster would be nice too but I'm happy with the speed for now. I'm assuming a bigger buffer is better up to a point, I just need to find what that point is.

Answer 1

You are ignoring the size of data actually read.

while (in.read(b) != -1) {
    fout.write(b);
}

will always write 8192 bytes even if only one byte is read. Instead I suggest using

for(int len; ((len = in.read(b)) > 0;)
   fout.write(b, 0, len);

Your buffers are the same size as your byte[] so they are not really doing anything at the moment.

The MTU for most networks is around 1500 bytes and you get a performance improvement on slower networks (up to 1 GB) up to 2 KB. 8 KB as fine as well. Larger than that is unlikely to help.