Large file transfer over java socket [duplicate]

Question

I have written a small client-server code for transferring small file. It uses Data output stream and readFully() method of data input stream. This code does not work for larger files for obvious reasons. I was thinking of fragmenting large files into smaller chunks of 1Kb each before sending them to client. But I can't think of any solution (like how to write multiple chunks on data output stream with correct offset and how to reassemble them at receiving end. Can anyone provide a workaround? It would be very helpful if you could modify my code:

Sender (Server):

public void sendFileDOS() throws FileNotFoundException {
    runOnUiThread( new Runnable() {
          @Override
          public void run() {
              registerLog("Sending. . . Please wait. . .");
          }
        });
    final long startTime = System.currentTimeMillis();
    final File myFile= new File(filePath); //sdcard/DCIM.JPG
    byte[] mybytearray = new byte[(int) myFile.length()];
    FileInputStream fis = new FileInputStream(myFile);  
    BufferedInputStream bis = new BufferedInputStream(fis);
    DataInputStream dis = new DataInputStream(bis);
    try {
        dis.readFully(mybytearray, 0, mybytearray.length);
        OutputStream os = socket.getOutputStream();
        //Sending file name and file size to the client  
        DataOutputStream dos = new DataOutputStream(os);     
        dos.writeUTF(myFile.getName());     
        dos.writeLong(mybytearray.length);     
        int i = 0;
        final ProgressBar myProgBar=(ProgressBar)findViewById(R.id.progress_bar);
        while (i<100) {
            dos.write(mybytearray, i*(mybytearray.length/100), mybytearray.length/100);
            final int c=i;
            runOnUiThread( new Runnable() {
                  @Override
                  public void run() {
                      myProgBar.setVisibility(View.VISIBLE);
                      registerLog("Completed: "+c+"%");
                      myProgBar.setProgress(c);
                      if (c==99)
                          myProgBar.setVisibility(View.INVISIBLE);
                  }
                });
            i++;
        }    
        dos.flush();

    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
    runOnUiThread( new Runnable() {
          @Override
          public void run() {
              long estimatedTime = (System.currentTimeMillis() - startTime)/1000;
              registerLog("File successfully sent");
              registerLog("File size: "+myFile.length()/1000+" KBytes");
              registerLog("Elapsed time: "+estimatedTime+" sec. (approx)");
              registerLog("Server stopped. Please restart for another session.");
              final Button startServerButton=(Button)findViewById(R.id.button1);
              startServerButton.setText("Restart file server");
          }
        });
}

Receiver (Client):

public class myFileClient {
final static String servAdd="10.141.21.145";
static String filename=null;
static Socket socket = null;
static Boolean flag=true;

/**
 * @param args
 */
public static void main(String[] args) throws IOException {
    // TODO Auto-generated method stub
    initializeClient();
    receiveDOS();      
}
public static void initializeClient () throws IOException {
    InetAddress serverIP=InetAddress.getByName(servAdd);
    socket=new Socket(serverIP, 4444);
}
public static void receiveDOS() {
    int bytesRead;
    InputStream in;
    int bufferSize=0;

    try {
        bufferSize=socket.getReceiveBufferSize();
        in=socket.getInputStream();
        DataInputStream clientData = new DataInputStream(in);
        String fileName = clientData.readUTF();
        System.out.println(fileName);
        OutputStream output = new FileOutputStream("//home//evinish//Documents//Android//Received files//"+ fileName);
        long size = clientData.readLong();
        byte[] buffer = new byte[bufferSize];
        while (size > 0
                && (bytesRead = clientData.read(buffer, 0,
                        (int) Math.min(buffer.length, size))) != -1) {
            output.write(buffer, 0, bytesRead);
            size -= bytesRead;
        }

    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
}

Please help! Thanks in advance! :)

Answer 1

You're right, this is a poor way to do it. It wastes both memory and time; it assumes the file size is 32 bits; it assumes the entire file fits into memory; it assumes the entire file is read in one read; and it doesn't send anything until the entire file has been read.

The canonical way to copy a stream in Java is this:

while ((count = in.read(buffer)) > 0)
{
  out.write(buffer, 0, count);
}

It will work with any size buffer you like and therefore with any size file you can come up with. Use the same code at both ends, although you don't have to use the same size buffer at both ends. As you're copying over a network you might think that 1k or 1.5k is the best size, but that overlooks the presence of the socket send and receive buffers in the kernel. When you take them into account it is probably better to use 8k or more.

Answer 2

I finally solved the problem. Here is my modified source code for server and client. Hope this would help other people too! :) Server Side code snippet (sender):

final File myFile= new File(filePath); //sdcard/DCIM.JPG
    byte[] mybytearray = new byte[8192];
    FileInputStream fis = new FileInputStream(myFile);  
    BufferedInputStream bis = new BufferedInputStream(fis);
    DataInputStream dis = new DataInputStream(bis);
    OutputStream os;
    try {
        os = socket.getOutputStream();
        DataOutputStream dos = new DataOutputStream(os);
        dos.writeUTF(myFile.getName());     
        dos.writeLong(mybytearray.length);
        int read;
        while((read = dis.read(mybytearray)) != -1){
            dos.write(mybytearray, 0, read);
        }

    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }

Client side code snippet (Receiver):

int bytesRead;
    InputStream in;
    int bufferSize=0;

    try {
        bufferSize=socket.getReceiveBufferSize();
        in=socket.getInputStream();
        DataInputStream clientData = new DataInputStream(in);
        String fileName = clientData.readUTF();
        System.out.println(fileName);
        OutputStream output = new FileOutputStream("//home//evinish//Documents//Android//Received files//"+ fileName);
        byte[] buffer = new byte[bufferSize];
        int read;
        while((read = clientData.read(buffer)) != -1){
            output.write(buffer, 0, read);
        }

    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }

A bit faster way of writing to output stream:

long acc=0;
long N=myFile.length();
 while(acc<N){
            noofbytes=dis.read(mybytearray, 0, 16384);
            dos.write(mybytearray, 0, noofbytes);
            acc=acc+noofbytes; } dos.flush();

I saved around 7 seconds while transferring a video file of 72MB.