How can AsynchronousFileChannel read large file?

Question

  Path file = Paths.get("c:/large.log");
  AsynchronousFileChannel channel = AsynchronousFileChannel.open(file);
  final ByteBuffer buffer = ByteBuffer.allocate(1000);
  channel.read(buffer, 0, buffer,
      new CompletionHandler<Integer, ByteBuffer>() {
        public void completed(Integer result, ByteBuffer attachment) {
          System.out.println(new String(buffer.array()));
        }
  }); 

In this way, I can read the first 1000 byte from large.log. How can I read following log If I don't want to allocate bigger byte array like ByteBuffer.allocate(1000*1000). Because I think this will lead to OutOfMemory.

Could someone give me the sample code? Thanks.

ps:I can loop read the large file with JIO because I can check the return value of java.io.BufferedReader.read(). But I don't know how to do with NIO2.

Answer 1

Here's a hack that works.

A couple of things that you'll want to note:

1. I've just used your buffer.array() for the output. I had to use buffer.clear() to reset the position so that the asynchronous read will see that there are 1000 spare bytes, but this doesn't clear the existing data out of the array. As a result when you're at the end of the file, if you read fewer than 1000 bytes it prints the whole buffer: however much you just read, plus the remaining 1000 bytes of whatever was last in the end of the buffer. In real life you'd want to do something about that (perhaps with result or with the position of the buffer.
2. For reasons I couldn't figure out buffer which is a class variable is fine within the completed method, but channel which is also a class variable is null. I haven't yet figured out why that would be. So I changed it so it passes channel as the attachment rather than buffer. Still makes no sense to me.
3. The asynchronous read thread isn't important enough to keep the jvm running. So I've simply put a read at the end of the main method. Press Enter to exit.
4. Class variable pos maintains the position in the file you're reading from.
5. The magic happens when you initiate another asynchronous read during the complete method. This is why I discarded the anonymous class and implemented the interface itself.
6. You'll want to switch the path back to yours.

Have fun.

import java.nio.*;
import java.nio.channels.*;
import java.nio.file.*;
import java.io.IOException;

public class TryNio implements CompletionHandler<Integer, AsynchronousFileChannel> {

       // need to keep track of the next position.
        int pos = 0;
        AsynchronousFileChannel channel =  null;
        ByteBuffer buffer = null;

        public void completed(Integer result, AsynchronousFileChannel attachment) {
                 // if result is -1 means nothing was read.
                if (result != -1) {
                        pos += result;  // don't read the same text again.
                                        // your output command.
                        System.out.println(new String(buffer.array()));

                        buffer.clear();  // reset the buffer so you can read more.
                }
                        // initiate another asynchronous read, with this.
                attachment.read(buffer, pos , attachment, this );


        }
        public void failed(Throwable exc,
                        AsynchronousFileChannel attachment) {
                System.err.println ("Error!");
                exc.printStackTrace();
        }

        public void doit() {
                Path file = Paths.get("/var/log/syslog");
                AsynchronousFileChannel channel =  null;
                try {
                        channel = AsynchronousFileChannel.open(file);
                } catch (IOException e) {
                        System.err.println ("Could not open file: " + file.toString());
                        System.exit(1); // yeah.  heh.
                }
                buffer = ByteBuffer.allocate(1000);

                 // start off the asynch read. 
                channel.read(buffer, pos , channel, this );
                // this method now exits, thread returns to main and waits for user input.
        }

        public static void main (String [] args) {
                TryNio tn = new TryNio();
                tn.doit();
             // wait fur user to press a key otherwise java exits because the 
             // asynch thread isn't important enough to keep it running.
                try { System.in.read(); } catch (IOException e) { }
        }
}

Answer 2

The GregHNZ solution is great and since I have to use this kind of code several times in different projects, I ended up putting it in an auxiliary library RxIo which I published in Maven Central Repository and is also available at RxIo github repository. With RxIo you can use the RxIo utility class to read all bytes of a file like:

AsyncFiles
    .readAllBytes(Paths.get("input.txt"))
    .thenApply(bytes -> { /*... use bytes... */});

The readAllBytes(Path file) allocates a ByteBuffer with a default size of 262144, but you can specify a different value using the readAllBytes(Path file, int bufferSize).

You can see other use cases in unit tests folder.