I'd like to apply a regex efficiently to an entire file

Question

I have a complex regex, and I'd like to match it with the contents of an entire huge file. The main concern is efficiency, since the file is indeed very big and running out of memory is a distinct possibility.

Is there a way I can somehow "buffer" the contents while pumping it through a regex matcher?

Answer 1

Yes, Pattern.match() will take a CharSequence.

If your input is already in a charset which uses exactly 2 bytes to represent a character without any 'prologue', you need only:

ByteBuffer bb = ...; // acquire memory mapped byte buffer
CharBuffer cb = bb.asCharBuffer();  // get a char[] 'view' of the bytes

... and since CharBuffer implements CharSequence, you're done.

On the other hand, if you need to decode the bytes into some other charset, you'll have your work cut out, since CharBuffer is charset-agnostic, and CharsetDecorder.decode(ByteBuffer) internally allocates a new CharBuffer roughly the same size as the input bytes.

Whether or not you'll be able to get away with a smaller buffer depends a fair bit on your regex and what you want to do with the match results. But the basic approach would be to implement CharSequence and wrap the memory-mapped ByteBuffer, a smaller CharBuffer for 'working space', and a CharsetDecoder. You'll use Charset.decode(ByteBuffer,CharBuffer,boolean) to decode the bytes 'on demand', and hope that the general direction of the regex matcher is 'forward', and that the input you're interested in comes in fairly small chunks.

As a rough start:

class MyCharSequence implements CharSequence {

    public MyCharSequence(File file, Charset cs, int bufferSize) throws IOException {

        FileInputStream input = new FileInputStream(file);
        FileChannel channel = input.getChannel();
        this.fileLength = (int) channel.size();
        this.bytes = channel.map(FileChannel.MapMode.READ_ONLY, 0, fileLength);
        this.charBuffer = CharBuffer.allocate(bufferSize);
        this.decoder = cs.newDecoder();

    }

    public int length() {
        // ouch! have to decode the lot, even if you don't choose to keep it all handy
    }

    public char charAt(final int index) {
        while ( /* not yet decoded target char[] */ )  {
            this.decoder.decode(this.bytes, this.charBuffer, true);
        }
        // don't assume 2-bytes == a char unless that's true for your charset!
    }

    public CharSequence subSequence(final int start, final int end) {
        // this'll be fun, too
    }

    private long fileLength;
    private MappedByteBuffer bytes;
    private CharBuffer charBuffer;
    private CharsetDecoder decoder;

}

It might be instructive to wrap a fully-decoded CharBuffer in a much simpler CharSequence wrapper of your own, and log how the methods are actually called for your given input, when you run it with a big heap on your development box. That will give you an idea if this approach is going to work for your particular scenario.