Mime4j: DefaultMessageBuilder fails to parse mbox content

Question

I've downloaded mime4j 0.8.0 snapshot from subversion and built it with maven. The relevant jars I generated can be found here.

Now I try to parse a toy mbox file from mime4j test.

I use this sample code. Briefly:

final File mbox = new File("c:\\mbox.rlug");
int count = 0;
for (CharBufferWrapper message : MboxIterator.fromFile(mbox).charset(ENCODER.charset()).build()) {
    System.out.println(messageSummary(message.asInputStream(ENCODER.charset())));
    count++;
}
System.out.println("Found " + count + " messages");

+

private static String messageSummary(InputStream messageBytes) throws IOException, MimeException {
    MessageBuilder builder = new DefaultMessageBuilder();
    Message message = builder.parseMessage(messageBytes);
    return String.format("\nMessage %s \n" +
            "Sent by:\t%s\n" +
            "To:\t%s\n",
            message.getSubject(),
            message.getSender(),
            message.getTo());
}

The output is:

Message null Sent by: null To: null

Message null Sent by: null To: null

Message null Sent by: null To: null

Message null Sent by: null To: null

Message null Sent by: null To: null

Found 5 messages

There are indeed 5 messages, but why are all fields null?

Answer 1

Based on @zvisofer answer, I found the guilty piece of code in BufferedLineReaderInputStream:

@Override
public int readLine(final ByteArrayBuffer dst)
        throws MaxLineLimitException, IOException {
    if (dst == null) {
        throw new IllegalArgumentException("Buffer may not be null");
    }
    if (!readAllowed()) return -1;

    int total = 0;
    boolean found = false;
    int bytesRead = 0;
    while (!found) {
        if (!hasBufferedData()) {
            bytesRead = fillBuffer();
            if (bytesRead == -1) {
                break;
            }
        }
        int i = indexOf((byte)'\n');
        int chunk;
        if (i != -1) {
            found = true;
            chunk = i + 1 - pos();
        } else {
            chunk = length();
        }
        if (chunk > 0) {
            dst.append(buf(), pos(), chunk);
            skip(chunk);
            total += chunk;
        }
        if (this.maxLineLen > 0 && dst.length() >= this.maxLineLen) {
            throw new MaxLineLimitException("Maximum line length limit exceeded");
        }
    }
    if (total == 0 && bytesRead == -1) {
        return -1;
    } else {
        return total;
    }
}

The best thing to do would be to report the bug but here is a fix, a little dirty but it's working fine

Create the class org.apache.james.mime4j.io.BufferedLineReaderInputStream in your project

Replace the method public int readLine(final ByteArrayBuffer dst) by this one :

@Override
public int readLine(final ByteArrayBuffer dst)
        throws MaxLineLimitException, IOException {
    if (dst == null) {
        throw new IllegalArgumentException("Buffer may not be null");
    }
    if (!readAllowed()) return -1;

    int total = 0;
    boolean found = false;
    int bytesRead = 0;
    while (!found) {
        if (!hasBufferedData()) {
            bytesRead = fillBuffer();
            if (bytesRead == -1) {
                break;
            }
        }

        int chunk;
        int i = indexOf((byte)'\r');
        if (i != -1) {
            found = true;
            chunk = i + 2 - pos();
        } else {
            i = indexOf((byte)'\n');
            if (i != -1) {
                found = true;
                chunk = i + 1 - pos();
            } else {
                chunk = length();
            }
        }
        if (chunk > 0) {
            dst.append(buf(), pos(), chunk);
            skip(chunk);
            total += chunk;
        }
        if (this.maxLineLen > 0 && dst.length() >= this.maxLineLen) {
            throw new MaxLineLimitException("Maximum line length limit exceeded");
        }
    }
    if (total == 0 && bytesRead == -1) {
        return -1;
    } else {
        return total;
    }
}

Enjoy both unix and dos files :)