Why should I not loop or block in Spout.nextTuple()

Question

I saw many code snippets in which a loop was used inside Spout.nextTuple() (for example to read a whole file and emit a tuple for each line):

public void nextTuple() {
    // do other stuff here

    // reader might be BufferedReader that is initialized in open()
    String str;
    while((str = reader.readLine()) != null) {
        _collector.emit(new Values(str));
    }

    // do some more stuff here
}

This code seems to be straight forward, however, I was told that one should not loop inside nextTuple(). The question is why?

Answer 1

When a Spout is executed it runs in a single thread. This thread loops "forever" and has multiple duties:

1. call Spout.nextTuple()
2. retrieve "acks" and process them
3. retrieve "fails" and process them
4. time-out tuples

For this to happen, it is essential, that you do not stay "forever" (ie, loop or block) in nextTuple() but return after emitting a tuple to the system (or just return if no tuple can be emitted, but do not block). Otherwise, the Spout cannot does its work properly. nextTuple() will be called in a loop by Storm. Thus, after ack/fail messages are processed etc. the next call to nextTuple() happens quickly.

Therefore, it is also considered bad practice to emit multiple tuples in a single call to nextTuple(). As long as the code stays in nextTuple(), the spout thread cannot (for example) react on incoming acks. This might lead to unnecessary time-outs because acks cannot be processed timely.

Best practice is to emit a single tuple for each call to nextTuple(). If no tuple is available to be emitted, you should return (without emitting) and not wait until a tuple is available.