Apache NiFi - Using multiple FlowFiles as input to a processor

Question

I have a workflow where two or more inputs have set operations performed on them (union, complement, etc.) to produce a single output. I expect to have to write a processor to do the set logic myself, but is it even possible to work with multiple flowfiles of different provenance and work on them simultaneously?

Answer 1

NiFi processors can operate on all of the flowfiles in their input queue(s). For example:

@Override
public void onTrigger(final ProcessContext context, final ProcessSession session) {
    List<FlowFile> flowFiles = session.get(context.getProperty(BATCH_SIZE).asInteger());
    if (flowFiles == null || flowFiles.size() == 0) {
        return;
    }
    // process flowFiles
    ...

You can use the Funnel component to bring multiple inputs together into a single input queue, which can then share the same backpressure and prioritization settings.