PutFile append file

Question

New to Nifi!

I'm wondering if there is a way in nifi to use a processor such as "PutFile" and have it write to one single file (append data to this file, or over-write the data in this file) - rather than create multiple different files? Is there another processor I need to use in order to accomplish this?

Answer 1

For those who don't want to override the data in the file but want to append data.

Appending to single file Using ExecuteStreamCommand processor:

It isn't possible with putFile processor but you can use the ExecuteStreamCommand processor to accomplish this.

In command arguments put attributes you want to log speparated by delimiter ${aatr1};${aatr2};${attr3}

In command path put the absolute path of a bash script: /path/logger.sh

logger.sh:

#!/bin/bash
echo "$1|$2|$3">> /path/attributes.log

attibutres.log will append the three attributes line by line. Make sure that the bash script is executable with nifi.

do a chmod 777 logger.sh

Appending to single file Using ExecuteScript Processor:

Try this ECMAscript:

var flowFile = session.get();
var File = Java.type("java.io.RandomAccessFile");
if (flowFile != null) {
    var filename = flowFile.getAttribute("filename");
    /// write to file
    var filePath = "/path/attributes.log" ;
    flowFile = session.putAttribute(flowFile, "filePath", filePath);
     var file = new File(filePath, "rws");
     file.seek(file.length());
        file.write(filename.getBytes());
        file.write("\n".getBytes());
        file.close();
    // Finish by transferring the FlowFile to an output relationship
    session.transfer(flowFile, REL_SUCCESS);
}

Answer 2

Currently there is no way to append data to a file but you can overwrite the file using PutFile.

The PutFile processor writes a file to disk using the the attribute "filename" on the FlowFile. So if you put an UpdateAttribute processor before a PutFile that updates all incoming FlowFiles with to same "filename" then the PutFile processor will write them all with the same file name on disk.

To do this with PutFile, make sure you configure processor property "Conflict Resolution Strategy" to "Replace".