Nifi failed to write to FileSystemRepository Stream

Question

I have a flow where I am using the getFile processor. The input directory is a network mount point. When I test the flow on small files (les than 1GB), it works well. When I test it on bigger files (more than 1GB), I get the following error :

GetFile[id=f1a533fd-1959-16d3-9579-64e64fab1ac6] Failed to retrieve files due to org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import data from /path/to/directory for StandardFlowFileRecord[uuid=f8389032-c6f5-43b9-a0e3-7daab3fa115a,claim=,offset=0,name=490908299598990,size=0] due to java.io.IOException: Failed to write to FileSystemRepository Stream [StandardContentClaim [resourceClaim=StandardResourceClaim[id=1486976827205-28, container=default, section=28], offset=0, length=45408256]]

Do you have any idea about the origin of this error ?

Thank you for your answers

Answer 1

Based on the comments the answer in this specific case was found by Andy and confirmed by the asker:

The content repositories were too small in proportion to the file size.

Another thing to look at for future readers is whether the memory of the Nifi node is large enough to hold individual messages.

Answer 2

Although the answer provided by Dennis is correct at analysing the root cause, there is not a solution for it, so let me provide one.

Answer/Solution for Containers

Since you can't specify a size for a docker volume, we can't use them for this task if you are lacking the required space for your flowfiles content.

Instead, I recommend using bind mounts. This way, you could use up to (theoretically) all your machine disk.

# Create a folder where to locate the NiFi Flowfiles content in the host
mkdir -p /tmp/data/nifi/nifi-data-content_repository

Then, modify your docker-compose.yml file to modify the type of storage to be used. Specifically, you have to look for the contents repository volume:

- type: source
  source: nifi-data-content_repository
  target: /opt/nifi/nifi-current/content_repository

And replace it with the bind mount targeting the folder that we've just created above:

- type: bind
  source: /tmp/data/nifi/nifi-data-content_repository
  target: /opt/nifi/nifi-current/content_repository

Ready, now you can re-deploy your NiFi with this mount capable of using your host disk space.

Read more on bind mounts in the official docs.