Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to persist nifi flowfiles by using docker-compose?

right now I'm using nifi and its processors for some streaming stuff (mqtt listener, json evaluating, text replacement, write into db ...). I'm trying to persist the flowfiles and therefore I did some volume mapping (see below). But it doesn't work; after restarting the container it seems the flowfiles arent't saved ...

Could anybody give me a hint how to solve that problem?

nifi:
image: apache/nifi
restart: on-failure
ports:
  - "8000:8000"
networks:
  - traefik
environment:
  - NIFI_WEB_HTTP_PORT=8000
volumes:
  - nifi_conf:/opt/nifi/conf
  - nifi_state:/data/nifi/state
  - nifi_db:/opt/nifi/database_repository
  - nifi_flowfile:/opt/nifi/flowfile_repository
  - nifi_content:/opt/nifi/content_repository
  - nifi_provenance:/opt/nifi/provenance_repository 


volumes:
 nifi_provenance:{}
 nifi_flowfile: {}
 nifi_content: {}
 nifi_db: {}
 nifi_state: {}
 nifi_conf: {}

Thanks.

like image 616
T_F Avatar asked Mar 24 '19 16:03

T_F


2 Answers

you could map docker container folders directly to the host machine like this:

services:
  nifi:
    ...
    volumes:
      - ./conf:/opt/conf
      - ./nifi_state:/data/nifi/state
      ...

no additional volume definition required

note that under windows with virtualbox this feature works only in the current user directory.

like image 59
daggett Avatar answered Oct 21 '22 14:10

daggett


One update from my side. Apache Nifi changed their directories after 1.8.0. So you should use the following:

volumes:
      - ./nifi_state:/opt/nifi/nifi-current/state
      - ./nifi_db:/opt/nifi/nifi-current/database_repository
      - ./nifi_flowfile:/opt/nifi/nifi-current/flowfile_repository
      - ./nifi_content:/opt/nifi/nifi-current/content_repository
      - ./nifi_provenance:/opt/nifi/nifi-current/provenance_repository   
like image 28
Patt Rick Avatar answered Oct 21 '22 15:10

Patt Rick