automating NIFI template deployment

Question

I'm new to nifi and am trying to understand (since it looks many GUI based) if there is a way to automate scale up and down on Nifi and how one would take an xml Nifi template and deploy it to a cluster.

Essentially what we're trying to do is use Nifi to collect JMX and log files off of kafka servers as they come up in an automated fashion so logging and JMX counters start flowing to, lets say and Elasticsearch cluster.

For example, right now we've automated deployment of the kafka servers using Ansible, so the hope was to configure and deploy Nifi xml teamplates along with the cluster deployment so no manual intervention is required to collect the kafka jmx counters and log files.

How would one automate this in Nifi?

Answer 1

There are a number of tools and techniques available to automate deployment of a template, including:

• curl, see SO: Post a NIFI template via REST? - most basic API template deployment.
• nifi-api-deploy - performs template deployment plus some migration tasks.
• nifi-deploy-process-group - performs process group migration.

The hard part is migrating from the existing flow to the new flow, especially where an existing flow may have queued flowfiles, and provenance history references the existing flow. But if you are automating the entire deployment of the NiFi cluster and deploying the template to an empty canvas, this might not be a problem.

You can read a full treatment of the issues involved and future plans in Configuration Management of Flows.