The rate of the dataflow is exceeding the provenance recording rate. slowing down flow to accommodate

Question

I am using nifi 1.1.0 and running too many processors on that so it has too much load to run. Because of too much load processors are running very slow and I am getting some error :

The rate of the dataflow is exceeding the provenance recording rate. slowing down flow to accommodate.

I changed the storage size of provenance repository in "nifi.properties" file but no improvement. I changed the following property :

nifi.provenance.repository.max.storage.size = 2 GB

I googled but didn't get proper solution. Could anyone help me how can I resolve this issue.

Please find the below screenshot for the same :

Answer 1

That error belongs to following property.

nifi.provenance.repository.index.threads=1

The default value is 1.

For flows that operate on a very high number of FlowFiles, the indexing of Provenance events could become a bottleneck.

If this is the case, a bulletin will appear indicating, "The rate of the dataflow is exceeding the provenance recording rate. Slowing down flow to accommodate."

If this happens, increasing the value of this property may increase the rate at which the Provenance Repository is able to process these records, resulting in better overall throughput.

Keep in mind that as you increase the number of threads allocated to one process, you reduce the number available to another. So you should leave this at one unless the above error message is encountered.

For your consideration you may configure NiFi with following best practices

https://community.hortonworks.com/articles/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.html

you might get better performance of your results.