Logo Questions Linux Laravel Mysql Ubuntu Git Menu
HTML CSS JAVASCRIPT SQL PYTHON PHP BOOTSTRAP JAVA JQUERY R React Kotlin
 

Airbnb Airflow vs Apache Nifi [closed]

Tags:

python

airflow

apache-nifi

Are Airflow and Nifi perform the same job on workflows? What are the pro/con for each one? I need to read some json files, add more custom metadata to it and put it in a Kafka queue to be processed. I was able to do it in Nifi. I am still working on Airflow. I am trying to choose the best workflow engine for my project Thank you!

like image 690
CMPE Avatar asked Sep 08 '16 19:09

CMPE

People also ask

What is the difference between Apache NiFi and Airflow?

Summary. By nature, Airflow is an orchestration framework, not a data processing framework, whereas NiFi's primary goal is to automate data transfer between two systems. Thus, Airflow is more of a “Workflow Manager” area, and Apache NiFi belongs to the “Stream Processing” category.

Is Apache NiFi an ETL?

Apache NiFi is an ETL tool with flow-based programming that comes with a web UI built to provide an easy way (drag & drop) to handle data flow in real-time. It also supports powerful and scalable means of data routing and transformation, which can be run on a single server or in a clustered mode across many servers.

What are the limitations of Apache airflow?

Airflow has a strict dependency on a specific time: the execution_date . No DAG can run without an execution date, and no DAG can run twice for the same execution date.

Is NiFi a good tool?

Apache NiFi is considered one of the best open-source ETL tools because of its well-rounded architecture. It's a powerful and easy-to-use solution. FlowFile includes meta-information, so the tool's capabilities aren't limited to CSV. You can work with photos, videos, audio files, or binary data.

1 Answers

For a great overview of Airflow and Apache NiFi checkout this reddit post: https://www.reddit.com/r/bigdata/comments/51mgk6/comparing_airbnb_airflow_and_apache_nifi/

For your specific use-case of ingesting Json files, enriching them and routing them to Kafka I believe NiFi is the right tool for the job. A couple of processors you could potentially use, as well as documentation for each, are below:

GetFile: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.GetFile/index.html

JoltTransformJSON: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.JoltTransformJSON/index.html

PublishKafka (or PublishKafka_0_10 depending on your version): https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-kafka-0-9-nar/1.9.2/org.apache.nifi.processors.kafka.pubsub.PublishKafka/index.html

like image 74
JDP10101 Avatar answered Sep 20 '22 05:09

JDP10101
Sign in to Comment

Related questions

How to expand all the subsections on the sidebar toctree in Sphinx?
How to work within individual files rather than projects in PyCharm
Execute terminal commands in python3 [duplicate]
conda environment to AWS Lambda
git log --follow, the gitpython way
Understanding matplotlib: plt, figure, ax(arr)?
Python Json loads() returning string instead of dictionary?
How to connect PyCharm to a python interpreter located inside a Docker container?
What is the equivalent of a Python docstring in Ruby?
How to limit python traceback to specific files
Kmeans without knowing the number of clusters? [duplicate]
Relative imports with unittest in Python
Detect if a text image is upside down
Given the name of a Python package, what is the name of the module to import?
numerically stable way to multiply log probability matrices in numpy
How to obtain the same font(-style, -size etc.) in matplotlib output as in latex output?
Name of a function returning a generator
Scrapy Python Set up User Agent
Necessity of explicit cursor.close()
"Too many indexers" with DataFrame.loc

Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!