Logo Questions Linux Laravel Mysql Ubuntu Git Menu
HTML CSS JAVASCRIPT SQL PYTHON PHP BOOTSTRAP JAVA JQUERY R React Kotlin
 

Running R scripts in Airflow?

Tags:

r

airflow

Is it possible to run an R script as an airflow dag? I have tried looking online for documentation on this and am unable to do so. Thanks

like image 359
alexsmith2 Avatar asked Aug 22 '17 19:08

alexsmith2

People also ask

Can Airflow Run R script?

As stated here, there are different ways to run R scripts in Airflow and maybe the best way is to containerize your R script and run it using the DockerOperator, which is included in the standard distribution.

What is Apache airflow good for?

What is Airflow Used For? Apache Airflow is used for the scheduling and orchestration of data pipelines or workflows. Orchestration of data pipelines refers to the sequencing, coordination, scheduling, and managing complex data pipelines from diverse sources.

Can we use Airflow for streaming?

Airflow is not a data streaming solution. Tasks do not move data from one to the other (though tasks can exchange metadata!). Airflow is not in the Spark Streaming or Storm space, it is more comparable to Oozie or Azkaban. Workflows are expected to be mostly static or slowly changing.

Is Apache airflow free to use?

Airflow is free and open source, licensed under Apache License 2.0.

2 Answers

There doesn't seem to be a R Operator right now.

You could either write your own and contribute to the community or simply run your task as a BashOperator calling RScript.

like image 117
Pierre Sutter Avatar answered Oct 13 '22 05:10

Pierre Sutter

Another option is to containerize your R script and run it using the DockerOperator, which is included in the standard distribution. This removes the need to have your worker nodes configured with the correct version of R and any needed R libraries.

like image 29
gcbenison Avatar answered Oct 13 '22 04:10

gcbenison
Sign in to Comment

Related questions

Converting an object of class "table" to matrix in R
Swapping values between two columns using data.table
Efficiency of matrix rowSums() vs. colSums() in R vs Rcpp vs Armadillo
In ggplot restrict y to be >0 in LOESS
Image analysis in R
R: two scatterplots on single graph using ggplot
Multivariate Linear Mixed Model in lme4
Avoid argument duplication passed through (...)
R: change background color of plot for specific area only (based on x-values)
How is 95% CI calculated using confint in R?
TwitteR, ROAuth and Windows: register OK, but certificate verify failed
RStudio Shiny Conditional Plot
How to get environment of a variable in R
Generate a Filled geom_step
How to change the position of the table of contents in rmarkdown?
python equivalent of get() in R (= use string to retrieve value of symbol)
Dynamic plot height in Shiny
Alternative to R's `memory.size()` in linux?
Complexe non-equi merge in R
ggplot 'non-finite values' error

Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!