Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to use Hive jobs with Azkaban?

I would like to use Azkaban for periodic Hive jobs, I have looked through Azkaban documentation, and it seems like by default it doesn't support Hive jobs, do you know how can I use these two together?

I think, I'll have to run Hive jobs as a "command job" available in Azkaban, but maybe someone has worked it out.
I was using Oozie for some time, but It didn't meet my needs.

Thanks.

like image 213
wlk Avatar asked Nov 20 '25 10:11

wlk


2 Answers

Right now we don't have an easy way. You can certainly hack into the HiveCliDriver and do it from there, but it's suboptimal... Alternatively, just run it as a command line job. We're using a different system at LI. I hope to add this ability pretty quickly, but not sure when I'll have the chance.

like image 126
Jakob Homan Avatar answered Nov 22 '25 03:11

Jakob Homan


As of Azkaban 2.1, Hive jobs are supported. For details, see the documentation.

like image 25
Michael Hausenblas Avatar answered Nov 22 '25 03:11

Michael Hausenblas



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!