Menu
I'm researching ETL tools to import flat files into a database and subsequently export xml files.
Many of the tools support generating code to use in your application; however, I haven't found any that support using code already in your application. Our model is complex (relationships, validations, polymorphic associations, callbacks, etc.).
What tools are available that will allow reuse of existing code? Or am I stuck recreating (and maintaining) my model in the ETL tool?
Note: My requirements for an ETL (as opposed to bulk inserts or activerecord-import) are the transformations. We receive data from over 200 different sources in a variety of formats, level of completeness, and cleanliness. Also, the "designer" most include is more realistic for the less-technical users who will be defining the transformations.
598
Google Cloud Dataflow This software can be deployed for both batch and real-time processing, and in either a scheduled or a real-time on demand mode. Because Google Cloud Dataflow is cloud-based, it can automatically scale to accommodate the processing and storage that you need for any ETL job.
Which ETL tool is used most? There is no such ETL tool that is used most but here are some of the ETL Tools that are in high demand across industries Xplenty, Skyvia, Talend, Apache Nifi.
Yes, Alteryx is a ETL and data wrangling tool but it does a lot more than pure ETL. Alteryx wraps up pre-baked connectivity (Experian / Tableau etc) options alongside a host of embedded features (like data mining, geospatial, data cleansing) to provide a suite of tools within one product.
ActiveWarehouse might prove useful. Initial search results make the project feel a bit old and defunct. A little digging yielded a fairly active, well documented branch of the project on GitHub: https://github.com/activewarehouse/activewarehouse-etl
112
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
