Straight Java/Groovy versus ETL tool (Talend/etc) - what libraries would you use?

Question

Assume you have a small project which on the surface looks like a good match for an ETL tool like Talend.

But assume further, that you have never used Talend and furthermore, you do not trust "visual programming" tools in general and would rather code everything the old fashioned way (text on a nice IDE!) with the help of an appropriate language & support libraries.

What are some language patterns & support libraries that could help you stay away from the ETL tool temptation/trap?

Answer 1

It depends on whether the deliverable is the processor or the output itself. If you just need to deliver the output, you don't need to maintain the code. If the code needs to be maintained then will it be you maintaining it or somebody else?

If somebody else needs to maintain I'd use Java or give them Talend.

If it's throwaway code, I'd use what will be easier or fun to program with.

If you need to maintain it and the processing is complex, I'd use Scala. It has:

• some libraries to interact with databases
• xml literals
• parser combinators
• interesting features on its collection packages (map, filter, groupBy, partition, ...)
• and of course any other existing Java libraries.

Answer 2

I used to think that "visual programming" is something for people who can't program. Then I was exposed to Talend in a project, and I realized that this type of tool is exactly right for the job, when it comes to moving data from A to B, and transforming it in the process. It's component-oriented software design, by a more academic label.

I still consider myself a decent programmer who can do anything, and then some, with a text editor and a shell prompt. But I've become a big fan of Talend as well.

Full disclosure: I now work for the company :-)