How do i use cache in kettle pentaho?

Question

I am processing data, where i get some information from rest api, based on the value of a field.

Now, value may repeat for that field and if I already have fetched the data for that value, from REST, i would like to reuse that value and saving an API call (slowest operation in the transformation).

is is possible? if yes, how?

Regards Ajay

Answer 1

@RFVoltini you are right, maybe we could try to setup a H2 db server for this purpouse: http://type-exit.org/adventures-with-open-source-bi/2011/01/using-an-on-demand-in-memory-sql-database-in-pdi/

other option is using memcached in java : http://sacharya.com/using-memcached-with-java/

Answer 2

I've did an example transformation, that gets from a webservice country names by country codes. I've used the idea where you just need to get from the webservice the distinct country codes/names then lookup them on your main pipeline.

Take a look at this example: https://docs.google.com/open?id=0B-AwXLgq0XmaV0V0cHlfTFZlVUU and see if this method applies to you.