Informatica - Duplicate elimination

Question

I am having a flat file with 10 records, and out of it 5 records are duplicate records (unique key column: Customer_Id and source_system). This flat file has to load to a Oracle table and this is the first load.

How can I eliminate the duplicates in it?

Answer 1

There are quite a few ways to handle this:

1. Load the data to a temp table and then just select the distinct records
2. Use a sorter or aggregator transformation and then select the distinct option (In the properties tab if I remember correctly)
3. Use the dynamic lookup which will only insert records into the cache if they have not been inserted already.
4. Use variable port in expression transformation and by using filter we can delete the duplicate records even if it is relational table or flat files.since dynamic lookup is costly, this way is preferable.