Is Web Service suitable for ETL purpose?

Question

My company is considering using web service as mean of ETL process. However I don't think web service fit into this purpose, for several reasons: 1. web service could possibly consume a lot of memory when generating large xml. 2. xml is a bloated format. 3. possibly time-out if the server takes huge amount of time to generate data 4. file size limitation? (for windows, it's 2Gb, if my memory serves me right)

I am not a web service expert, so I need your opinions. :)

Thanks.

Answer 1

There are plenty of technologies in the Web Services tool shed that circumvent all the problems you elaborate. There is stream oriented XML shredding, there are XML compression formats for delivery, protocols that deal with fragmentation and fairness and there are many a storage systems that can hold terabytes upon terabytes of data.

If by web service you imagine some college freshmen homework concoction of an interface that accepts a single glop argument with a 2GB serialized table in it then all your arguments are valid. But if you give your requirements to an experienced team with knowledge of the concepts involved in WS-ReliableMessaging and WS-Transaction then there is no reason not to have an ETL process around Web Services. Note that I do not advocate the SOAP protocols per-se, but I do advocate knowledge and understanding of the concepts involved.

Now that being said, whether an Web Service oriented ETL process makes sense for you or not it depends on a whole set of other reasons. However, your rebuttal of the Web Service technologies does not hold water.