Is it against the Apache Beam Programming Model to Invoke an API?

Question

When using Apache Beam to enrich data, would it be wrong to make an API call for each data item

(I'm new to Apache Beam)

Answer 1

No, but you can batch API calls for better performance. Check out "Batched RPC" example in this blog post.

Another thing to note is that beam cannot guarantee exactly-once for external API calls. If the pipeline in question needs exactly-once QoS, you should strive to make API calls idempotent.

Answer 2

This depends on the type of API call and the size of your data. If the data specify the API call that needs to be made, this is appropriate. However, if you have some limited set of data that is used to enrich your input elements, it may be possible to use parts of the Beam programming model to reduce the number of external calls required.

As an example, if the data that produces the results of your API call can be preloaded, you may be able to use a side input by reading all of the data and using View.asMap (or whatever view is appropriate), reducing the time number of API calls to some relatively constant number per execution. The side input can then be consumed by using ParDo.withSideInputs; see the programming guide, specifically about ParDo and SideInputs