Spring Integration or Spring Cloud Data Flow

Question

I'm in the process of moving some of my application to microservices. One example service is a RSS crawler which uses Spring Integration to send items to Kafka. I also see there is Spring Cloud Data Flow which uses Spring Integration and Spring Batch under the hood.

If I used Spring Cloud Data Flow, would I use this within each microservice or is it used as orchestration between microservices?

I'm trying understand the benefits of introducing Spring Cloud Data Flow as opposed to keeping the microservices as light as possible using Spring Integration.

Any advice would be appreciated.

Answer 1

The SI based "RSS crawler" service can be packaged as a Spring Cloud Stream application. Once you have done that, your "RSS crawler" turns into a standalone event-driven microservice that automatically can either talk to Kafka, Rabbit, or other brokers (depending on the binder implementation in the classpath). The same app is a portable workload that can run on any public or private cloud platforms.

Once you have an "n" number of such standalone applications, you'd need a higher-level orchestration layer to compose the applications into a coherent data pipeline. Spring Cloud Data Flow helps with it. You can build a pipeline like the following with SCDF's DSL.

stream create foo --definition "rss-crawler | filter | transform | cassandra"

Here, four applications come together to form a data pipeline. Each one of them is a Spring Cloud Stream application that can be independently developed and tested in isolation. Finally, SCDF would deploy the applications onto target platforms like Cloud Foundry or Kubernetes as native applications.

Hope this further clarifies.