Best approach to construct a real-time rule engine for our streaming events

Question

We are at the beginning of building an IoT cloud platform project. There are certain well known portions to achieve complete IoT platform solution. One of them is real-time rule processing/engine system which is needed to understand that streaming events are matched with any rules defined dynamically by end users with readable format (SQL or Drools if/when/then etc.)

I am so confused because there are lots of products, projects (Storm, Spark, Flink, Drools, Espertech etc.) in internet so, considering we have 3-person development team (a junior, a mid-senior, a senior), what would it be the best choice ?

• Choosing one of the streaming projects such as Apache Flink and learn well ?
• Choosing one of the complete solution (AWS, Azure etc.)

Answer 1

The BRMS(Business Rule Management System) like Drools is mainly built for quickly adapting changes in business logic and are more matured and stable compared to stream processing engines like Apache Storm, Spark Streaming, and Flink. Stream processing engines are built for high throughput and low latency. The BRMS may not be suitable to serve hundreds of millions of events in IOT scenarios and may be difficult to deal with event-time-based window calculations.

All these solutions can be used in Iaas providers. In AWS you may also want to take a look at AWS EMR and Kinesis/Kinesis Analytics.

Some use cases I've seen.

1. Stream data directly to FlinkCEP.

2. Use rule engines to do fast response with low latency, at the same time stream data to Spark for analysis and machine learning.

You can also run Drools in Spark and Flink to hot-deploy user-defined rules.

Answer 2

Disclaimer, I work for them. But, you should check out Losant. It's developer friendly and it's super easy to get started. We also have a workflow engine, where you can build custom logic/rules for your application.

Answer 3

check out the Waylay rules engine built specifically for real-time IoT data streams.