Creating anomaly detection using machine learning

Question

I'm very impressed from the new x-pack ML of the elastic stack. It seems their technique learns data patterns over time and can predict anomalies in multiple domains.

Zoomed in:

I was wondering what approach and network topology could be used, in order to create a similar feature. Is it fair to assume that since x-pack works on time series data, RNN would be a good start?

Interested in your opinions and references.

Answer 1

From what I read, I'd be surprised if they're using neural networks. Here's how they say they detect anomalies:

Detect outliers in a population by building a profile of a “typical” user or machine to know when one starts to stray from the pack.

Doing anomaly detection like that requires nothing more than a statistical test of whether or not observed behavior is within 2-3 standard deviations of the expected behavior.

If you want to use neural networks for some reason, you could go with CNNs, RNNs, or attention-only networks. Google recently showed that you don't need RNNs or CNNs to do state-of-the-art translation.