Predicting a Poisson process

Question

I want to predict the inter-arrival times of road traffic with Poisson distribution. At the moment, I produce the (synthetic) arrival times with Poisson process so that the inter-arrival times have exponential distribution.

Observing the past data, I want to predict the next/future inter-arrival time. For that I want to implement a learning algorithm.

I have used various approaches, e.g., Bayesian predictor (maximum a posteriori) and multi-layer neural network. In both of these methods, I use a moving window of a certain length n of the input features (inter-arrival times).

In Bayesian predictor, I use the inter-arrival times as binary features (1->long, 0-> short to predict the next inter-arrival time to be long or short), whereas for neural network of n-neurons input layer and m-neurons hidden layer (n=13, m=20), I input n previous inter-arrival times and generate the future estimated arrival time (the weights are threshold are updated by the back-propagation algorithm).

The problem with Bayesian approach is that it becomes biased if the number of short inter-arrival times is higher than long ones. So that, it never predicts the long idle period (as the posterior of short always remains larger. Whereas, in multi-layer neural predictor, the prediction accuracy is not sufficient. Specially for higher inter-arrival times, the prediction accuracy decreases drastically.

My question is "Can the stochastic process (Poisson) not be predicted with a good accuracy? or my approach is not correct?". Any help will be appreciated.

Answer 1

If it really follows a poisson distribution you can only predict the probability that the next traffic item will have arrived at given interval - and the probability curve is simply the normalized integral (i.e. the curve with an asymptote of 1) of the poisson distribution. Why all the messing about with neural networks/Bayesian predictors?

Answer 2

Well, if the generating process is a homogeneous Poisson process, there isn't that much to predict, right? There's the rate parameter which stays constant throughout time and can be trivially estimated but past that, recent history should have no effect on the inter-arrival times. You're using binary features of recent arrivals but the whole point of homogeneous Poisson processes is that the arrivals are iid exponential and exponential distributions are memoryless.

Now, if the homogeneous assumption is not correct, you need to think more about the details and the answer depends on what mean measure you wish to use for the process. Take a look at Cox processes (double-stochastic Poisson processes, where the mean measure is also a random variable) or possibly Hawkes processes (where each arrival causes a burst of further activity).