Is there a way to train an Encog neural network without loading all training set in memory?

Question

I've been using a .NET framework to train a neural network, and my data set is really large, with a large number of inputs.

I've decided to switch to Encog due to some problems with the old framework I was using, and also because Encog seems much richer and offers additional features.

My problem is that Encog loads all training data at once, into a 2D array, and this doesn't suit me since I have a very large data set. I've tried working around this, and also checked other questions (such as this one but couldn't find a good answer).

I tried working with SQLNeuralDataSet and other implementations that support streaming operation but they all seem to eventually load the data into memory (using MemoryDataLoader) and this doesn't solve my problem.

Is there a way I can load (and train) my network one item at a time? Or is this option not available in Encog?

Thanks

EDIT

I ended up copying the source code and modifying it according to my needs. Basically, it all comes down to Process(IMLDataPair pair) in GradientWorker which, according to the documentation:

Process one training set element.

Answer 1

Looking at the source code for the data sources for Encog, there are a number of options. Look at Buffer/BufferedMLDataSet.cs, for example:

/// This class is not memory based, so very long files can be used, without
/// running out of memory. This dataset uses a Encog binary training file as a
/// buffer.