Menu
I am currently trying to set up an Neural Network for information extraction and I am pretty fluent with the (basic) concepts of Neural Networks, except for one which seem to puzzle me. It is probably pretty obvious but I can't seem to found information about it.
Where/How do Neural Networks store their memory? ( / Machine Learning)
There is quite a bit of information available online about Neural Networks and Machine Learning but they all seem to skip over memory storage. For example after restarting the program, where does it find its memory to continue learning/predicting? Many examples online don't seem to 'retain' memory but I can't imagine this being 'safe' for real/big-scale deployment.
I have a difficult time wording my question, so please let me know if I need to elaborate a bit more. Thanks,
EDIT: - To follow up on the answers below
Every Neural Network will have edge weights associated with them. These edge weights are adjusted during the training session of a Neural Network.
This is exactly where I am struggling, how do/should I vision this secondary memory? Is this like RAM? that doesn't seem logical.. The reason I ask because I haven't encountered an example online that defines or specifies this secondary memory (for example in something more concrete such as an XML file, or maybe even a huge array).
550
Long-term memories are therefore stored in the neural network, resulting in a close Page 2 Memory and neural networks relationship between how information is represented, processed, stored and recalled.
Introduction. It is well known that the capacity of a Hebbian neural network consisting of n neurons is approximately 0.15n [1]. This capacity can be increased to n if delta learning is used [2] and substantially more if non-binary networks are used where the capacity depends on how the data is quantized [3].
In other words, recalling a memory involves re-activating a particular group of neurons. The idea is that by previously altering the strengths of particular synaptic connections, synaptic plasticity makes this possible. Memories are stored by changing the connections between neurons.
So it is certainly storing some data. The theoretical limit of data compression is -∑ p ln(p). The data in a neural network however is stored as weights and biases and not variable length coding, so does this equation even apply?
Memory storage is implementation-specific and not part of the algorithm per se. It is probably more useful to think about what you need to store rather than how to store it.
Consider a 3-layer multi-layer perceptron (fully connected) that has 3, 8, and 5 nodes in the input, hidden, and output layers, respectively (for this discussion, we can ignore bias inputs). Then a reasonable (and efficient) way to represent the needed weights is by two matrices: a 3x8 matrix for weights between the input and hidden layers and an 8x5 matrix for the weights between the hidden and output layers.
For this example, you need to store the weights and the network shape (number of nodes per layer). There are many ways you could store this information. It could be in an XML file or a user-defined binary file. If you were using python, you could save both matrices to a binary .npy file and encode the network shape in the file name. If you implemented the algorithm, it is up to you how to store the persistent data. If, on the other hand, you are using an existing machine learning software package, it probably has its own I/O functions for storing and loading a trained network.
170
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
