I've seen "sparse" and "sparsity" used in a way that suggests it's something that improves a model's accuracy. For example:
I think the unsupervised phase might be not so important if some sparse connections or neurons are used, such as rectifier units or convolutional connection, and big training data is available.
From https://www.quora.com/When-does-unsupervised-pre-training-improve-classification-accuracy-for-a-deep-neural-network-When-does-it-not
What does "sparse" mean in this context?
We define a sparse neural network as a network in which only a percentage of the possible connections exists. You can imagine a fully connected layer with some of the connections missing.
Definition: A set of numbers (e.g. vector, matrix, etc.), is considered sparse when a high percentage of the values are assigned a constant default value.
In AI inference and machine learning, sparsity refers to a matrix of numbers that includes many zeros or values that will not significantly impact a calculation.
Sparse coding is the representation of items by the strong activation of a relatively small set of neurons. For each stimulus, this is a different subset of all available neurons.
TL;DR: Sparsity means most of the weights are 0. This can lead to an increase in space and time efficiency.
Detailed version: In general, neural networks are represented as tensors. Each layer of neurons is represented by a matrix. Each entry in the matrix can be thought of as representative of the connection between two neurons. In a simple neural network, like a classic feed-forward neural network, every neuron on a given layer is connected to every neuron on the subsequent layer. This means that each layer must have n2 connections represented, where n is the size of both of the layers. In large networks, this can take a lot of memory and time to propagate. Since different parts of a neural network often work on different subtasks, it can be unnecessary for every neuron to be connected to every neuron in the next layer. In fact, it might make sense for a neural network to have most pairs of neurons with a connection weight of 0. Training a neural network might result in these less significant connection weights adopting values very close to 0 but accuracy would not be significantly affected if the values were exactly 0.
A matrix in which most entries are 0 is called a sparse matrix. These matrices can be stored more efficiently and certain computations can be carried out more efficiently on them provided the matrix is sufficiently large and sparse. Neural networks can leverage the efficiency gained from sparsity by assuming most connection weights are equal to 0.
I must say that neural networks are a complex and diverse topic. There are a lot of approaches used. There are certain kinds of neural networks with different morphologies than the simple layer connections I referenced above. Sparsity can be leveraged in many types of neural networks since matrices are fairly universal to neural network representation.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With