Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

What does "sparse" mean in the context of neural nets?

I've seen "sparse" and "sparsity" used in a way that suggests it's something that improves a model's accuracy. For example:

I think the unsupervised phase might be not so important if some sparse connections or neurons are used, such as rectifier units or convolutional connection, and big training data is available.

From https://www.quora.com/When-does-unsupervised-pre-training-improve-classification-accuracy-for-a-deep-neural-network-When-does-it-not

What does "sparse" mean in this context?

like image 756
Leo Jiang Avatar asked Dec 26 '16 01:12

Leo Jiang


People also ask

What is a sparse neural network?

We define a sparse neural network as a network in which only a percentage of the possible connections exists. You can imagine a fully connected layer with some of the connections missing.

What does sparse mean in ML?

Definition: A set of numbers (e.g. vector, matrix, etc.), is considered sparse when a high percentage of the values are assigned a constant default value.

What does sparsity mean in deep learning?

In AI inference and machine learning, sparsity refers to a matrix of numbers that includes many zeros or values that will not significantly impact a calculation.

What does sparse mean in programming?

Sparse coding is the representation of items by the strong activation of a relatively small set of neurons. For each stimulus, this is a different subset of all available neurons.


Video Answer


1 Answers

TL;DR: Sparsity means most of the weights are 0. This can lead to an increase in space and time efficiency.


Detailed version: In general, neural networks are represented as tensors. Each layer of neurons is represented by a matrix. Each entry in the matrix can be thought of as representative of the connection between two neurons. In a simple neural network, like a classic feed-forward neural network, every neuron on a given layer is connected to every neuron on the subsequent layer. This means that each layer must have n2 connections represented, where n is the size of both of the layers. In large networks, this can take a lot of memory and time to propagate. Since different parts of a neural network often work on different subtasks, it can be unnecessary for every neuron to be connected to every neuron in the next layer. In fact, it might make sense for a neural network to have most pairs of neurons with a connection weight of 0. Training a neural network might result in these less significant connection weights adopting values very close to 0 but accuracy would not be significantly affected if the values were exactly 0.

A matrix in which most entries are 0 is called a sparse matrix. These matrices can be stored more efficiently and certain computations can be carried out more efficiently on them provided the matrix is sufficiently large and sparse. Neural networks can leverage the efficiency gained from sparsity by assuming most connection weights are equal to 0.

I must say that neural networks are a complex and diverse topic. There are a lot of approaches used. There are certain kinds of neural networks with different morphologies than the simple layer connections I referenced above. Sparsity can be leveraged in many types of neural networks since matrices are fairly universal to neural network representation.

like image 152
morsecodist Avatar answered Oct 14 '22 03:10

morsecodist