what is a 'layer' in a neural network

Question

Below I've drawn a typical feed forward neural network:

Now my question is, as far as lingo goes, what is a layer?

Could each individual process (rectangle) be considered a layer? or is a layer the combination a single row of the flow diagram? I sometimes see the Multiply + Add as a single layer, and the nonlinear function (relu) as a separate layer. But I would really appreciate a definitive answer.

I often find on online videos teaching people about Neural Networks, the instructors themselves mix up the number of layers within a single example.

Answer 1

With your diagram, each row is essentially a layer. But as @beaker states it is not the best way to visualize a neural network.

Taking an image from here will help make this clear.

Layer is a general term that applies to a collection of 'nodes' operating together at a specific depth within a neural network.

The input layer is contains your raw data (you can think of each variable as a 'node').

The hidden layer(s) are where the black magic happens in neural networks. Each layer is trying to learn different aspects about the data by minimizing an error/cost function. The most intuitive way to understand these layers is in the context of 'image recognition' such as a face. The first layer may learn edge detection, the second may detect eyes, third a nose, etc. This is not exactly what is happening but the idea is to break the problem up in to components that different levels of abstraction can piece together much like our own brains work (hence the name 'neural networks').

The output layer is the simplest, usually consisting of a single output for classification problems. Although it is a single 'node' it is still considered a layer in a neural network as it could contain multiple nodes.