How are neurons in deeper layers capable of making more complex decisions than neurons in shallower/earlier layers?

Question

I'm brand new to ML and am reading the online book at http://neuralnetworksanddeeplearning.com.

In the first chapter the author describes a single perceptron using a Cheese Festival example. Basically he illustrates an example of a perceptron trying to decide whether or not to go to a Cheese Festival on the upcoming weekend. The perceptron has 3 inputs (a weather factor, a social factor and transportation factor). And while I completely understand 100% the author's "Cheese Festival" example for a single perceptron, I don't understand the next (seemingly non-sequitur) conclusion that he draws, where he evolves the example to contain multiple perceptrons organized into 2 layers:

"What about the perceptrons in the second layer? Each of those perceptrons is making a decision by weighing up the results from the first layer of decision-making. In this way a perceptron in the second layer can make a decision at a more complex and more abstract level than perceptrons in the first layer....In this way, a many-layer network of perceptrons can engage in sophisticated decision making."

What I don't understand is: Why are perceptrons in the 2nd layer able to "make a decision at a more complex and more abstract level" than their counterparts in the 1st layer?

Can someone provide an example, perhaps using the Cheese Festival example already started by this author? What is an example of a more complex/abstract decision that could be made by a perceptron in Layer 2?

---

Example

The top of the diagram (above) shows the decision-making perceptron behind the author's Cheese Festival example. The lone perceptron takes in 3 inputs/factors: weather, social and transportation factors. From these it calculates/determines whether one should go to the Cheese Festival or not.

The bottom of the diagram above (that is, underneath the horizontal line) illustrates perfectly where my mental hangup is. Here, we have 3 neurons in Layer 1 feeding into a single Layer 2 node:

• A "Should I go to the Cheese Festival" Node: this is the same node as in the top half of the diagram; and
• A "Should I go to the Milk Festival" Node: omitted in the diagram for simplicity/brevity
• A "Should I go to the Bacon Festival" Node: also omitted for simplicity/brevity

The outputs (Yes/No decisions) from these 3 perceptrons feed, as inputs, into the lone Layer 2 node. Given my example here, what is an example of a Layer 2 node decision that illustrates this concept of "feature hierarchy" and/or the author's statement that deeper layers can make more complex/abstract decisions?

Answer 1

The basic idea of multi layered networks is that each layer adds some amount of abstraction. The 2nd layer itself has the same structure as the 1st layer, therefore it cannot really make more complex decisions on its own. But it can build upon the abstractions (outputs) created the 1st layer.

I like to add that this statement is quite a bit idealized. In practice, it is often difficult to understand what exactly the inner layers do.

In your example, the decision made by the 2nd layer node could be "Should I go to a festival?"