How to understand "individual neurons are the basis directions of activation space"?

Question

In a recent article at Distill (link) about visualizing internal representation of convolutional neural networks, there is the following passage (bold is mine):

If neurons are not the right way to understand neural nets, what is? In real life, combinations of neurons work together to represent images in neural networks. Individual neurons are the basis directions of activation space, and it is not clear that these should be any more special than any other direction.

Szegedy et al.[11] found that random directions seem just as meaningful as the basis directions. More recently Bau, Zhou et al.[12] found basis directions to be interpretable more often than random directions. Our experience is broadly consistent with both results; we find that random directions often seem interpretable, but at a lower rate than basis directions.

I feel like they are talking about linear algebra representations, but struggle to understand how one neuron can represent a basis vector.

So at this point I have 2 main questions:

1. A neuron has only a scalar output, so how that can be a basic direction?
2. What is an activation space and how to intuitively think about it?

I feel like understanding these can really broaden my intuition about internal geometry of neural nets. Can someone please help by explaining or point me in the direction of understanding internal processes of neural nets from the linear algebra point of view?

Answer 1

My intuition would be: If you have a hidden layer with e.g. 10 neurons, then the activations of these 10 neurons span a 10-dimensional space. "Individual neurons are the basis directions of activation space" then means something like "the 10 states where exactly one neuron is 1 and the others are 0 are unit vectors that span this 'activation space'". But obviously, any independent set of 10 vectors spans the same space. And since a fully-connected layer is basically just a matrix product with the output of the previous layer, there's no obvious reason why these unit vectors should be special in any way.

This is important if you try to visualize what this hidden layer represents: Who says that "neuron 3" or the state "neuron 3 is active and the other neurons are 0" even does represent anything? It's equally possible that "neurons 2,3 and 5 are 1, neuron 7 is -2 and the others are 0" has a visual representation, but the unit vectors do not.

Ideally, you would hope that random vectors represent distinct concepts, because that way a hidden layer with n neurons can represent O(p^n) concepts (for some p > 1), instead of n concepts for n unit vectors