What is the output of fully connected layer in CNN?

Question

For example, in Caffe, one should define num_output in an Inner Product (Fully Connected) layer. What is the meaning of this output number?

Answer 1

Consider fully connect layer as a simple matrix-matrix multiplication of 1xN and NxM to produce a result of dimension 1xM.

Let us consider that we pass a data of dimension say 56x56x3 as the input of a fully connected layer. Let the dimension of the weight be unknown NxM. Consider, we set num_ouput = 4096.

For computing these data, the fully connected layer reshapes the input data of dimension 56x56x3 as 1xN, 1x(56x56x3) = 1x9408.

Thus,

N = 9408

M=num_output=4096

In effect we end up doing a (1x9408)matrix - (9408x4096) matrix multiplication.

If the num_output value was changed to say 100, it would end up doing (1x9408)matrix - (9408x100) matrix multiplication.

Thus increasing the num_ouput value will increase the number of weight parameters that the model has to learn.