Menu
What is the difference between the method that define layers in __init__() function, call layer in forward later and the method that directly use layer in forward() function ?
Should I define every layer in my compute graph in constructed function(eg. __init__) before I write my compute graph?
Could I direct define and use them in forward()?
774
The forward function computes output Tensors from input Tensors. The backward function receives the gradient of the output Tensors with respect to some scalar value, and computes the gradient of the input Tensors with respect to that same scalar value.
In the forward function, you define how your model is going to be run, from input to output.
A more elegant approach to define a neural net in pytorch. In the example above, fc stands for fully connected layer, so fc1 is represents fully connected layer 1, fc2 is the fully connected layer 2 and etc. Notice that when we print the model architecture the activation functions do not appear.
Everything which contains weights which you want to be trained during the training process should be defined in your __init__ method.
You don't need do define activation functions like softmax, ReLU or sigmoid in your __init__, you can just call them in forward.
Dropout layers for example also don't need to be defined in __init__, they can just be called in forward too. [However defining them in your __init__ has the advantage that they can be switched off easier during evaluation (by calling eval() on your model). You can see an example of both versions here.
Hope this is clear. Just ask if you have any further questions.
159
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
