why pytorch linear model isn't using sigmoid function

Question

I'm expecting that the linear model in pytorch yields sigmoid(WX+b). But I see it is only returning Wx+b. Why is this the case?

In Udacity "Intro to deep learning with pytorch" -> Lesson 2: Introduction to Neural Networks, They say that the output is sigmoid:

𝑦̂ =𝜎(𝑤1𝑥1+𝑤2𝑥2+𝑏)

From the below code I was expecting y cap to be 0.38391371665752183, but it is just the value of WX+b, which I confirmed the output. Why is this discrepancy?

import torch
from torch import nn
import numpy as np

torch.manual_seed(0)
model = nn.Linear(2,1)
w1 = model.weight.detach().numpy()
b1 = model.bias.detach().numpy()
print (f'model.weight = {w1}, model.bias={b1}')
x = torch.tensor([[0.2877, 0.2914]])
print(f'model predicted {model(x)}')
z = x.numpy()[0][0] * w1[0][0] + x.numpy()[0][1] * w1 [0][1] + b1[0]
print(f'manual multiplication yielded {z}')
ycap = 1/(1+ np.exp(-z))
print(f'y cap is {ycap}')

Output:

model.weight = [[-0.00529398  0.3793229 ]], model.bias=[-0.58198076]
model predicted tensor([[-0.4730]], grad_fn=<AddmmBackward>)
manual multiplication yielded -0.4729691743850708
y cap is 0.38391371665752183

Answer 1

The nn.Linear layer is a linear fully connected layer. It corresponds to wX+b, not sigmoid(WX+b).

As the name implies, it's a linear function. You can see it as a matrix multiplication (with or without a bias). Therefore it does not have an activation function (i.e. nonlinearities) attached.

If you want to append an activation function to it, you can do so by defining a sequential model:

model = nn.Sequential(
    nn.Linear(2, 1)
    nn.Sigmoid()
)

---

Edit - if you want to make sure:

x = torch.tensor([[0.2877, 0.2914]])
model = nn.Linear(2,1)
m1 = nn.Sequential(model, nn.Sigmoid())

m1(x)[0].item(), torch.sigmoid(model(x))[0].item()