How are neural networks, loss and optimizer connected in PyTorch?

Question

I've seen answers to this question, but I still don't understand it at all. As far as I know, this is the most basic setup:

net = CustomClassInheritingFromModuleWithDefinedInitAndForward()
criterion = nn.SomeLossClass()
optimizer = optim.SomeOptimizer(net.parameters(), ...)
for _, data in enumerate(trainloader, 0):
    inputs, labels = data
    optimizer.zero_grad()
    outputs = net(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()

What I don't understand is:

Optimizer is initialized with net.parameters(), which I thought are internal weights of the net.

Loss does not access these parameters nor the net itself. It only has access to net's outputs and input labels.

Optimizer does not access loss either.

So if loss only works on outputs and optimizer only on net.parameters, how can they be connected?

Answer 1

Optimizer is initialized with net.parameters(), which I thought are internal weights of the net.

This is because the optimizer will modify the parameters of your net during the training.

Loss does not access these parameters nor the net itself. It only has access to net's outputs and input labels.

The loss only computes an error between a prediction and the truth.

Optimizer does not access loss either.

It accesses the tensors that were computed during loss.backward