AttributeError: 'NoneType' object has no attribute 'zero_'

Question

The Grad sub object becomes "None" if expand the expression. Not sure why? Can somebody give some clue. If expand the w.grand.zero_() throw error as "AttributeError: 'NoneType' object has no attribute 'zero_'"

Thanks, Ganesh

import torch

x = torch.randint(size = (1,2), high = 10)
w = torch.Tensor([16,-14])
b = 36

y = w * x + b

epoch = 20
learning_rate = 0.01

w1 = torch.rand(size= (1,2), requires_grad= True)
b1 = torch.ones(size = [1], requires_grad= True)

for i in range(epoch):
    y1 = w1 * x + b1

    loss = torch.sum((y1-y)**2)

    loss.backward()

    with torch.no_grad():
        #w1 = w1 - learning_rate * w1.grad  //Not Working : w1.grad becomes "None" not sure how ;(
        #b1 = b1 - learning_rate * b1.grad  

        w1 -= (learning_rate * w1.grad)  // Working code.
        b1 -= (learning_rate * b1.grad)

        w1.grad.zero_()
        b1.grad.zero_()

    print("B ", b1)  
    print("W ", w1)

Answer 1

The thing is that in your working code you are modifying existing variable which has grad attribute, while in the non-working case you are creating a new variable.

As new w1/b1 variable is created it has no gradient attribute as you didn't call backward() on it, but on the "original" variable.

First, let's check whether that's really the case:

print(id(w1)) # Some id returned here
w1 = w1 - learning_rate * w1.grad

# In case below w1 address doesn't change
# w1 -= learning_rate * w1.grad 

print(id(w1)) # Another id here

Now, you could copy it in-place and not brake it, but there is no point to do so and your working case is much clearer, but for posterity's sake:

w1.copy_(w1 - learning_rate * w1.grad)