How to use autograd.gradcheck in PyTorch?

Question

The documentation does not include any example use case of gradcheck, where would it be useful?

Answer 1

There's an example use case provided in the documentation here:

https://pytorch.org/docs/master/notes/extending.html

You probably want to check if the backward method you implemented actually computes the derivatives of your function. It is possible by comparing with numerical approximations using small finite differences:

from torch.autograd import gradcheck

# gradcheck takes a tuple of tensors as input, check if your gradient
# evaluated with these tensors are close enough to numerical
# approximations and returns True if they all verify this condition.
input = (torch.randn(20,20,dtype=torch.double,requires_grad=True), torch.randn(30,20,dtype=torch.double,requires_grad=True))
test = gradcheck(linear, input, eps=1e-6, atol=1e-4)
print(test)

As the quote above suggests, the purpose of the gradcheck function is to verify that a custom backward function agrees with a numerical approximation of the gradient. The primary use case is when you're implementing a custom backward operation. In very few cases should you be implementing your own backward function in PyTorch. This is because PyTorch's autograd functionality takes care of computing gradients for the vast majority of operations.

The most obvious exceptions are

1. You have a function which can't be expressed as a finite combination of other differentiable functions (for example, if you needed the incomplete gamma function, you might want to write your own forward and backward which used numpy and/or lookup tables).

2. You're looking to speed up the computation of a particularly complicated expression for which the gradient could be drastically simplified after applying the chain rule.