PyTorch `torch.no_grad` vs `torch.inference_mode`

Question

PyTorch has new functionality torch.inference_mode as of v1.9 which is "analogous to torch.no_grad... Code run under this mode gets better performance by disabling view tracking and version counter bumps."

If I am just evaluating my model at test time (i.e. not training), is there any situation where torch.no_grad is preferable to torch.inference_mode? I plan to replace every instance of the former with the latter, and I expect to use runtime errors as a guardrail (i.e. I trust that any issue would reveal itself as a runtime error, and if it doesn't surface as a runtime error then I assume it is indeed preferable to use torch.inference_mode).

More details on why inference mode was developed are mentioned in the PyTorch Developer Podcast.

Answer 1

Yes, torch.inference_mode is indeed preferable to torch.no_grad in all situations where inference mode does not throw a runtime error. Check here.