On Pytorch Tensors and Autograd
Somehow, Pytorch blitz tutorial on Autograd completely confused me. I could not understand what does
Fortunately, I found an excellent explanation of Autograd and Computational Graph is here: https://blog.paperspace.com/pytorch-101-understanding-graphs-and-automatic-differentiation/. Just for my notes and anyone interested, I am going to leave my short recap here:
- Computational Graph - records the order of operations on tensors in the graph. Edges of the graph represent the local gradients. Leafs of the graph are independent variables (inputs and weights/biases in case of NN)
tensor.backward()computes the gradients all the way back through the computational graph and accumulate results in leafs. Can only be called on the 0-rank tensor.
tensor.gradholds the accumulated gradient from the call to
.backward()with respect to the given tensor