Skip to main content

Implementation of Tensor

What is Tensor?​

Tensor is a data container that stores the data and data's gradient and computational graph position (which operator is used to compute the data).

In deep learning, the tensor is the data flow from one operator to another operator. Connect all the operators together to form the computational graph. Similar to edges between nodes in the graph theory.

Tensor's Key Attributes​

the data attribute​

the data stored in the tensor. In TensorWeaver, the data attribute is a numpy array.

the grad attribute​

the gradient of the data, which is computed by the backward pass. In TensorWeaver, the grad attribute is a numpy array.

the creator attribute​

the operator instance used to compute the data. This will be used in the backward pass to compute the gradient of the data.

Compatibility Note: Pytorch using different attributes to store the backward information.

Implement Tensor​

The source code of the Tensor class can be found in the tensor.py file.

Here is the simplified version of the Tensor class:

class Tensor:
    def __init__(self, data, grad=None, creator=None):
        self.data = data
        self.grad = grad
        self.creator = creator

Usage of Tensor​

There are three ways to create a tensor: Directly from data, Generate by a helper function, Generate by an operator.

Directly from data​

Directly construct a tensor by using Tensor class is not recommended. It is not a good practice to use the Tensor class directly. User should use the tensor function to create a tensor.

a = torch.tensor(1.0)

In this case, the a tensor is created directly from the data 1.0. There is no creator for this tensor.

Generate by a helper function​

a = torch.ones(3, 4)

In this case, the a tensor is created by the torch.ones function. The torch.ones function is a helper function that creates a tensor.

Helper functions are not operators. They are just functions that create tensors. They are not used in the computational graph.

  1. Generate by an operator
a = torch.tensor(1.0)
b = torch.tensor(2.0)
c = torch.add(a, b)

In this case, the c tensor is the output of the torch.add operator. The torch.add operator takes the a and b tensors as input and returns the c tensor as output. So this torch.add operator instance is the creator of the c tensor. This fact can be checked by:

c.creator
<__main__.Add object at 0x100000000>