2 Tensor Basics

Everything starts with tensors. Let’s build them from scratch.

2.1 What is a Tensor?

A tensor is simply an n-dimensional array:

import numpy as np

# Scalar (0D) - a single number
scalar = 3.14

# Vector (1D) - a list of numbers
vector = [1.0, 2.0, 3.0]

# Matrix (2D) - a table of numbers
matrix = [[1.0, 2.0],
          [3.0, 4.0]]

# 3D Tensor - a cube of numbers
tensor_3d = [[[1, 2], [3, 4]],
             [[5, 6], [7, 8]]]

In deep learning:

Scalars: Loss values, learning rates
Vectors: Biases, 1D embeddings
Matrices: Weight matrices, batch of vectors
3D+: Images, sequences, attention scores

2.2 Our First Tensor Class

Let’s create a minimal Tensor class:

import numpy as np

class Tensor:
    """A multi-dimensional array for deep learning."""

    def __init__(self, data):
        # Convert to NumPy array with float32
        if isinstance(data, np.ndarray):
            self.data = data.astype(np.float32)
        else:
            self.data = np.array(data, dtype=np.float32)

    @property
    def shape(self):
        """Return tensor dimensions."""
        return self.data.shape

    @property
    def ndim(self):
        """Return number of dimensions."""
        return self.data.ndim

    def __repr__(self):
        return f"Tensor({self.data.tolist()})"

Note

Code Reference: See src/tensorweaver/autodiff/tensor.py for the complete implementation.

2.3 Testing Our Tensor

# Create tensors
scalar = Tensor(3.14)
vector = Tensor([1.0, 2.0, 3.0])
matrix = Tensor([[1.0, 2.0], [3.0, 4.0]])

print(f"Scalar: {scalar}, shape: {scalar.shape}")
# Scalar: Tensor(3.14), shape: ()

print(f"Vector: {vector}, shape: {vector.shape}")
# Vector: Tensor([1.0, 2.0, 3.0]), shape: (3,)

print(f"Matrix: {matrix}, shape: {matrix.shape}")
# Matrix: Tensor([[1.0, 2.0], [3.0, 4.0]]), shape: (2, 2)

2.4 Temperature Data as Tensors

Let’s represent our temperature conversion data:

# Celsius temperatures (input)
celsius = Tensor([[0.0],
                  [20.0],
                  [37.0],
                  [100.0]])

print(f"Celsius shape: {celsius.shape}")  # (4, 1)

# Model parameters
w = Tensor([[1.8]])   # Weight: shape (1, 1)
b = Tensor([32.0])    # Bias: shape (1,)

print(f"Weight shape: {w.shape}")  # (1, 1)
print(f"Bias shape: {b.shape}")    # (1,)

Why these shapes?

celsius: (4, 1) = 4 samples, 1 feature each
w: (1, 1) = 1 input feature → 1 output feature
b: (1,) = 1 bias term

2.5 Essential Properties

Let’s add more useful properties:

class Tensor:
    # ... __init__ as before ...

    @property
    def size(self):
        """Total number of elements."""
        return self.data.size

    @property
    def dtype(self):
        """Data type."""
        return self.data.dtype

    def item(self):
        """Extract scalar value."""
        if self.size != 1:
            raise ValueError("item() only works for single-element tensors")
        return self.data.item()

    def numpy(self):
        """Convert to NumPy array."""
        return self.data.copy()

Usage:

t = Tensor([[1.0, 2.0], [3.0, 4.0]])

print(f"Size: {t.size}")      # 4
print(f"Dtype: {t.dtype}")    # float32
print(f"NumPy: {t.numpy()}")  # array([[1., 2.], [3., 4.]], dtype=float32)

scalar = Tensor(3.14)
print(f"Item: {scalar.item()}")  # 3.14

2.6 Reshaping

Tensors need to change shape for different operations:

class Tensor:
    # ... previous methods ...

    def reshape(self, *shape):
        """Return tensor with new shape."""
        return Tensor(self.data.reshape(*shape))

    def transpose(self, *axes):
        """Permute dimensions."""
        if not axes:
            axes = None  # Default: reverse all dimensions
        return Tensor(self.data.transpose(axes))

    @property
    def T(self):
        """Shorthand for 2D transpose."""
        return self.transpose()

Usage:

t = Tensor([[1.0, 2.0, 3.0],
            [4.0, 5.0, 6.0]])

print(f"Original shape: {t.shape}")  # (2, 3)

reshaped = t.reshape(3, 2)
print(f"Reshaped: {reshaped.shape}")  # (3, 2)

transposed = t.T
print(f"Transposed: {transposed.shape}")  # (3, 2)
print(transposed)
# Tensor([[1.0, 4.0],
#         [2.0, 5.0],
#         [3.0, 6.0]])

2.7 Why Not Just Use NumPy?

Good question! We wrap NumPy because:

Abstraction: Later we’ll add gradient tracking
Consistency: Unified API across the framework
Extensibility: Easy to add GPU backends later (Part VII)
Learning: Understanding what frameworks actually do

For now, Tensor is a thin wrapper. It will grow more powerful in Part II.

2.8 Summary

We’ve built a Tensor class that:

Wraps NumPy arrays
Has shape, ndim, size, dtype properties
Supports reshape and transpose
Can extract scalar values

Our temperature data is ready:

celsius = Tensor([[0.0], [20.0], [37.0], [100.0]])  # 4 samples
w = Tensor([[1.8]])                                  # weight
b = Tensor([32.0])                                   # bias

Next, we’ll implement operators to compute celsius @ w.T + b.