Autodiff Engine

Auto-differetiating is a technique that allows you to compute the derivative of a function with respect to its input. It's a powerful tool for training neural networks. Every modern deep learning framework has its own autodiff engine.

Technically speaking, autodiff engine is not hard to implement. It contains two pass:

• Forward pass: Compute the output of the model. Some intermediate variables are recorded for backward pass.
• Backward pass: Compute the gradient of the loss with respect to the input. In other point of view, it's a reverse mode of forward pass.

We will discuss the details of autodiff engine in later chapters. To reveal the mystery of autodiff engine, we will show how TensorWeaver implements its autodiff engine.