Build Your Own Deep Learning Framework

Master Deep Learning by Building One

Author

Xiaoquan Kong

Published

December 30, 2025

Preface

You’ve used deep learning frameworks — PyTorch, TensorFlow, JAX, or whatever’s trending this month. But when something breaks, you’re staring at stack traces a mile deep, with no idea what went wrong.

The abstractions that made “getting started” easy now make debugging impossible. loss.backward() feels like magic. optimizer.step() is a black box. And when gradients explode or vanish, you have no idea where to look.

This book takes a different path: instead of teaching you to use a framework, we’ll build one together — from an empty file to a working GPT model.

What You’ll Build

By the end of this book, you’ll have built:

  • A Tensor class with automatic differentiation
  • Complete backpropagation through computational graphs
  • Optimizers: SGD, Momentum, Adam, AdamW
  • Neural network layers: Linear, Dropout, LayerNorm
  • A modular architecture with Module/Layer patterns
  • ONNX export for production deployment
  • GPU acceleration with cuNumeric
  • A working GPT model that generates text

Who This Book Is For

This book is for developers who:

  • Want to deeply understand how deep learning works
  • Are frustrated by “magic” in existing frameworks
  • Need to debug gradient issues and understand what’s happening
  • Want to build custom layers and operations
  • Are preparing for ML engineering interviews

The TensorWeaver Framework

Throughout this book, we’ll build TensorWeaver — a minimal, transparent, and debuggable deep learning framework.

About the Author

Xiaoquan Kong is a Google Developer Expert in Machine Learning and author of Conversational AI with Rasa (Packt, 2021). He spent a decade building AI systems at Alibaba and other leading tech companies. He holds a Master’s in AI from Duke University. He wrote this book because he was tired of treating loss.backward() as magic.

How to Read This Book

The book progresses from simple to complex, with each part building on the previous:

Part Chapters What You’ll Build Milestone
I: Forward Only 1-3 Tensors and operators Temperature converter (inference)
II: Learning to Learn 4-7 Autodiff and backprop Temperature converter (trained)
III: Smarter Training 8-11 Optimizers and schedulers Faster convergence
IV: Going Deeper 12-15 Activations, regularization Iris classifier
V: Clean Architecture 16-19 Modules and DataLoader Reusable components
VI: To Production 20-22 ONNX export Deployable models
VII: Hardware Acceleration 23-25 GPU with cuNumeric Fast training
VIII: The Grand Finale 26-31 Transformer and GPT Text generation

Learning path:

  • Parts I-III use temperature conversion (F = 1.8C + 32) as a running example — simple enough to debug, meaningful enough to understand
  • Part IV tackles Iris flower classification — your first multi-class problem
  • Parts V-VII focus on engineering: modularity, export, and performance
  • Part VIII brings it all together with GPT

Each chapter builds on the previous one, with working code you can run.

Prerequisites

  • Python 3.12+
  • Basic understanding of calculus (derivatives, chain rule)
  • Familiarity with NumPy
  • No deep learning experience required — we build from scratch

Getting Started

pip install tensorweaver

You’re ready to go. Let’s understand what a deep learning framework actually is in Chapter 1.

Conventions

Note

Code Reference: Points to source files in the TensorWeaver repository.

Tip

Checkpoint: Marks a working milestone. Each version is complete code you can run.

Let’s build something together.