Distributed AI Systems: A practical guide to building scalable training, inference, and serving systems for production AI

Scaling an AI model is not just a matter of adding GPUs. Memory limits, network communication, scheduling, and serving performance all shape whether a system works reliably beyond a single machine. Distributed AI Systems brings those connected engineering concerns into one practical guide, following the path from training workloads to production inference and deployment.

Understand the infrastructure beneath the model

Fuheng Wu begins with resource estimation, data preparation, GPU architecture, interconnects, and parallelism strategies. This grounding helps readers reason about why a workload needs distribution—and which constraints are likely to matter—before choosing a framework or scaling strategy.

Scale training with a clearer mental model

The training material moves through PyTorch DDP and FSDP, then into DeepSpeed ZeRO and Megatron-style approaches. Rather than treating these tools as disconnected names, the book places data parallelism, sharding, and other forms of parallel computation in the context of practical scaling and memory use.

Carry the system through inference and deployment

Training is only part of operating a large model. The book also explores distributed inference with vLLM and SGLang, including techniques related to attention, caching, batching, and request routing. Later sections address multi-node jobs with SLURM and production serving with Kubernetes, GPU scheduling, and observability.

Measure performance, not just activity

Benchmarking and optimization receive dedicated attention, giving readers a framework for investigating bottlenecks instead of relying on guesswork. The closing discussion looks toward topics such as mixture-of-experts architectures, edge–cloud coordination, and the evolving demands of distributed AI.

Who will find it useful?

This book is for engineers and technical practitioners working to train, serve, or support AI models at scale—including machine-learning engineers, platform and infrastructure teams, AI researchers, and HPC or cloud professionals. Familiarity with Python and PyTorch provides a useful foundation.

For readers who want to understand how hardware, software, and operations fit together in production AI, this is a systems-minded guide to the full journey.