Lilly Tech Systems
Designing Real-Time ML Inference
Build production-grade, low-latency ML serving infrastructure from the ground up. Learn to deploy model servers, optimize inference with quantization and compilation, implement dynamic batching, auto-scale GPU clusters, and safely roll out model updates — the complete playbook for engineers who ship ML to production.
Your Learning Path
Follow these lessons in order for a complete understanding of ML inference system design, or jump to any topic that interests you.
1. ML Inference Architecture Overview
Batch vs real-time vs near-real-time inference patterns. Latency requirements by use case (ads: 10ms, search: 100ms, chat: 1s). Inference server landscape and how to choose.
2. Model Server Design
TorchServe, Triton, vLLM, and TGI architecture deep-dives. Model loading, warm-up strategies, GPU memory management, multi-model serving, and production Triton config.
3. Inference Optimization Techniques
Quantization (INT8, FP16, GPTQ, AWQ), model distillation, TensorRT compilation, ONNX Runtime, speculative decoding for LLMs, and benchmarks with real numbers.
4. Request Batching & Routing
Dynamic batching, continuous batching for LLMs, model routing (small model first, escalate to large), load balancing strategies, and queue management.
5. Auto-Scaling GPU Infrastructure
Kubernetes GPU scheduling, scale-from-zero patterns, custom metrics (queue depth, GPU utilization), spot/preemptible instances, and cold start mitigation.
6. A/B Testing & Canary Deployments
Shadow deployments, traffic splitting, model performance comparison, rollback strategies, and statistical significance for model experiments.
7. Best Practices & Checklist
Inference optimization checklist, cost per request calculations, SLA design, monitoring essentials, and frequently asked questions.
What You'll Learn
By the end of this course, you will be able to:
Design Inference Pipelines
Architect end-to-end ML serving systems that meet strict latency SLAs — from model loading to response delivery — for real-time, near-real-time, and batch workloads.
Optimize for Production
Apply quantization, compilation, and batching techniques that cut inference latency by 2-10x and reduce GPU costs by 50-80% on real workloads.
Scale GPU Clusters
Configure Kubernetes-based GPU autoscaling with custom metrics, handle cold starts, and manage spot instances for cost-effective ML serving at scale.
Ship Models Safely
Deploy model updates using shadow deployments, canary releases, and A/B tests with proper statistical rigor — and roll back instantly when things go wrong.