Lilly Tech Systems
Designing ML Training Pipelines
Build production-grade, reproducible ML training infrastructure from scratch. Learn to orchestrate data preparation at scale, distribute training across GPU clusters, track experiments systematically, schedule GPU resources efficiently, and automate retraining with CI/CD — the complete engineering playbook for teams that train models in production.
Your Learning Path
Follow these lessons in order for a complete understanding of ML training pipeline design, or jump to any topic that interests you.
1. Training Pipeline Architecture
End-to-end pipeline components, orchestration tools comparison (Airflow, Kubeflow, Prefect, Dagster), pipeline vs notebook training, and reproducibility requirements.
2. Data Preparation at Scale
Data validation with Great Expectations and TFX, data splits strategy, augmentation pipelines, handling imbalanced datasets, and streaming data prep with production code.
3. Distributed Training Design
Data parallelism (DDP, FSDP), model parallelism, pipeline parallelism, DeepSpeed ZeRO stages, multi-node training on Kubernetes, and real GPU utilization numbers.
4. Experiment Tracking & Model Registry
MLflow and W&B architecture, experiment comparison, model versioning, artifact storage, promotion workflows, and production integration code.
5. GPU Cluster Scheduling
Kubernetes GPU scheduling, job queuing with Volcano and Kueue, fair-share scheduling, preemption policies, multi-tenant GPU clusters, and cost allocation.
6. CI/CD for ML
Automated retraining triggers, model validation gates, integration testing for models, deployment automation, rollback strategies, and GitHub Actions/Argo workflows.
7. Best Practices & Checklist
Training pipeline checklist, cost optimization strategies, debugging training failures, and frequently asked questions about production ML training.
What You'll Learn
By the end of this course, you will be able to:
Design Training Pipelines
Architect end-to-end ML training systems with proper orchestration, data validation, and reproducibility guarantees that survive team growth and model iteration.
Scale Training to Clusters
Distribute training across multi-GPU and multi-node setups using DDP, FSDP, and DeepSpeed — with real performance numbers and configuration for each strategy.
Manage Experiments at Scale
Track hundreds of experiments, compare results systematically, version models through staging to production, and build promotion workflows your team can trust.
Automate the Full Lifecycle
Build CI/CD pipelines that automatically validate data, retrain models on schedule or drift detection, run integration tests, and deploy with confidence.