Lilly Tech Systems
Designing AI Monitoring Systems
Build production-grade observability for ML systems from the ground up. This course covers the complete monitoring stack — from data drift detection and model performance tracking to LLM-specific monitoring, alerting, and dashboard design. Every lesson includes production code, real architecture patterns, and battle-tested strategies used by MLOps teams running models at scale.
Course Lessons
Follow the lessons in order or jump to any topic you need.
1. Why ML Monitoring is Different
Traditional monitoring vs ML monitoring, the 4 pillars (data, model, infrastructure, business), silent failures in ML, and real production incidents.
2. Data Drift Detection
Statistical tests (KS, PSI, chi-squared), feature drift monitoring, training-serving distribution comparison, drift detection code, and alerting thresholds.
3. Model Performance Monitoring
Online metrics tracking, ground truth delay handling, proxy metrics, performance degradation detection, A/B test monitoring, and metrics tracker code.
4. LLM-Specific Monitoring
Token usage tracking, latency monitoring, hallucination detection, cost per query, quality scoring, prompt performance tracking, and guardrail trigger rates.
5. Alerting & Incident Response
Alert design (severity, routing, dedup), runbooks for ML incidents, escalation procedures, PagerDuty/OpsGenie integration, and reducing alert fatigue.
6. Dashboard Design
Executive dashboards vs team dashboards, key metrics per role, Grafana dashboard templates, real-time vs historical views, and SLA tracking.
7. Best Practices & Checklist
Monitoring checklist by model type, tool comparison (Evidently, WhyLabs, Arize, custom), and comprehensive FAQ.