AI Monitoring Best Practices Advanced

This final lesson consolidates the course into production-ready monitoring patterns. These best practices ensure your monitoring stack is reliable, scalable, and cost-effective as your ML platform grows from a handful of GPUs to hundreds or thousands.

High Availability for Monitoring

• Run Prometheus in HA pairs — Two identical Prometheus instances scraping the same targets ensure no monitoring gaps during upgrades or failures
• Deploy Alertmanager in a cluster — Use Alertmanager's built-in clustering to deduplicate alerts across HA Prometheus instances
• Monitor the monitoring — Use a separate lightweight Prometheus to monitor your primary monitoring stack (meta-monitoring)
• Grafana HA — Run multiple Grafana replicas with a shared database for dashboard state

Long-Term Metrics Storage

Monitoring as Code

Treat your entire monitoring configuration as code, version-controlled and deployed through GitOps:

• Prometheus rules — Store recording and alerting rules in Git; deploy via PrometheusRule CRDs
• Grafana dashboards — Store as JSON or Grafonnet; deploy via GrafanaDashboard CRDs or provisioning
• Alertmanager config — Store routing trees and receivers in Git; deploy via AlertmanagerConfig CRDs
• ServiceMonitor/PodMonitor — Define scrape targets declaratively alongside the applications they monitor

Cardinality Management

ML workloads can generate high-cardinality metrics (unique label combinations) that overwhelm Prometheus:

• Avoid per-sample labels — Do not use individual data sample IDs as metric labels
• Use recording rules — Pre-aggregate metrics to reduce query-time cardinality
• Set metric relabeling — Drop unnecessary labels before ingestion
• Monitor cardinality — Track prometheus_tsdb_head_series and alert when it grows unexpectedly