Lilly Tech Systems
Best Practices
Production-ready patterns for cost optimization, monitoring, error handling, and reliable serverless AI inference deployments.
Cost Optimization
- Right-size memory: Profile your model's actual memory usage. Over-provisioning memory wastes money; under-provisioning causes OOM failures.
- Use ARM architectures: AWS Graviton and GCP Tau T2A offer 20-40% better price-performance for CPU inference.
- Batch requests: On Cloud Run, configure concurrency to batch multiple inference requests per instance rather than one-per-instance.
- Scale-to-zero wisely: If cold starts are acceptable, allow scaling to zero. If not, set minimum instances to the expected baseline traffic level.
- Monitor idle costs: Track the cost of provisioned concurrency vs actual request volume. If utilization drops below 30%, reduce warm capacity.
Monitoring and Observability
Inference Metrics
Track p50, p95, p99 latency, cold start frequency, error rates, and model prediction distribution to detect drift and performance issues.
Business Metrics
Monitor cost per prediction, throughput capacity, and SLA compliance. Set alerts when cost per prediction exceeds thresholds.
Model Health
Log input/output distributions to detect data drift. Compare prediction distributions against baseline to catch model degradation early.
Error Handling Patterns
Graceful Degradation
Return cached or default predictions when the model fails to load. Never return a 500 error for transient initialization failures.
Input Validation
Validate input shapes, types, and ranges before passing to the model. Return clear 400 errors for invalid inputs rather than cryptic model errors.
Timeout Management
Set inference timeouts shorter than the function timeout. This allows graceful error handling rather than abrupt function termination.
Dead Letter Queues
For async inference, configure dead letter queues to capture failed requests for retry or manual review.
Deployment Patterns
| Pattern | Description | When to Use |
|---|---|---|
| Blue/Green | Deploy new model version alongside old; switch traffic at once | Critical models where rollback speed matters |
| Canary | Route 5-10% of traffic to new version; gradually increase | Models where you need to validate accuracy in production |
| Shadow | Send requests to both versions; compare outputs without serving new version | High-risk model updates where accuracy must be validated first |