Advanced

Best Practices & Checklist

This final lesson compiles everything from the course into a production-ready checklist, documents the most common failure modes, and provides debugging techniques for when things go wrong. Bookmark this page — you will reference it every time you ship an LLM feature.

Production LLM Checklist

Use this checklist before launching any LLM-powered feature. Items are ordered by priority — address the top items first.

Security & Safety (Must-Have)

• Prompt injection protection: Pattern-based + LLM-based detection on all user inputs
• PII redaction: Redact emails, SSNs, credit cards, phone numbers before sending to LLM API
• Output content policy: Validate responses against your content policy before showing to users
• System prompt protection: Never put secrets, API keys, or sensitive logic in system prompts
• Rate limiting: Per-user and per-API-key rate limits to prevent abuse
• Input length limits: Cap input size to prevent token abuse and cost spikes

Reliability (Must-Have)

• Fallback providers: Configure at least one fallback LLM provider (e.g., Anthropic if primary is OpenAI)
• Timeout handling: Set timeouts on all LLM calls (typically 30-60 seconds)
• Retry with backoff: Retry transient failures (429, 500, 503) with exponential backoff
• Graceful degradation: Define fallback behavior when all LLM providers are down
• Error handling: Never expose raw LLM errors to users. Map to user-friendly messages.
• Output format validation: Parse and validate structured outputs (JSON) before using them

Quality (Should-Have)

• Evaluation suite: At least 10-20 test cases covering critical flows
• Regression testing: Run evaluation suite on every prompt change
• Prompt versioning: Version control all prompts with rollback capability
• A/B testing: Test prompt changes on a small traffic percentage before full rollout
• Human evaluation: Periodic human review of sampled responses (weekly or bi-weekly)

Cost & Observability (Should-Have)

• Cost tracking: Per-request cost logging with team, feature, and model breakdowns
• Budget alerts: Alerts when daily or weekly spend exceeds thresholds
• Model routing: Route simple tasks to cheap models
• Semantic caching: Cache responses for semantically similar queries
• Logging: Log prompts, responses, latency, token counts, and costs (with PII redacted)
• Monitoring dashboard: Track latency, error rates, cost, and quality metrics

User Experience (Nice-to-Have)

• Streaming responses: Stream long responses to reduce perceived latency
• Conversation memory: Maintain context across messages
• Feedback collection: Thumbs up/down on responses for quality monitoring
• Transparency: Indicate when responses are AI-generated

Common Failure Modes

These are the failures you will encounter in production. Learn to recognize and fix them before your users report them.

Failure Mode	Symptoms	Root Cause	Fix
Prompt injection	Model ignores system prompt, reveals instructions, changes persona	No input validation	Add guardrails layer (Lesson 4)
Hallucination	Model states false facts confidently	No grounding data or weak prompt	Add RAG, explicit "I don't know" instructions, fact-checking
Context overflow	Model loses early conversation context, gives inconsistent answers	Conversation exceeds context window	Summary or vector memory (Lesson 5)
Cost spike	Monthly bill jumps 3-10x unexpectedly	No rate limiting, no model routing, default max_tokens	Add gateway with cost tracking (Lessons 3, 7)
Format errors	App crashes parsing LLM output as JSON	No output validation, model returns invalid format	Add format validation, retry on parse failure, use response_format
Latency spikes	Response times go from 2s to 30s+	Provider degradation, no caching, large prompts	Semantic cache, streaming, fallback providers, prompt compression
Prompt regression	Quality drops after a prompt update	No regression testing	Evaluation suite in CI/CD (Lesson 6)

Debugging LLM Issues

When your LLM application behaves unexpectedly, follow this debugging flowchart:

# LLM Debugging Flowchart
#
# 1. Is the issue with INPUT or OUTPUT?
#    |
#    |- INPUT: Check the full prompt being sent
#    |  |- Is the system prompt correct? (check prompt version)
#    |  |- Is user input being properly sanitized?
#    |  |- Is context/memory being injected correctly?
#    |  |- Is the prompt too long? (check token count)
#    |
#    |- OUTPUT: Check the raw LLM response
#       |- Is the model returning valid format?
#       |- Is the content accurate?
#       |- Is the response being post-processed incorrectly?
#       |- Did the guardrails modify the response?
#
# 2. Is the issue CONSISTENT or INTERMITTENT?
#    |
#    |- CONSISTENT: Likely a prompt or configuration issue
#    |  |- Reproduce with temperature=0
#    |  |- Compare prompt with last known good version
#    |  |- Check if model version changed
#    |
#    |- INTERMITTENT: Likely a non-determinism or context issue
#       |- Log full request/response for failing cases
#       |- Check if context/memory varies between requests
#       |- Check if rate limiting is causing fallback to different model
#
# 3. Quick fixes to try:
#    |- Set temperature=0 for debugging
#    |- Print the FULL prompt (system + user + context)
#    |- Test the same prompt in the provider's playground
#    |- Revert to the last known good prompt version
#    |- Check provider status page for outages

Production Architecture Checklist Code

Here is a validation function that checks your LLM application against the production checklist:

def validate_production_readiness(config: dict) -> dict:
    """Check if an LLM application meets production standards."""
    checks = []

    # Security checks
    checks.append({
        "category": "Security",
        "check": "Prompt injection detection",
        "status": "pass" if config.get("guardrails_enabled") else "FAIL",
        "priority": "critical"
    })
    checks.append({
        "category": "Security",
        "check": "PII redaction",
        "status": "pass" if config.get("pii_redaction") else "FAIL",
        "priority": "critical"
    })
    checks.append({
        "category": "Security",
        "check": "Rate limiting",
        "status": "pass" if config.get("rate_limit_per_user") else "FAIL",
        "priority": "critical"
    })

    # Reliability checks
    checks.append({
        "category": "Reliability",
        "check": "Fallback providers configured",
        "status": "pass" if len(config.get("fallback_models", [])) >= 1 else "FAIL",
        "priority": "high"
    })
    checks.append({
        "category": "Reliability",
        "check": "Request timeout set",
        "status": "pass" if config.get("timeout_seconds", 0) > 0 else "FAIL",
        "priority": "high"
    })
    checks.append({
        "category": "Reliability",
        "check": "Output format validation",
        "status": "pass" if config.get("output_validation") else "warn",
        "priority": "high"
    })

    # Quality checks
    checks.append({
        "category": "Quality",
        "check": "Evaluation test suite",
        "status": "pass" if config.get("eval_test_count", 0) >= 10 else "warn",
        "priority": "medium"
    })
    checks.append({
        "category": "Quality",
        "check": "Prompt versioning",
        "status": "pass" if config.get("prompt_versioning") else "warn",
        "priority": "medium"
    })

    # Cost checks
    checks.append({
        "category": "Cost",
        "check": "Cost tracking enabled",
        "status": "pass" if config.get("cost_tracking") else "warn",
        "priority": "medium"
    })
    checks.append({
        "category": "Cost",
        "check": "Model routing (cheap model for simple tasks)",
        "status": "pass" if config.get("model_routing") else "warn",
        "priority": "medium"
    })
    checks.append({
        "category": "Cost",
        "check": "Budget alerts configured",
        "status": "pass" if config.get("budget_alert_usd") else "warn",
        "priority": "low"
    })

    # Summary
    failures = [c for c in checks if c["status"] == "FAIL"]
    warnings = [c for c in checks if c["status"] == "warn"]
    passed = [c for c in checks if c["status"] == "pass"]

    return {
        "ready_for_production": len(failures) == 0,
        "passed": len(passed),
        "warnings": len(warnings),
        "failures": len(failures),
        "total_checks": len(checks),
        "critical_failures": [c for c in failures if c["priority"] == "critical"],
        "details": checks
    }


# Usage
result = validate_production_readiness({
    "guardrails_enabled": True,
    "pii_redaction": True,
    "rate_limit_per_user": 100,
    "fallback_models": ["claude-sonnet", "gpt-4o-mini"],
    "timeout_seconds": 30,
    "output_validation": True,
    "eval_test_count": 15,
    "prompt_versioning": True,
    "cost_tracking": True,
    "model_routing": True,
    "budget_alert_usd": 500
})

print(f"Production ready: {result['ready_for_production']}")
print(f"Score: {result['passed']}/{result['total_checks']} passed, {result['warnings']} warnings")

Frequently Asked Questions

Course Summary

You now have the complete toolkit for building production LLM applications: