Best Practices & Checklist Advanced

This lesson consolidates everything from the course into actionable best practices, a production readiness checklist, voice UX design principles, testing strategies, and accessibility requirements. Use this as your reference guide when building and shipping voice AI systems.

Voice UX Design Principles

# The 10 Rules of Voice UX Design

VOICE_UX_RULES = {
    1: {
        "rule": "Front-load the answer",
        "bad": "Based on your account details and recent transactions, "
               "your current available balance as of today is $1,234.56.",
        "good": "Your balance is $1,234.56.",
        "why": "Users tune out after the first few seconds. "
               "Lead with the information they asked for."
    },
    2: {
        "rule": "Maximum 3 options per menu",
        "bad": "You can check your balance, make a payment, transfer funds, "
               "view recent transactions, update your profile, or speak to an agent.",
        "good": "I can check your balance, make a payment, or help with something else. "
               "What would you like?",
        "why": "Working memory holds 3-4 items. More options = users forget the first one."
    },
    3: {
        "rule": "Confirm by restating, not by asking",
        "bad": "You said you want to transfer fifty dollars to your savings account. "
               "Is that correct? Please say yes or no.",
        "good": "I'll transfer fifty dollars to savings. Sound good?",
        "why": "Natural confirmations are faster and feel less robotic."
    },
    4: {
        "rule": "Use progressive disclosure",
        "bad": "Your account number is 1234567890, your routing number is 021000021, "
               "your balance is $5,432.10, your last transaction was...",
        "good": "Your balance is $5,432.10. Would you like to hear more details?",
        "why": "Give the minimum needed info first. Let users ask for more."
    },
    5: {
        "rule": "Fill silence with context",
        "bad": "[3 seconds of silence while looking up data]",
        "good": "Let me pull up your account... [1 second] Here it is.",
        "why": "Silence on a phone call means 'disconnected' to most users."
    },
    6: {
        "rule": "Match formality to context",
        "bad": "Greetings, valued customer. How may I be of assistance today?",
        "good": "Hi there. How can I help?",
        "why": "Overly formal voice AI feels fake. Match the tone your users expect."
    },
    7: {
        "rule": "Always provide an escape hatch",
        "bad": "[No way to reach a human or exit the flow]",
        "good": "Say 'agent' at any time to speak with a person.",
        "why": "Users panic when they feel trapped in an automated system."
    },
    8: {
        "rule": "Handle 'I don't know' gracefully",
        "bad": "Invalid input. Please try again.",
        "good": "No problem. Let me ask a different way. Are you calling about "
               "a recent order, or something else?",
        "why": "Users often don't know how to phrase their request. Guide them."
    },
    9: {
        "rule": "Keep responses under 15 seconds",
        "bad": "[30-second monologue explaining all account options]",
        "good": "[5-second focused answer with option to hear more]",
        "why": "Anything over 15 seconds and users stop listening."
    },
    10: {
        "rule": "Test with real humans, not just scripts",
        "bad": "Only testing with predefined test cases",
        "good": "Weekly user testing with 5 real callers",
        "why": "Real users say things you never imagined. Scripts miss edge cases."
    }
}

Production Readiness Checklist

# Voice AI Production Readiness Checklist
# Check each item before going live

PRODUCTION_CHECKLIST = {
    "Pipeline": [
        "[ ] ASR provider configured with fallback",
        "[ ] ASR custom vocabulary for domain terms",
        "[ ] TTS voice selected and tested across devices",
        "[ ] TTS pre-cached for top 20 common responses",
        "[ ] Dialog manager handles all expected intents",
        "[ ] Fallback response for unrecognized input",
        "[ ] End-to-end latency under 1500ms (P95)",
        "[ ] Streaming enabled for ASR and TTS",
    ],
    "Error Handling": [
        "[ ] Graduated error recovery (3 levels)",
        "[ ] No-speech-detected handling with retries",
        "[ ] Low-confidence handling with clarification",
        "[ ] API failure handling with graceful degradation",
        "[ ] Timeout handling at every pipeline stage",
        "[ ] User can always reach a human agent",
    ],
    "Telephony (if applicable)": [
        "[ ] Phone number provisioned and tested",
        "[ ] DTMF handling for key commands (0=agent, *=cancel)",
        "[ ] Call recording with consent announcement",
        "[ ] Warm transfer to agents with context",
        "[ ] Hold music / hold messaging configured",
        "[ ] After-hours message and voicemail",
    ],
    "Security & Compliance": [
        "[ ] PII redaction in transcripts before storage",
        "[ ] Recording consent compliant with jurisdiction",
        "[ ] Data retention policy with automated deletion",
        "[ ] Authentication flow for sensitive actions",
        "[ ] Audit logging for all actions",
        "[ ] No sensitive data in logs or error messages",
    ],
    "Monitoring & Observability": [
        "[ ] Latency tracking per pipeline stage",
        "[ ] ASR confidence score monitoring",
        "[ ] Error rate dashboards",
        "[ ] Alert on latency > 2000ms (P95)",
        "[ ] Alert on ASR confidence < 0.80 (average)",
        "[ ] Alert on error rate > 5%",
        "[ ] Call volume and concurrent call monitoring",
        "[ ] Audio sample storage for QA (1% sample)",
    ],
    "Testing": [
        "[ ] 50+ golden test conversations with expected outcomes",
        "[ ] Noise/accent testing with diverse audio samples",
        "[ ] Load testing for target concurrent calls",
        "[ ] Barge-in / interruption testing",
        "[ ] Long conversation testing (20+ turns)",
        "[ ] Edge cases: silence, background noise, multiple speakers",
        "[ ] Red-team testing for prompt injection via voice",
    ],
    "Accessibility": [
        "[ ] DTMF alternative for every voice command",
        "[ ] Adjustable speaking rate",
        "[ ] Clear enunciation (avoid fast speech)",
        "[ ] Spell-out option for codes and numbers",
        "[ ] TTY/TDD compatibility (if telephony)",
        "[ ] Multi-language support or clear language routing",
    ]
}

Testing Voice AI Systems

import asyncio
from dataclasses import dataclass
from typing import List, Optional

@dataclass
class VoiceTestCase:
    """A single voice system test case."""
    name: str
    audio_file: str                        # Path to test audio
    expected_transcript: str               # What ASR should produce
    expected_intent: str                   # What NLU should classify
    expected_response_contains: List[str]  # Keywords in response
    max_latency_ms: int = 2000             # Latency SLA
    asr_min_confidence: float = 0.80       # Minimum ASR confidence

@dataclass
class VoiceTestResult:
    test_name: str
    passed: bool
    actual_transcript: str
    actual_intent: str
    actual_response: str
    latency_ms: float
    asr_confidence: float
    failures: List[str]

class VoiceTestRunner:
    """Automated testing framework for voice AI systems.

    Runs test audio through the full pipeline and validates:
    - ASR accuracy (transcript matches expected)
    - NLU accuracy (intent matches expected)
    - Response quality (contains expected keywords)
    - Latency (within SLA)
    """

    def __init__(self, pipeline):
        self.pipeline = pipeline
        self.results: List[VoiceTestResult] = []

    async def run_test(self, test: VoiceTestCase) -> VoiceTestResult:
        """Run a single test case."""
        failures = []

        # Load test audio
        with open(test.audio_file, "rb") as f:
            audio = f.read()

        # Run through pipeline
        import time
        t0 = time.monotonic()
        result = await self.pipeline.process_audio(audio)
        latency_ms = (time.monotonic() - t0) * 1000

        # Validate ASR
        transcript_similarity = self._text_similarity(
            result.transcript, test.expected_transcript
        )
        if transcript_similarity < 0.85:
            failures.append(
                f"ASR mismatch: expected '{test.expected_transcript}', "
                f"got '{result.transcript}' (similarity: {transcript_similarity:.2f})"
            )

        if result.asr_confidence < test.asr_min_confidence:
            failures.append(
                f"ASR confidence too low: {result.asr_confidence:.2f} "
                f"(min: {test.asr_min_confidence})"
            )

        # Validate NLU
        if result.intent != test.expected_intent:
            failures.append(
                f"Intent mismatch: expected '{test.expected_intent}', "
                f"got '{result.intent}'"
            )

        # Validate response content
        for keyword in test.expected_response_contains:
            if keyword.lower() not in result.response.lower():
                failures.append(
                    f"Response missing keyword: '{keyword}'"
                )

        # Validate latency
        if latency_ms > test.max_latency_ms:
            failures.append(
                f"Latency exceeded: {latency_ms:.0f}ms > {test.max_latency_ms}ms"
            )

        test_result = VoiceTestResult(
            test_name=test.name,
            passed=len(failures) == 0,
            actual_transcript=result.transcript,
            actual_intent=result.intent,
            actual_response=result.response,
            latency_ms=latency_ms,
            asr_confidence=result.asr_confidence,
            failures=failures
        )
        self.results.append(test_result)
        return test_result

    async def run_suite(self, tests: List[VoiceTestCase]) -> dict:
        """Run a full test suite and return summary."""
        for test in tests:
            await self.run_test(test)

        passed = sum(1 for r in self.results if r.passed)
        failed = sum(1 for r in self.results if not r.passed)
        avg_latency = sum(r.latency_ms for r in self.results) / len(self.results)

        return {
            "total": len(self.results),
            "passed": passed,
            "failed": failed,
            "pass_rate": passed / len(self.results) * 100,
            "avg_latency_ms": avg_latency,
            "failures": [
                {"test": r.test_name, "issues": r.failures}
                for r in self.results if not r.passed
            ]
        }

    def _text_similarity(self, a: str, b: str) -> float:
        """Simple word overlap similarity for transcript comparison."""
        words_a = set(a.lower().split())
        words_b = set(b.lower().split())
        if not words_a or not words_b:
            return 0.0
        intersection = words_a & words_b
        union = words_a | words_b
        return len(intersection) / len(union)


# --- Example test suite ---
GOLDEN_TESTS = [
    VoiceTestCase(
        name="balance_check",
        audio_file="tests/audio/check_balance.wav",
        expected_transcript="what is my account balance",
        expected_intent="check_balance",
        expected_response_contains=["balance", "$"],
        max_latency_ms=1500
    ),
    VoiceTestCase(
        name="transfer_money",
        audio_file="tests/audio/transfer_savings.wav",
        expected_transcript="transfer fifty dollars to my savings account",
        expected_intent="transfer_funds",
        expected_response_contains=["transfer", "fifty", "savings", "confirm"],
        max_latency_ms=2000
    ),
    VoiceTestCase(
        name="noisy_environment",
        audio_file="tests/audio/noisy_balance_check.wav",
        expected_transcript="what is my balance",
        expected_intent="check_balance",
        expected_response_contains=["balance"],
        max_latency_ms=2000,
        asr_min_confidence=0.70  # Lower threshold for noisy audio
    ),
]

Accessibility Guidelines

Frequently Asked Questions

Architecture Summary

# Complete Voice AI Platform Architecture (simplified)

# 1. Audio arrives via phone (Twilio) or browser (WebRTC/WebSocket)
# 2. Audio preprocessed: noise filter, volume normalization, VAD
# 3. Streaming ASR converts audio to text in real-time
#    - Partial results for responsiveness
#    - Final result triggers NLU
# 4. NLU classifies intent and extracts entities
#    - Fast classifier (GPT-4o-mini) for simple intents
#    - Full LLM for complex understanding
# 5. Dialog manager generates response:
#    - Check cached responses first
#    - Route to intent handler or LLM
#    - Apply confirmation level based on confidence + risk
# 6. TTS converts response to audio:
#    - Check TTS cache first (saves 150-500ms)
#    - Stream TTS for long responses
#    - Match voice/tone to context
# 7. Audio streamed back to user via same channel
# 8. Metrics recorded: latency per stage, ASR confidence, intent, outcome
# 9. Session state updated in Redis
# 10. If transfer needed: warm transfer with context to human agent
# 11. Recording stored with PII redaction, scheduled for deletion