Lilly Tech Systems
Model Governance Audit Trails
Build comprehensive, tamper-proof audit trail systems that capture every model-related decision, change, and event to satisfy regulatory requirements and enable forensic analysis.
What to Capture
| Event Category | Examples | Retention Period |
|---|---|---|
| Lifecycle Events | Stage transitions, approvals, rejections | Model lifetime + 7 years |
| Configuration Changes | Feature changes, threshold updates, retraining | Model lifetime + 5 years |
| Access Events | Who accessed model, data, or documentation | 3-7 years per regulation |
| Performance Events | Monitoring alerts, drift detections, degradation | Model lifetime + 5 years |
| Decision Records | Committee decisions, exception approvals | Model lifetime + 7 years |
Audit Trail Architecture
Event Collection
Instrument all model governance systems to emit structured audit events. Use a standard event schema with timestamp, actor, action, target, and context fields.
Centralized Storage
Store audit events in a centralized, immutable log store. Separate audit data from operational data with independent access controls.
Search and Retrieval
Index audit events for efficient querying. Support searches by model, user, time range, event type, and custom metadata.
Compliance Reporting
Generate audit reports on demand for regulatory examinations, internal audits, and incident investigations.
ML-Specific Audit Considerations
- Training Data Lineage: Record exactly which data was used for each training run, including dataset versions, filtering criteria, and sampling methods
- Experiment Lineage: Link production models back to their experiment runs, capturing all hyperparameters, code versions, and evaluation results
- Prediction Logging: For high-risk models, log individual predictions with inputs, outputs, and confidence scores for post-hoc analysis
- Model Card Versioning: Track changes to model documentation over time with diff-style change tracking
Implementation Approaches
Event Sourcing
Store the complete sequence of events rather than just current state, enabling perfect reconstruction of any historical state.
Signed Logs
Cryptographically sign audit entries and chain them together to detect any tampering or insertion of fraudulent records.
Cross-System Correlation
Correlate audit events across model registry, ML platform, deployment systems, and monitoring tools using shared identifiers.
Automated Verification
Periodically verify audit trail completeness and integrity with automated checks that detect gaps or inconsistencies.