Lilly Tech Systems
Responsible AI Practice
Master the practice of responsible AI inside an organisation. 50 deep dives across 300 lessons covering RAI program foundations (program overview, principles, strategy, operating model, charter, business case), translating principles to practice (fairness, accountability, transparency, safety, human oversight as operational disciplines), responsible AI by design (engineering integration, design reviews, bias mitigation in the pipeline, responsible MLOps, CI/CD gates, monitoring, feedback loops, engineering toolkits), stakeholder engagement and impact (affected communities, participatory design, accessibility, algorithmic and societal impact assessment), transparency and documentation practice (model cards, system cards, end-user disclosures and explanations, transparency reports), RAI testing and evaluation (pre-deployment testing, RAI red teaming, adversarial evaluation, continuous evaluation, third-party RAI audit), RAI culture and training (culture building, training programs, AI literacy, ethics champions, internal comms, whistleblower channels), and RAI metrics and continuous improvement (KRIs, maturity model, benchmarking, board reporting).
Responsible AI Practice is the track for the people who actually run responsible AI inside an organisation. The legal, regulatory, and audit tracks tell you what the rules are; this track is about the everyday operating model that turns those rules — and an organisation’s own values — into the way AI actually gets built, shipped, monitored, and improved. The discipline now has a recognised shape: published principles operationalised into engineering controls, design reviews integrated into the product lifecycle, RAI red teams running on a cadence, model and system cards landing alongside every release, stakeholder engagement built into the impact-assessment process, and a measurable maturity curve that boards and customers are starting to expect.
The lessons here are written for the practitioners doing this work day to day: responsible-AI program leads, ML engineers integrating RAI controls into pipelines, product managers running design reviews, RAI red-team operators, transparency and documentation owners, training and comms partners, and the program managers tracking maturity and metrics. Every topic explains the underlying RAI discipline (drawing on NIST AI RMF, ISO/IEC 42001, the Microsoft and Google responsible AI playbooks, Partnership on AI, the OECD AI principles, and the practitioner literature), the practical artefacts and rituals that operationalise it, and the failure modes (where RAI theatre creeps in, where the program loses traction, what stops red-team findings from being acted on). The aim is that a reader of the track can stand up an RAI program, run it for a portfolio of AI systems, and report on its effectiveness to leadership, customers, and regulators with confidence.
All Topics
50 responsible-AI practice topics organized into 8 categories. Each has 6 detailed lessons with frameworks, rituals, controls, and operational templates.
RAI Program Foundations
RAI Program Overview
Master the foundations of a responsible-AI program. Learn the program scope, goals, sponsorship, integration with adjacent programs, and the operating-model archetypes most enterprises adopt.
6 LessonsDefining RAI Principles
Define the principles that anchor your RAI program. Learn how to translate global frameworks (OECD, UNESCO, NIST AI RMF) into your own short, durable principles, and how to test each principle for operational tractability.
6 LessonsRAI Strategy & Roadmap
Build the RAI strategy and a 3-year roadmap. Learn current-state assessment, ambition setting, prioritised initiatives, sequencing, dependencies, and the executive-approved roadmap document.
6 LessonsRAI Operating Model
Design the RAI operating model. Learn the org placement, role definitions (RAI lead, ethics committee, RAI engineers, champions), decision rights, and the federation pattern most large enterprises end up with.
6 LessonsRAI Charter
Draft the RAI charter. Learn the charter sections, authority and budget statements, scope and exclusions, decision authority, and the formal sign-off process.
6 LessonsRAI Business Case
Build the RAI business case. Learn quantified risk reduction, customer/regulator-driven revenue protection, productivity and rework savings, talent and trust benefits, and the model the CFO will actually accept.
6 LessonsPrinciples to Practice
Principles-to-Practice Translation
Translate abstract principles into concrete controls, metrics, and processes. Learn the translation framework, the 'every principle ladders to a control' rule, and how to test the result.
6 LessonsOperationalising Fairness
Operationalise fairness as a discipline. Learn fairness definitions in context, fairness measurement choices, in-pipeline mitigations, governance for unresolvable trade-offs, and external review.
6 LessonsOperationalising Accountability
Operationalise accountability. Learn the named-owner rule, RACI for AI decisions, immutable audit trails, after-action review discipline, and the link to performance management.
6 LessonsOperationalising Transparency
Operationalise transparency. Learn the audience taxonomy (users, deployers, regulators, public), the artefact-per-audience map, what to disclose vs. what to keep proprietary, and the transparency-by-design pattern.
6 LessonsOperationalising Safety
Operationalise AI safety. Learn the safety case discipline, hazard identification, mitigation hierarchy, monitoring for emergent harm, and the safety-bar policy that gates risky launches.
6 LessonsOperationalising Human Oversight
Operationalise human oversight. Learn the oversight modes (in-the-loop, on-the-loop, in-command), capacity planning for human reviewers, anti-automation-bias practices, and the AI Act Article 14 alignment.
6 LessonsRAI by Design (Engineering Integration)
RAI by Design Overview
Master responsible AI by design. Learn the lifecycle integration points, the 'shift-left' principle, the engineer's view of RAI, and how to avoid bolting RAI on after launch.
6 LessonsRAI Design Reviews
Run RAI design reviews. Learn the review trigger (what AI work needs a review), the review template, the reviewer panel, decision outcomes, and the review SLA.
6 LessonsBias Mitigation in Pipeline
Mitigate bias inside the pipeline. Learn pre-processing, in-processing, and post-processing techniques, when each is appropriate, and how to verify the mitigation does not introduce other harms.
6 LessonsResponsible MLOps
Practice responsible MLOps. Learn data versioning + dataset cards, experiment tracking + model cards, model registry + RAI metadata, deployment with RAI gates, and the run-time RAI observability stack.
6 LessonsRAI CI/CD Gates
Add RAI gates to CI/CD. Learn fairness gates, robustness gates, prompt-injection gates, model-card freshness gates, and the gate-bypass governance.
6 LessonsResponsible AI Monitoring
Monitor AI for responsible operation. Learn the metric set (drift, fairness over time, harmful-output rate, override rate, satisfaction), instrumentation, alerting thresholds, and the runbook on alert.
6 LessonsUser Feedback Loops
Build user feedback loops for RAI. Learn the in-product feedback widget, complaint channels, escalation routing, root-cause analysis on feedback themes, and the closed-loop disclosure to users.
6 LessonsRAI Engineering Toolkits
Pick the right RAI engineering toolkits. Learn fairness toolkits (Fairlearn, AIF360), explainability (SHAP, LIME, Captum), robustness (ART), red-team (Garak, PyRIT), and the integration patterns.
6 LessonsStakeholder Engagement & Impact
Stakeholder Engagement
Engage stakeholders in RAI. Learn stakeholder mapping, engagement formats by stakeholder type, the engagement plan, and the documentation that survives team turnover.
6 LessonsAffected-Community Engagement
Engage affected communities. Learn the difference between stakeholders and affected communities, recruitment without exploitation, compensation, the listening protocol, and the publish-back commitment.
6 LessonsParticipatory AI Design
Practice participatory AI design. Learn co-design methods, prototyping with users, value-sensitive design, the participation gradient (Arnstein), and how to integrate participatory output into engineering decisions.
6 LessonsAccessibility & Inclusion
Design AI for accessibility and inclusion. Learn WCAG-equivalent guidance for AI, AT compatibility, multilingual and cultural accessibility, accessibility testing, and the link to the EU Accessibility Act.
6 LessonsAlgorithmic Impact Assessment
Run an Algorithmic Impact Assessment (AIA). Learn the Canadian government AIA model, the Ada Lovelace Institute guidance, the EU FRIA tie-in, the AIA template, and the publication discipline.
6 LessonsSocietal Impact Analysis
Analyse societal-level impact of AI. Learn second-order effects, distributional impacts, externalities, the precautionary principle versus the proactionary principle, and how to feed analysis into product decisions.
6 LessonsTransparency & Documentation Practices
Transparency Practices Overview
Master the transparency practices stack. Learn the artefact catalogue (data, model, system, deployment, transparency reports), per-artefact ownership, freshness SLAs, and the publication policy.
6 LessonsModel Cards in Practice
Author model cards in practice. Learn the Mitchell et al. core fields, AI Act / ISO 42001 extensions, automated section generation, internal review, and the publishing pipeline.
6 LessonsSystem Cards
Author system cards. Learn the difference between model cards and system cards, the OpenAI / Meta system-card patterns, content sections (purpose, scope, limitations, evaluation, deployment, monitoring), and the customer-facing version.
6 LessonsAI Disclosure to Users
Disclose AI to users. Learn the AI Act Article 50 obligations, deepfake labelling, AI-content marking, chatbot identification, the design patterns that work, and the failure modes (notice fatigue).
6 LessonsEnd-User Decision Explanations
Explain individual AI decisions to end users. Learn what an explanation is for (recourse, contest, trust), the layered-explanation pattern, contesting and human review, and AI Act Article 86.
6 LessonsAI Transparency Reports
Publish AI transparency reports. Learn the report structure, the metrics to publish, comparability across reporting periods, the executive sign-off, and the relationship to investor and customer communications.
6 LessonsRAI Testing & Evaluation
RAI Evaluation Overview
Master RAI evaluation. Learn the evaluation taxonomy (capability, fairness, robustness, safety, privacy, transparency), benchmark choice, custom-eval design, eval reproducibility, and reporting.
6 LessonsPre-Deployment RAI Testing
Run pre-deployment RAI testing. Learn the test plan template, slice testing, perturbation testing, jailbreak testing, the launch-bar mapping, and the go/no-go decision.
6 LessonsResponsible-AI Red Teaming
Run RAI red teaming. Learn the program scope, recruiting red-teamers (internal and external), structured threat modelling for AI, the campaign cadence, and the link to bug bounty.
6 LessonsAdversarial Evaluation
Run adversarial evaluation. Learn the threat model, attack libraries (TextAttack, ART, Garak, PyRIT), defence verification, and the relationship between adversarial robustness and operational reliability.
6 LessonsContinuous RAI Evaluation
Run RAI evaluations continuously. Learn the production-shadow eval pipeline, drift-triggered re-evaluation, golden-eval set maintenance, scoreboard publication, and degradation alerts.
6 LessonsThird-Party RAI Audit
Engage third-party RAI auditors. Learn the auditor landscape, scope-of-work definition, evidence sharing, finding management, the public-summary decision, and the link to ISO 42001 / SOC 2 attestation.
6 LessonsRAI Culture, Training & Comms
Building RAI Culture
Build an RAI culture that holds. Learn the leadership-tone signals, the everyday rituals, recognition systems, learning-from-incidents discipline, and the cultural anti-patterns to watch for.
6 LessonsRAI Training Program
Design the RAI training program. Learn the audience matrix, role-based curricula, delivery formats, completion tracking, effectiveness measurement, and the EU AI Act Article 4 alignment.
6 LessonsAI Literacy Across the Org
Build AI literacy across the org. Learn the literacy framework, the baseline-everyone-needs curriculum, the role-specific overlays, the assessment tooling, and the ongoing literacy maintenance.
6 LessonsEthics Champions Network
Build the ethics-champions network. Learn the champion role, recruitment, training, the operating cadence, escalation paths, and the relationship to the central RAI team.
6 LessonsRAI Internal Comms
Run RAI internal comms. Learn the comms calendar, the channel mix, the storytelling discipline (real cases, named heroes), the contested-topic handling, and the comms-effectiveness measurement.
6 LessonsWhistleblower & Concerns Channels
Establish whistleblower channels for AI concerns. Learn the channel design, anonymity options, anti-retaliation protections, intake triage, investigation discipline, and the regulator-disclosure obligations.
6 LessonsRAI Metrics, Maturity & Improvement
RAI Metrics Overview
Master RAI metrics. Learn the leading vs lagging distinction, the metric categories (program, control, outcome, perception), measurement integrity, and the publish-vs-internal split.
6 LessonsRAI Key Risk Indicators
Define and track RAI KRIs. Learn KRI selection criteria, the threshold-and-action model, KRI ownership, monthly reporting cadence, and how KRIs trigger committee escalation.
6 LessonsRAI Maturity Model
Use an RAI maturity model. Learn the dimensions (governance, principles, lifecycle, eval, transparency, culture, incident response), the five-level scale, the assessment, and the roadmap to next level.
6 LessonsRAI Benchmarking
Benchmark your RAI program. Learn the benchmark sources (industry surveys, peer exchanges, public RAI reports, regulator reports), the comparability problem, the analysis pattern, and the action set.
6 LessonsContinuous Improvement
Run RAI continuous improvement. Learn the improvement backlog, the experimentation discipline, the lessons-learned loop from incidents, the annual program retrospective, and the link to RAI strategy refresh.
6 LessonsRAI Board Reporting
Report RAI to the board. Learn the board-pack structure, the right level of detail, the bad-news discipline (escalating early), the board-question playbook, and the link to ESG and risk reports.
6 Lessons