Lilly Tech Systems
AI Algorithmic Fairness
Master AI algorithmic fairness as a first-class discipline. 50 deep dives across 300 lessons covering fairness foundations (harms, protected attributes, sources of bias, fairness-accuracy trade-offs, landmark cases), fairness metrics (group, individual, counterfactual, calibration, impossibility results, metric selection), bias detection (auditing, disaggregated eval, intersectionality, proxy detection, statistical tests, bias bounty), pre-processing (representative data, reweighting, augmentation, fair collection, label bias, synthetic), in-processing (adversarial debiasing, fairness constraints, fair representations, multi-task, transfer, fair ensembling), post-processing (threshold optimisation, recalibration, reject-option, equalised-odds, abstain & defer, fairness monitoring), system-specific fairness (LLMs, ranking, recommenders, vision, NLP, generative AI), and operations & governance (program, toolkits, regulation, anti-discrimination law, incident response, stakeholder engagement, fairness reporting).
Algorithmic fairness is the discipline of making AI systems treat people equitably across the dimensions society and the law cares about — and proving it. It sits at the intersection of civil-rights and anti-discrimination law (Title VII, ECOA, FHA, the EU AI Act, NYC Local Law 144), statistics and measurement (group, individual, and causal fairness, calibration, impossibility results), ML engineering (pre-, in-, and post-processing mitigations), and the operational machinery that runs in production (audits, monitoring, incident response, redress, board reporting). Over the last five years it has stopped being an academic side topic and has become an operating commitment for any organisation deploying AI at scale: fairness audits gate launches, AEDT-style bias reports are published publicly, regulators issue guidance, plaintiffs file complaints, and customers ask for evidence.
This track is written for the practitioners doing this work day to day: fairness engineers, ML engineers integrating fairness controls into pipelines, RAI leads, data scientists running audits, product managers shipping AI features that touch decisions about people, governance leads writing fairness policies, lawyers collaborating with engineering, incident-response commanders running fairness playbooks, and program leads stitching the program together. Every topic explains the underlying fairness discipline (drawing on the canonical literature — Friedler, Barocas-Hardt-Narayanan, Mitchell et al., Chouldechova, Kleinberg-Mullainathan-Raghavan, Buolamwini-Gebru, Raji et al. — and from regulator guidance, NIST AI RMF, ISO/IEC 24027, IEEE 7003), the practical artefacts and rituals that operationalise it (audits, model cards, monitoring, runbooks, redress mechanisms), and the failure modes where fairness work quietly breaks down in practice. The aim is that a reader can stand up a credible algorithmic-fairness function, integrate it with engineering and governance, and defend it to boards, regulators, customers, plaintiffs, and the people the system actually affects.
All Topics
50 AI algorithmic fairness topics organized into 8 categories. Each has 6 detailed lessons with frameworks, templates, and operational patterns.
Fairness Foundations
Algorithmic Fairness Overview
Master what algorithmic fairness actually is. Learn the scope, the lineage from civil rights law and statistics, the deliverables, and the operating model used by mature fairness teams.
6 LessonsAllocative, Representational & QoS Harms
Build a working taxonomy of algorithmic harms. Learn allocative harms, representational harms, quality-of-service harms, and the link from each harm type to a measurable signal.
6 LessonsProtected Attributes & Sensitive Categories
Engineer with protected attributes the right way. Learn US protected classes, GDPR special categories, intersectionality, the disparate-impact / disparate-treatment split, and proxy attributes.
6 LessonsSources of Bias in ML Pipelines
Map bias to its sources. Learn historical, representation, measurement, aggregation, evaluation, and deployment bias, and the diagnostic that tells you which one is driving your problem.
6 LessonsFairness-Accuracy Trade-offs
Reason about fairness-accuracy trade-offs honestly. Learn the Pareto frontier, when the trade-off is real vs apparent, the role of better data, and how to communicate the choice to leadership.
6 LessonsFairness History & Landmark Cases
Stand on the shoulders of the people who built this field. Learn the landmark cases (COMPAS, Apple Card, Amazon hiring, healthcare risk scores) and the lessons each one cemented.
6 LessonsFairness Metrics
Fairness Metrics Overview
Map the fairness-metrics zoo. Learn group vs individual vs causal metrics, the categories regulators reference, and a practical decision tree for picking the right metric per problem.
6 LessonsGroup Fairness Metrics
Compute group-fairness metrics correctly. Learn demographic parity, equal opportunity, equalised odds, predictive parity, and the assumptions each one relies on.
6 LessonsIndividual Fairness
Apply individual-fairness criteria. Learn the Lipschitz formulation, similarity metrics, the link to consistency, and the practical limits of individual fairness in production systems.
6 LessonsCounterfactual & Causal Fairness
Use causal reasoning for fairness. Learn counterfactual fairness, path-specific effects, mediation analysis, and the data and assumptions a credible causal-fairness claim requires.
6 LessonsCalibration & Predictive Parity
Use calibration as a fairness criterion. Learn group-conditional calibration, reliability diagrams, ECE per group, the COMPAS calibration debate, and recalibration patterns.
6 LessonsFairness Impossibility Results
Understand why some fairness criteria conflict. Learn the Chouldechova / Kleinberg-Mullainathan-Raghavan results, when they bite, and how to navigate them in practice.
6 LessonsChoosing the Right Fairness Metric
Pick fairness metrics defensibly. Learn the harm-to-metric mapping, regulatory anchors, stakeholder input, the hold-this-up-to-board test, and the metric-decision document.
6 LessonsBias Detection & Measurement
Bias Auditing Methodology
Audit a model for bias rigorously. Learn the audit lifecycle, scoping, data acquisition, metric computation, statistical testing, root-cause analysis, and the auditor-facing report.
6 LessonsDisaggregated Evaluation
Evaluate models per subgroup, not just on average. Learn slicing strategies, sample-size requirements, multiple-comparison correction, and the disaggregated-eval dashboard pattern.
6 LessonsIntersectional Fairness Analysis
Analyse fairness intersectionally. Learn why marginal slicing hides harm, the gender-shades methodology, sample-size strategies for thin intersections, and reporting standards.
6 LessonsProxy Variable Detection
Find proxies for protected attributes. Learn correlation tests, mutual information, redundancy detection, and the workflow that decides whether to remove, transform, or keep a proxy.
6 LessonsStatistical Tests for Bias
Apply statistical tests to fairness claims. Learn proportion tests, permutation tests, equivalence vs significance, confidence intervals, and the misuse patterns that mislead reviewers.
6 LessonsBias Bounty & Red-Team Programs
Run a bias-bounty program. Learn the model (Twitter, OpenAI, US AISI), scoping, recruitment, scoring rubrics, payout structure, finding-to-fix workflow, and program ROI.
6 LessonsPre-processing & Data Fairness
Representative Training Data
Build training data that reflects who you serve. Learn target-population mapping, coverage gaps, oversampling strategies, dataset cards, and the link to disaggregated eval.
6 LessonsReweighting & Resampling for Fairness
Use sample weights and resampling to nudge fairness. Learn IPW, Kamiran-Calders reweighting, SMOTE-for-fairness, the limits of resampling, and the eval to verify the fix worked.
6 LessonsData Augmentation for Fairness
Augment data to reduce bias. Learn counterfactual augmentation, demographic name swapping, image attribute editing, eval guardrails, and the failure modes that mask but don't fix bias.
6 LessonsFair Data Collection Practices
Collect data fairly upstream. Learn participatory design, consent for sensitive attributes, compensation, source-of-truth diversity, and the data-collection ethics review.
6 LessonsLabel Bias & Annotator Quality
Manage label bias. Learn annotator demographic effects, inter-annotator agreement per group, gold-standard construction, label-bias audits, and the label-correction playbook.
6 LessonsSynthetic Data for Fairness
Use synthetic data to address fairness gaps. Learn fairness-aware generators, fidelity vs fairness trade-offs, the Goodhart trap, and the eval discipline for synthetic-augmented training.
6 LessonsIn-processing & Model Fairness
Adversarial Debiasing
Train models that resist predicting the protected attribute. Learn the adversarial-debiasing architecture, training stability, eval after adversarial training, and the limits of the approach.
6 LessonsFairness Constraints in Optimization
Bake fairness into the loss. Learn constrained optimisation (Zafar et al., Agarwal et al.), Lagrangian methods, slack budgets, multiple-metric handling, and convergence diagnostics.
6 LessonsFair Representation Learning
Learn representations that hide protected attributes by construction. Learn LFR, FFVAE, contrastive methods, and the downstream-utility / leakage trade-off.
6 LessonsMulti-Task & Multi-Objective Training
Train one model that serves multiple tasks fairly. Learn shared-vs-task-specific layers, gradient surgery, Pareto multi-task learning, and the per-task fairness audit.
6 LessonsFairness in Transfer Learning
Audit fairness through transfer-learning steps. Learn pretraining-bias inheritance, fine-tuning effects, domain-shift fairness, and the layer-by-layer fairness diagnostic.
6 LessonsFair Ensembling
Use ensembling for fairness. Learn diversity-promoting ensembles, group-specialist mixtures, fairness-aware bagging, ensemble calibration, and the cost-vs-fairness profile.
6 LessonsPost-processing & Deployment Fairness
Threshold Optimization
Set decision thresholds fairly. Learn group-specific thresholds, the Hardt-Price-Srebro post-processing method, the legality question, and operational considerations.
6 LessonsOutput Calibration & Recalibration
Recalibrate outputs per group. Learn Platt scaling, isotonic regression, group-conditional recalibration, the link to threshold optimisation, and the calibration drift dashboard.
6 LessonsReject-Option Classification
Use reject-option classification to reduce harm. Learn the uncertainty-band approach, group-aware reject regions, downstream human-review handoff, and audit requirements.
6 LessonsEqualized Odds Post-processing
Apply the canonical equalised-odds post-processing fix. Learn the convex combination construction, the closed-form solution, evaluation requirements, and the legality fine print.
6 LessonsAbstain & Defer Patterns
Design models that abstain when fairness is uncertain. Learn abstain heads, defer-to-human routing, safe-abstain costs, and the fairness audit on the abstain population.
6 LessonsFairness Monitoring in Production
Monitor fairness in production. Learn the fairness KPI set, drift detection per slice, automated alerts, on-call response, and the fairness-incident-to-PIR loop.
6 LessonsFairness in Specific AI Systems
LLM Fairness & Stereotypes
Audit and mitigate fairness issues in LLMs. Learn the evaluation suites (BBQ, BOLD, StereoSet, HolisticBias), the limits of those benchmarks, RLHF effects, and red-teaming patterns.
6 LessonsRanking & Search Fairness
Make ranking and search fair. Learn exposure-based metrics, fair top-k, position bias, the producer-vs-consumer fairness split, and constrained-ranking algorithms.
6 LessonsRecommender System Fairness
Engineer fair recommenders. Learn user-side, item-side, and platform fairness, popularity bias, filter bubbles, and the fairness-aware re-ranking layer pattern.
6 LessonsComputer Vision Fairness
Audit and mitigate fairness in CV systems. Learn skin-tone evaluation (Fitzpatrick, MST), face-recognition disparities, dataset audits, generation-model fairness, and deployment guard rails.
6 LessonsNLP Fairness Beyond LLMs
Audit and mitigate fairness in classic NLP. Learn embedding-bias diagnostics (WEAT, SEAT), translation gender bias, ASR disparities, sentiment bias, and the NLP fairness eval kit.
6 LessonsGenerative AI Fairness
Audit and mitigate fairness in generative AI. Learn text-to-image stereotypes, prompt-occupation tests, generation-time guardrails, attribution and dignity harms, and the GenAI fairness disclosure.
6 LessonsFairness Operations & Governance
Algorithmic Fairness Program
Stand up a fairness program. Learn the program charter, the operating model, RACI across product / engineering / legal / RAI, KPIs, and the board-level fairness report.
6 LessonsFairness Toolkits & Libraries
Pick and operate fairness toolkits. Learn AIF360, Fairlearn, What-If Tool, Aequitas, FairML, and the boundary between toolkit-supplied measurement and home-built audit.
6 LessonsFairness Regulation Landscape
Navigate the fairness regulation landscape. Learn the EU AI Act fairness provisions, NYC AEDT Local Law 144, EEOC AI guidance, CFPB / fair-lending, and state-level patchwork.
6 LessonsAnti-Discrimination Law for Engineers
Translate anti-discrimination law into engineering controls. Learn disparate treatment vs disparate impact, the 4/5ths rule, business necessity, less-discriminatory alternative, and pretext.
6 LessonsFairness Incident Response & Redress
Run fairness incident response. Learn incident definitions, severity ladders, intake (user complaint, internal alert, regulator), redress mechanisms, and the post-incident review.
6 LessonsStakeholder Engagement for Fairness
Engage stakeholders meaningfully. Learn participatory design, affected-community panels, civil-society engagement, compensated co-design, and the boundary with marketing/PR.
6 LessonsFairness Reporting & Disclosure
Report fairness honestly. Learn fairness-section model cards, AEDT bias audit publication, transparency reports, system cards, regulator-facing audit packets, and customer-facing summaries.
6 Lessons