AI Trust & Safety Operations
Master AI Trust & Safety Operations as a first-class operational discipline. 50 deep dives across 300 lessons covering operations foundations (T&S as a discipline, function disentanglement, history, career paths, operating models, strategy), threat & risk frameworks (harm taxonomy, threat modeling, abuse vectors, risk prioritisation, adversary modeling, emerging threats), detection engineering (signal engineering, behavioural detection, graph-based detection, ML tradecraft, scale, evaluation), investigations & threat actors (workflow, evidence handling, threat-actor tracking, influence ops, fraud rings, ATO, attribution), operations & workflow (runbooks, queue engineering, agent tooling, workforce planning, follow-the-sun, BCP/DR), metrics & programs (KPI suite, prevalence measurement, SLAs / SLOs / error budgets, OKRs, budgeting, program reviews), crisis & escalation (crisis doctrine, escalation protocols, war rooms, crisis comms, regulator emergency response, post-crisis learning), and industry, standards & collaboration (TSPA / GIFCT / Tech Coalition, cross-platform collaboration, vendor landscape, research & policy engagement, NIST / ISO / OECD / AISI / Santa Clara standards, the future of T&S).
AI Trust & Safety Operations is the operational discipline of running a T&S function the way modern engineering organisations run SRE or security: as a profession with detection-engineering tradecraft, investigation rigour, runbooks, SLAs, error budgets, on-call rotations, post-incident reviews, professional bodies, and standards. It complements content-moderation policy work but is distinct from it — policy decides what stays up; operations decides whether the system that enforces policy is fast, accurate, humane, durable, and defensible. Over the last five years T&S has stopped being a back-office activity and has become an engineering-grade discipline subject to regulator inspection, public reporting, and material business risk. The DSA, the UK Online Safety Act, the AI Act, NetzDG, India IT Rules, and a growing list of sectoral regulators all assume there is a running operational function with the rigour to defend its work.
This track is written for the practitioners doing this work day to day: T&S leaders, T&S operations managers, detection engineers, T&S software engineers, investigators, integrity / civic teams, threat-intel and CIB analysts, program managers, on-call commanders, and the cross-functional partners (security, RAI, legal, comms, product) who interlock with T&S. Every topic explains the underlying operational discipline (drawing on the T&S literature, TSPA professional materials, GIFCT and Tech Coalition operational guides, the SRE / IR playbooks adapted for T&S, regulator expectations, and hard-won production experience), the practical artefacts and rituals that operationalise it (runbooks, dashboards, OKRs, threat models, evidence packets, after-action reports), and the failure modes where T&S operations quietly break down in practice. The aim is that a reader can stand up a credible T&S operations function, integrate it with engineering and governance, and defend it to boards, regulators, customers, and the people the platform actually affects.
All Topics
50 AI Trust & Safety Operations topics organized into 8 categories. Each has 6 detailed lessons with frameworks, templates, and operational patterns.
T&S Operations Foundations
Trust & Safety as a Discipline
Master what Trust & Safety actually is as a professional discipline. Learn the scope, the difference from related functions, the deliverables, and the operating model used by mature teams.
6 LessonsT&S vs Content Moderation vs Security vs RAI
Disentangle T&S from related functions. Learn the boundary with content moderation, security, RAI, and CX, the overlap zones, the failure modes when boundaries blur, and the integration patterns.
6 LessonsHistory & Evolution of T&S
Trace the evolution of T&S from spam fighting to a regulated discipline. Learn the eras (early web, social platforms, ad-tech, generative AI) and the lessons each cemented for practitioners.
6 LessonsT&S Career Paths & Roles
Map the T&S career landscape. Learn the role taxonomy (analyst, investigator, policy, ops manager, detection engineer, T&S engineering, leadership), the IC-vs-management split, and the credentialing question.
6 LessonsT&S Operating Models
Pick the right T&S operating model. Learn centralised, federated, hub-and-spoke, and product-embedded patterns, regional structure, RACI across teams, and the model-evolution decision.
6 LessonsT&S Strategy & Vision
Write a T&S strategy that survives two budget cycles. Learn the strategy template, prioritisation against business risk, OKR setting, the 18-month roadmap, and the board-level pitch.
6 LessonsThreat & Risk Frameworks
T&S Harm Taxonomy
Build a working T&S harm taxonomy. Learn the canonical categories, severity ladders, victim-centric vs platform-centric framings, taxonomy versioning, and the link to detection labels.
6 LessonsT&S Threat Modeling
Run T&S threat modeling on a product surface. Learn the methodology (LINDDUN-T, abuse cases, misuse cases), the workshop format, the output artefact, and the link to detection.
6 LessonsAbuse Vector Mapping
Map abuse vectors per product surface. Learn the surface inventory, vector taxonomy (account, content, payments, agents, API, ads), coverage gaps, and the prioritisation matrix.
6 LessonsT&S Risk Prioritization
Prioritise T&S risk credibly. Learn risk scoring, severity * likelihood * exposure, regulatory weight, public-attention weight, and the quarterly risk-prioritisation ritual.
6 LessonsAdversary Modeling
Model adversaries the way T&S work needs. Learn the adversary taxonomy, capability profiles, motivation profiles, the kill-chain mapping, and the link to detection investments.
6 LessonsEmerging Threat Identification
Spot emerging threats early. Learn signal sources (researchers, journalists, OSS communities, regulators), triage cadence, false-positive discipline, and the emerging-threat watchlist.
6 LessonsDetection Engineering
T&S Detection Engineering Overview
Build a detection-engineering practice for T&S. Learn the detection lifecycle, the detection backlog, runbook attachment, eval discipline, and the relationship to the analyst pipeline.
6 LessonsSignal Engineering
Engineer high-quality signals upstream of detection. Learn signal sources, instrumentation, signal hygiene, deduplication, and the signal-quality dashboard.
6 LessonsBehavioral Detection
Detect bad behaviour, not just bad content. Learn account-velocity rules, session-pattern detection, behavioural fingerprints, anomaly detection, and the false-positive trade-off.
6 LessonsNetwork & Graph-Based Detection
Use graphs to detect coordinated abuse. Learn account / device / IP / payment graphs, community detection, propagation analysis, graph features for ML, and the takedown-cluster pattern.
6 LessonsML Tradecraft for T&S
Build T&S ML the way T&S needs. Learn label sourcing under adversarial drift, calibration, slice eval, robustness to evasion, and the model-versioning discipline.
6 LessonsDetection at Petabyte Scale
Run detection at platform scale. Learn streaming vs batch, sampling, fan-out engineering, cost management, hot/cold storage, and the latency-vs-cost trade-off chart.
6 LessonsDetection Evaluation & Tuning
Evaluate and tune detections in production. Learn precision / recall / hit-rate, golden-set design, drift monitoring, threshold ops, the false-discovery / false-omission rate split, and review.
6 LessonsInvestigations & Threat Actors
Investigation Workflow
Run investigations like a professional T&S team. Learn the investigation lifecycle, lead intake, scoping, hypothesis-driven analysis, peer review, and the investigation-to-action handoff.
6 LessonsEvidence Handling & Chain-of-Custody
Handle evidence to a standard that holds up downstream. Learn chain-of-custody, hashing for integrity, retention policy, redaction, and the legal-hold / law-enforcement pattern.
6 LessonsThreat Actor Tracking & Profiling
Track threat actors over time. Learn actor profiles, naming conventions, TTP cataloguing (MITRE-style), continuous tracking, attribution confidence, and the cross-team handoff.
6 LessonsInfluence Operations & Information Ops
Investigate influence operations. Learn the IO taxonomy, the Stanford Internet Observatory / Atlantic Council DFRLab method, attribution discipline, AI-generated content, and disclosure.
6 LessonsSpam, Scam & Fraud Ring Investigation
Investigate spam / scam / fraud rings. Learn signal triangulation, money-flow analysis, infrastructure attribution, ring takedown patterns, and the ROI / recidivism trade-off.
6 LessonsAccount Takeover Investigation
Investigate account takeover at scale. Learn the ATO indicator set, credential-stuffing patterns, post-compromise behaviour, the recovery flow, and the link to security IR.
6 LessonsAttribution & Confidence
Attribute responsibly. Learn attribution-confidence levels (low / medium / high), the diamond model, alternative-hypothesis discipline, and the public-vs-internal attribution split.
6 LessonsOperations & Workflow
T&S Runbook Discipline
Write T&S runbooks engineers and analysts can actually run. Learn the runbook standard, decision trees, escalation triggers, after-action review, and the runbook-coverage metric.
6 LessonsQueue Engineering at Scale
Engineer review queues that survive surges. Learn queue topology, severity-aware routing, SLA partitioning, starvation prevention, and the queue-engineering reliability pattern.
6 LessonsAgent / Reviewer Tooling
Build T&S analyst tooling that respects the work. Learn tool-build vs buy, policy-aware UI, evidence presentation, decision-rationale capture, and the wellness-aware UX standard.
6 LessonsT&S Workforce Planning
Plan T&S workforce honestly. Learn capacity models, surge buffers, language coverage, vendor-mix decisions, attrition modelling, and the link to detection automation.
6 LessonsShift Coverage & Follow-the-Sun
Cover the world without burning the team. Learn follow-the-sun design, handoff rituals, time-zone fairness, holiday coverage, and the on-call rotation specific to T&S.
6 LessonsBCP / DR for T&S
Plan business-continuity and disaster-recovery for T&S. Learn the BCP/DR scope specific to T&S, region-loss scenarios, dependency mapping, and the annual DR drill.
6 LessonsMetrics, SLAs & Programs
T&S Metrics Suite
Run a comprehensive T&S metrics suite. Learn the canonical KPIs (prevalence, time-to-action, proactive ratio, appeal-reversal, IRA), reviewer wellness, drift, and the audience-specific dashboards.
6 LessonsPrevalence Measurement
Measure prevalence credibly. Learn the prevalence definition (Meta-style, regulator-aligned), sampling design, statistical confidence, drift, comparability, and the regulator-facing artefact.
6 LessonsSLAs, SLOs & Error Budgets
Run T&S like an SRE-grade discipline. Learn time-to-action SLAs per severity, SLOs and error budgets, alerting on burn rate, and the trade-off between throughput and quality.
6 LessonsT&S OKRs & Goal-Setting
Set T&S OKRs that change behaviour. Learn the OKR template, the vanity-metric trap, harm-reduction OKRs vs activity OKRs, and the cross-team OKR alignment ritual.
6 LessonsT&S Budgeting & Headcount Modeling
Model T&S budget and headcount. Learn the budget structure, capacity-driver mapping, automation-investment ROI, the regulator-driven floor, and the budget-defence narrative.
6 LessonsProgram Reviews & Roadmaps
Run program reviews that drive change. Learn the review cadence (weekly / monthly / quarterly), the roadmap discipline, decision logging, escalation, and the board-level review.
6 LessonsCrisis, Incidents & Escalation
Crisis Operations Doctrine
Operate during T&S crises. Learn the crisis taxonomy, doctrine (declare, command, isolate, communicate, recover, learn), and the link to product, comms, legal, and regulator-facing teams.
6 LessonsEscalation Protocols
Design escalation protocols for hard cases. Learn the L1 / L2 / specialist tier structure, criteria, SLAs, the public-figure / executive-decision pathway, and the escalation audit.
6 LessonsWar Room Operations
Run a unified war-room without burning the team. Learn the war-room charter, command structure, log-keeping, shift rotation, comms cadence, and the post-war-room handoff.
6 LessonsCommunications During Crisis
Communicate during T&S crisis honestly. Learn the comms tracks (internal, user, public, regulator, law-enforcement), holding statements, update cadence, and the legal review.
6 LessonsRegulator-Facing Emergency Response
Respond to regulator emergency requests. Learn DSA crisis mechanism (Article 36), Ofcom / BNetzA / eSafety urgent procedures, evidence packaging, legal-hold, and the comms posture.
6 LessonsPost-Crisis Learning & Hardening
Learn from crises so the next one is smaller. Learn the blameless PIR template, action-item discipline, hardening cycles, runbook updates, and the public after-action report.
6 LessonsIndustry, Standards & Collaboration
T&S Professional Bodies
Engage with T&S professional bodies. Learn TSPA, GIFCT, Tech Coalition, NCMEC, IWF, the Christchurch Call, and the cost / value calculus of each membership.
6 LessonsCross-Platform Collaboration
Collaborate across platforms responsibly. Learn hash-sharing programs, signal-sharing protocols, due-process safeguards, antitrust limits, and the public-trust calculus.
6 LessonsT&S Tooling & Vendor Landscape
Navigate the T&S tooling and vendor landscape. Learn the vendor map (Hive, Spectrum Labs, ActiveFence, Sentropy, Two Hat / Microsoft), classifier APIs, agent-tool platforms.
6 LessonsT&S Research & Policy Engagement
Engage with T&S research and policy communities. Learn the research-partnership pattern, data-access programs, the academic publication question, and the policy comments process.
6 LessonsT&S Standards & Frameworks
Adopt T&S standards and frameworks. Learn NIST AI RMF, ISO/IEC 42001, OECD AI principles, AISI eval patterns, Santa Clara Principles, and the standards-mapping discipline.
6 LessonsFuture of Trust & Safety
Reason about where T&S is heading. Learn the AI-agent threat landscape, the regulatory consolidation thesis, the professionalisation trend, and the open research questions worth tracking.
6 Lessons
Lilly Tech Systems