Case Study · Trust & Safety Analytics

AI Ad Review
Quality
Framework

Monitoring model quality, policy risk, and reviewer calibration in an AI-assisted ad review workflow.

Focus

AI quality · Policy risk · Reviewer calibration · Feedback loops

Stack

Python · SQL · DuckDB · Plotly · Streamlit

Type

Portfolio case study

AI Review · Live feed

0 processed

Approved

Rejected

Escalated

5,000

Ad review cases

71.1%

AI accuracy

15.7%

Risk miss rate

15.8%

Appeal reversal rate

Overview

AI pre-screening introduces risk.

This project simulates how a digital ad platform monitors the quality of an AI-assisted ad review workflow.

A model can be too strict and block valid advertisers, or too lenient and approve ads that should have been escalated. The challenge is to separate advertiser friction from policy exposure, then turn those signals into actions.

Three questions this answers

Policy riskWhere does the AI create policy exposure?

Reviewer gapsWhere do human reviewers need calibration?

Feedback loopsWhich errors should feed back into model improvement?

What I built

An end-to-end quality framework.

Synthetic dataset

5,000 ad review cases across 8 markets and 10 policy categories, designed to reflect realistic distribution and edge cases.

PythonData gen↓

Python pipeline

Data generation, preparation, and quality evaluation — fully reproducible from raw inputs to dashboard-ready outputs.

pandasDuckDB↓

SQL analysis

Queries covering model accuracy, policy risk misses, appeal reversals, market patterns, and BPO calibration gaps.

SQLDuckDB↓

Streamlit dashboard

Executive overview and operational monitoring views, built for senior review and day-to-day quality tracking.

StreamlitPlotly↓

Documentation

Methodology, error taxonomy, policy gap analysis, and recommendations for Policy, Ad Ops, and Algorithm teams.

MarkdownGitHub↓

Click any row to expand

Framework

Four pillars, one operating model.

Each pillar answers a different question for a different team.

Model quality

Measures AI accuracy, confidence distribution, disagreement rate, and high-confidence wrong decisions that carry the most operational risk.

Policy risk

Separates over-rejection (advertiser friction) from risk misses (policy exposure). Identifies approval misses in high-risk categories.

III

Reviewer calibration

Compares human accuracy by BPO team, reviewer tenure, market, and policy category to surface training and calibration gaps.

Feedback loops

Uses appeal reversals, ambiguous cases, and high-confidence AI errors to identify retraining opportunities and policy clarification needs.

Key metrics

What the data showed.

71.1%

AI accuracy

Below human benchmark

83.5%

Human accuracy

Baseline for calibration

60.8%

Human-AI agreement

Significant gap

15.7%

Risk miss rate

Policy exposure

13.2%

Over-rejection rate

Advertiser friction

154

High-risk approval misses

Priority for review

15.8%

Appeal reversal rate

Over-rejection signal

Dashboard preview

Built for operational review.

Executive Overview

Top-line accuracy, risk miss rate, appeal reversals, and reviewer calibration — one view for senior stakeholders.

Policy Category Analysis

Disaggregated model accuracy and risk miss rate by policy category, surfacing where under-rejection concentrates.

Insights

What the monitoring layer revealed.

Strong on clear-cut cases, weak on regulated ones↓

The AI performs well on standard policy categories but struggles with Financial Product Claim, Political Content, and Misleading Claim — where context determines risk.

Under-rejection is the primary risk↓

The main model failure mode is approving ads that should be escalated or rejected. This is a policy exposure problem, not just an accuracy problem.

Low-confidence decisions need mandatory routing↓

AI decisions below 0.60 confidence should go to human review. Letting the model decide on uncertain cases is where most errors concentrate.

Reviewer experience has a measurable impact↓

New reviewers show a 24.1% error rate versus 10.5% for experienced reviewers — a 2.3x gap that points to structured onboarding and calibration needs.

BPO team variation reflects calibration gaps↓

Performance varies significantly across BPO teams. This is not individual reviewer performance — it is a training and calibration issue at the team level.

Non-English markets show higher disagreement↓

BR, IT, DE, and ES have higher human-AI disagreement rates, suggesting market-specific policy examples and local reviewer guidance are needed.

Click any finding to expand

Recommendations

Route, review, retrain, clarify.

Route low-confidence decisions to human review

AI decisions below 0.60 should not be auto-approved. Routing them to reviewers reduces the highest-concentration error zone.

Review high-risk approval misses weekly

154 high-risk approvals need weekly review. This is a standing operational task, not a one-off audit.

Calibrate reviewers on regulated categories

Financial Product Claim and Health Claim show the widest calibration gaps. These categories need dedicated reviewer training.

Send high-confidence AI errors to the model team

Cases where the AI was wrong with high confidence are the most valuable retraining signals. They should be flagged and routed automatically.

Add market-specific policy guidance

BR, IT, DE, and ES need local policy examples and reviewer guidance to reduce the human-AI disagreement rate in non-English markets.

Use appeal reversals to reduce advertiser friction

A 15.8% reversal rate signals systematic over-rejection in specific categories. Reversals are a direct feedback signal for policy adjustment.

What this demonstrates

The value is the operating model, not the dashboard.

This project connects model performance, human review quality, advertiser friction, and policy exposure into one decision framework — what to route, what to review, what to retrain, and what policy teams need to clarify.

Generic accuracy reporting is not enough. Quality monitoring requires operational context.

Skills demonstrated

AI quality monitoring — beyond generic accuracy reporting.

Trust & Safety analytics — connecting model performance to policy exposure.

Python data pipelines — synthetic data, preparation, and evaluation.

SQL-based investigation — market patterns, reversal analysis, calibration gaps.

Executive dashboarding — operational views built for decision-making.

Model feedback loop design — identifying retraining and clarification opportunities.

Business-oriented data storytelling — connecting data to actions for multiple teams.

Tech stack

PythonpandasDuckDBSQLPlotlyStreamlitGitHub

Next.

Explore the rest of the work.

Back to Work

The repository includes the synthetic dataset, Python pipeline, SQL analysis, Streamlit dashboard, and full documentation.

AI Ad ReviewQualityFramework