Detecting checkable political claims — annotation schema & benchmark

Executive Summary

Newsrooms, broadcasters and media‑monitoring organizations struggle to reliably identify which political statements are checkable at scale, often wasting analyst time on non‑falsifiable rhetoric and producing noisy candidate lists from ASR transcripts and social feeds. The pain is measurable: the estimated 200,000 global newsrooms and monitoring orgs in the $8.0B market (roughly $40K ACV) need higher‑precision tools to reduce human review costs and to meet growing compliance and audit demands. We could build a practical annotation schema and public benchmark for "checkable political claim" detection: a clear taxonomy, detailed guidelines, and a multi‑modal labeled corpus (broadcast transcripts, social posts, press releases) of at least 100,000 annotated segments, plus baseline transformer models and an open leaderboard. Deliverables would include inter‑annotator agreement targets (e.g., Cohen’s kappa ≥0.7), pretrained checkpoints, and an API/SDK for vendors and in‑house teams to integrate the benchmark into editorial and compliance workflows. The market is attractive now because transformer NLP advances materially increase contextual precision and reduce false positives, regulatory scrutiny (e.g., DSA and election oversight) is creating demand for verifiable audit trails, and a growing ecosystem of fact‑checking and publisher partners can supply validation channels and early adopters. This can stand out by prioritizing annotation quality and operational utility—high‑agreement, multilingual labels that include noisy ASR sources, pragmatic licensing and turnkey integration guides—while being candid about challenges: political subjectivity in labels, the cost of high‑quality annotation, and the need for continual model retraining as political discourse evolves.