Blog

Inter-annotator agreement is a core measure of data quality in machine learning. When multiple annotators label the same data, agreement levels reveal how consistently a task can be interpreted and how reliable the resulting labels are. Low agreement often indicates unclear guidelines, task ambiguity, or expertise mismatches rather than annotator error. Measuring and monitoring inter-annotator agreement helps teams detect label noise early, improve annotation design, and produce datasets that lead to more stable and generalizable models.

A recent paper from MIT CSAIL researchers Alex L. Zhang, Tim Kraska, and Omar Khattab introduces Recursive Language Models (RLMs) as a solution. Published as a preprint in late 2025 and updated in January 2026, RLMs enable models to handle inputs far beyond standard context limits, often with better accuracy and comparable or lower cost.