Don’t Label Twice: Quantity Beats Quality when Comparing Binary Classifiers on a Budget
When building a test set for binary classification from noisy label, how many labels to collect per data point? Surprisingly under a simple budget constraint, the answer is a single label.
Florian E. Dorner, Moritz Hardt
Workshop on Navigating and Addressing Data Problems for Foundation Models (at ICLR 2024)