OpenAI ML Coding: Noisy Human-Labeled Text Classifier

You are given a dataset of text prompts labeled for a binary identity_attack tag, with multiple human annotators per example and pre-computed embeddings provided. Your goal is to quantify how annotation noise and annotator bias affect an embedding-based classifier, identify reliable annotators, filter training data, and improve robustness.

Start by loading the train/validation/test splits and answer core questions: how many unique annotators labeled the training set, what is the label distribution (counts and relative frequencies), and what is the embedding dimensionality. While exploring the data, note common difficulties with human labeling (disagreement, systematic bias, inconsistent guidelines, and class imbalance) without proposing solutions yet.

Next, train a baseline classifier that consumes the provided embeddings and report standard metrics (accuracy, precision, recall, F1) on validation and test splits. Observe any generalization gaps between validation and test.

For annotator identification, use only dataset signals (annotator_id, their labels, and model behavior) to distinguish Expert, Decent, and Spammer pools. Filter training examples to keep only those from reliable annotators, retrain the same model on the filtered subset, and compare metrics to the baseline.

Finally, propose next-step improvements (de-noising decent annotators, confidence-weighted label aggregation, LLM-assisted review, active re-sampling, calibration) with one-line rationales and assumptions. You should demonstrate practical evaluation choices and a clear path toward production readiness.