Make expert preference usable for training.
Preference data records why one response serves a task better than another. Reviewers compare correctness, completeness, reasoning, and usefulness against the same objective and source material.
We collect expert comparisons and rationales for RLHF and other preference-optimization workflows. The criteria and delivery format follow your training approach.
Make the comparison criteria explicit
Distinguish correctness from style and completeness from verbosity. Give reviewers guidance for ties, partial answers, and cases where both responses fail.
- Define the dimensions behind a preference.
- Preserve the prompt, context, and compared outputs.
- Record why the preferred response is better.
Use disagreement as a quality signal
Inspect disagreement for ambiguous criteria or missing context. Calibrate reviewers on shared examples and reserve separate material for evaluation.
- Review ambiguous and conflicting judgments.
- Calibrate ratings against representative examples.
- Maintain task coverage beyond easy comparisons.
What a program can include
Scope can include:
- Pairwise or ranked response comparisons
- Expert preference labels and written rationales
- Rating rubrics and calibration guidance
- Agreed schemas for your preference-training workflow
Illustrative task / Not a production record
Make the objective concrete.
- Objective
- Choose the more useful answer to a source-grounded research question.
- Context
- The question, reference material, two candidate answers, and a review rubric.
- Review
- Compare evidence use, completeness, reasoning, and instruction following; explain the preference and any remaining errors.
Common questions
Can preference data support DPO?
Preference pairs can be used in direct preference optimization workflows. Define your training method and data schema during scoping so that prompts, compared responses, and labels are organized appropriately.
Should reviewers always choose a winner?
The review policy should define ties and cases where both answers fail. Forcing a preference without guidance can hide important quality differences and disagreements.