Define a reward signal around the task.
A rubric defines what a task requires: which claims need evidence, which constraints matter, and which errors make the result unacceptable. A verifier checks outcomes such as a saved file, a populated field, or passing tests.
We build the scorecards, reviewer guidance, and verification criteria with domain experts. Scored examples record the evidence behind each decision so your team can use them for training and evaluation.
Turn requirements into reviewable criteria
Separate completion from quality. Define criteria for grounding, completeness, instruction following, and domain judgment, with examples of acceptable, borderline, and failed work.
- Define observable criteria and explicit failure conditions.
- Calibrate expert reviewers on shared examples.
- Retain the evidence behind a score or decision.
Use verifiers where outcomes can be checked
Use automated checks for results the environment exposes. Keep expert review for nuanced work, and test both against representative failures and shortcuts.
- Inspect final state as well as the action trace.
- Test checks against shortcuts and superficial completion.
- Keep evaluation tasks separate from training examples.
What a program can include
Scope can include:
- Task-specific rubric definitions and rating guidance
- Expert-scored examples with supporting evidence
- Verifier specifications for testable outcomes
- Failure categories to guide the next training cycle
Illustrative task / Not a production record
Make the objective concrete.
- Objective
- Review a proposal against its stated requirements.
- Context
- The proposal, requirement document, and expected review format.
- Review
- Check whether each requirement is addressed, recommendations cite evidence, and unresolved gaps are clearly identified.
Common questions
How do rubrics and verifiers work together?
Rubrics describe what good work means. Verifiers check outcomes that are observable and testable. A program can use both, keeping expert judgment for nuanced criteria and automated checks for concrete completion conditions.
What should an enterprise team bring to scoping?
Bring representative tasks, examples of accepted and rejected work, the target environment, and the requirements that matter most. The team can then define the rubric, review workflow, and verification scope.