[kaihackney-scale]

Why

When one agent judge gets a rubric with hundreds of criteria, it stops grading them one by one. In a real run with 419 criteria, the judge grepped the trajectory for a handful of keywords, wrote a script that hard-coded a short list of failing ids, and gave every other row the same one-word justification. Ten runs on the same task returned pass counts anywhere from 75 to 370 of 419, so the score carried no signal.

What

rubrics_verifier gets two optional fields. Like judge_effort, both are also read from user_overrides:

• judge_batch_size: splits the criteria into batches of N. Each batch gets its own judge call with the same prompt, trajectory and env access, and the rows are merged back in criterion order. The judge still sees the short keys c1..cN, so ids come back as before. A failure in any batch fails the step with that batch's error, as the unbatched judge would.
• judge_batch_concurrency: how many batches run at once (default 4). Every A2A call is already a fresh judge session (context_id=None), so batches can share the judge's sandbox.

With judge_batch_size set, each prompt also says to grade every row on its own evidence and cite it. A batch whose rows read as bulk verdicts gets a correction prompt and is retried within the existing retry budget. Bulk verdicts means at least 25% of rows (and at least 5) share one justification, or at least half the justifications are under 20 characters. If the last attempt is still bulk, the rows are kept and the batch is logged in judge_output_discrepancies with accepted: true, so the verify doesn't fail on it. A batched entry also records judge_batches and judge_batch_trajectory_object_urls.

Without judge_batch_size, the step behaves exactly as before: one call, no bulk check, and the same error types. rubric_evidence* formats reject batching, because their single reasoning/checks/mistakes block would be split across batches.

Testing

• tst/unit/task_step/verifiers/test_rubrics_verifier_judge_batches.py (16 tests): batches merge in order; the batch size can come from user_overrides; there's no bulk check without batching; the concurrency cap holds; bulk verdicts are retried with a correction, then kept and flagged on the last attempt; a failing batch raises its own error; options round-trip and validate; batch size and concurrency come from user_overrides and an invalid override raises; formats with whole-run rows reject batching; a cancelled batch cancels its remote judge task.

• tst/unit/task_step/verifiers: 363 pass. Full tst/unit: 8728 pass. The 2 failures in test_mcp_cli_builder.py fail the same way on clean main.

• End to end: I re-graded one real 419-criterion run with an agent judge that has read-only env access, at effort=high, three ways: as one judge call (two runs), and with judge_batch_size=30 (concurrency 4).

  	one call	14 batches of 30
  wall time	~650 s	625 s (4 batches overlapping throughout)
  judge tool calls (env MCP calls)	39–51 (18–27)	307 (144); 11–28 per batch
  rows passed	229 / 231	229
  distinct justifications / median length	303–380 / 31–59 chars	419 / 156 chars
  bulk-verdict retries	n/a	0

  The batched rows came back in criterion order with the real ids, and they agree with each single-call run on 407 of 419 rows (the two single-call runs agree with each other on 409). For comparison, the same run graded today at effort=low in one call made 6 tool calls and gave 339 rows the justification "Addressed." Replaying that output through the bulk check in batches of 30 flags all 14 batches. The same replay over the two single-call effort=high outputs flags 0/14 and 2/14; the two flagged batches were mostly terse "Not recorded." fails, so they would be retried and then kept.

Docs: the site's rubrics_verifier reference should list judge_batch_size and judge_batch_concurrency.

🤖 Generated with Claude Code