Which agents to manage —
and how to verify them.
Managed agents can supply real-world RL environments at scale only if three questions have answers. This registry is groundwork for each:
A managed agent is the source for an RL environment, not itself one — separate the pinned policy/harness, add a task distribution, a reset contract, and a verifier (which doubles as the training reward). Sequencing: ship verification as a production-reliability feature first; paid-traffic opt-in via data-for-tokens (credits/discounts for verifier + trajectory sharing), hard no-training default, A2A-style opacity. Open-ended tasks split: their structure — queries, calculations, citations, schemas — is checkable by code; only the residual judgment needs rubric grading.
The verifier schema — success_criteria (the task-verification field)
success_criteria:
evaluation_mode: verifiable_first # verifiable_first | hybrid_ensemble | trajectory_only
combine: gated_weighted_sum # not product — vanishing rewards
reliability: {metric: "pass^k", k: 4}
reference_pass_rate: 0.15 # solvable-but-not-trivial; guards degenerate tasks
budgets: {verifier_latency_budget_ms: 2000, verifier_cost_ceiling: 0.02, max_turns: 30}
hard_gates: # fail ⇒ reward 0, separate from scoring
- {type: safety | authorization | irreversible_side_effect}
checks:
- {type: test_execution | semantic_state | exact_match | must_communicate | llm_rubric,
scope: final_state | trajectory | side_effect, weight, canonicalize: [ignore_timestamps, ignore_ids]}
negative_rubrics: [{pattern_id, penalty_weight}] # anti-reward-hacking
Prioritization — value × demand × feasibility
V·D·F each 1–10, modeled heuristic (not measurement) · top 6 = core set · ★ = launch trio · tag = verifier type · click a row for evidence + sourcesGolden set — internal first
benchmark the builder on tasks the team can judge · target 100+ golden agents · seeds double as Tier-1/2 RL environmentsBuilder quality metric — skill to build skills: a generated skill is graded against a canonical source-of-truth skill (Cloud · Shopify · Vercel) — structural validity, functional equivalence on the same task suite (primary), rubric delta vs canonical. The census is the source-of-truth library.