Tests and councils

The corpus is the collection of test inputs, expected results, and the policy reasons for those expectations. Its coverage must include attacks the verifier should reject and benign agents it should accept. A large count of similar cases can leave an entire kind of failure untested, so case count alone will not determine readiness.

Use fixed deterministic checks for behavior that can be tested without a model. Use live semantic tests to measure how the selected model judges prepared agents. Add budgeted adversarial agents that search for failures beyond the prepared collection. Each method answers a different question. Saved-response replay remains useful for testing the surrounding code, but does not measure current model behavior.

Every authoritative test result identifies the code, configuration, runner, and inputs it concerns. A signed receipt records those references, the observed results, and what the signer verified. Tests may run outside a trusted execution environment, or TEE, provided the signer has adequate evidence of their execution. Signing an arbitrary uploaded success report would only authenticate the report's receipt.

Grow coverage while building the first working evaluation path. Existing fixtures are a starting inventory, not proof of adequate live-model coverage. Choose whole-program or chunked source review through evaluation of missed context and cost. Budget for evidence storage, model calls, reruns, and active attack search so the required release process is affordable to repeat.

Changes that remove a case, relax a threshold, or turn an expected failure into a pass receive explicit review. Correcting a mistaken expectation is allowed, with a recorded policy reason. A change that affects a frozen release's exam invalidates the affected evidence.

Keep exploit-disclosure rules separate from held-out evaluation. The former controls when a discovered attack becomes public. The latter limits tuning to known tests. Publication should depend on the fix and exposure of affected credentials, with a defined limit on withholding details. Keeping a private copy for reruns does not require withholding the public case.

Customer code enters the public corpus only when the customer deliberately prepares and contributes a sanitized case. Private review submissions are not a source of test material the DAO can publish at will. Use contributor-owned or synthetic examples for ordinary testing and defect reproduction.

Evaluation requirements

Set evaluation requirements before observing the candidate's results. Each requirement identifies a behavior, relevant cases, acceptable outcomes, repetition policy where the model varies, and the evidence needed to establish completion. Safety failures and false rejections must remain visible separately. An aggregate score must not conceal a failed mandatory requirement.

Full evaluation covers changes to models, prompts, policy, encryption, attestation, credential validity, and other behavior that can change what the verifier accepts or exposes. Targeted evaluation is available when the maintainer can establish a narrower effect under the classification rule. A small diff can still require full evaluation.

An example case may contain a synthetic embedded private key that the stated policy prohibits. Its expected result is rejection. A paired benign case can describe how to configure a key without containing one, with acceptance as its expected result. The actual policy must justify both expectations. Matching those expectations measures those cases, not every possible program.

Require the full prescribed result set, including failures and incomplete runs. Distinguish a transport retry from a fresh model trial. A contributor cannot rerun until the desired answer appears and report only that attempt. Changes to trial counts or thresholds after observing results need a recorded rule decision and any required reevaluation.

Pin provider versions where possible and record the provider evidence available. If a provider cannot guarantee stable behavior under an identifier, define periodic reevaluation and drift handling. An outage cannot silently replace the verifier's model with an unevaluated substitute. Budget recurring model evaluation as an operating cost.

Coverage of the actual change

Every candidate needs a mapping from changed behavior to the checks that exercise it. A contributor's label does not establish the scope. The controller compares the actual source, dependency, deployment, model, prompt, and permission changes with the adopted classification rules.

U02 opens the coverage task and identifies likely affected behavior. If code has not been written yet, its output is a preliminary evaluation requirements. U06 verifies the exact implementation diff before merge. A full candidate requires a coverage decision from a qualified council assigned separately from its author and scope approver. If U02 already assessed the same exact diff, U06 can reuse that decision after checking its references.

The coverage council reads the actual changes and identifies each affected behavior, the relevant existing tests, new tests or integration checks needed, and behavior the available tests cannot establish. Network access, secret handling, model settings, prompt changes, and permission changes deserve explicit treatment. A passing corpus run cannot demonstrate that an unrelated new outbound connection is harmless.

The decision either establishes adequate checks for the specified scope, requires additional evidence, or leaves a gap unresolved. Missing coverage prevents merge eligibility. A policy change can alter requirements through F9, but the council cannot disguise that change as a passing test. Once the required checks are established and pass, no second coverage opinion is required at U12.

Targeted candidates can use published mechanical classification rules without a council task. The founding rule permits targeted evaluation only for changes whose limited effect can be established through those rules. Unknown paths, changed dependencies, or uncertain behavioral effects escalate to full evaluation. Later rules can permit more targeted changes after their scope checks have been demonstrated.

A material revision invalidates affected coverage evidence. U07 checks combined contributions against the actual merged tree so individually acceptable changes do not bypass examination of their interactions.

Evaluation cost

Measure the first complete evaluation before adopting it as a repeatable production requirement. Record model calls, retries, live trials, attack-search work, runner time, evidence storage, hosting, council work, and cost of the challenge and recovery allowance. Report cost per attempt and per accepted release so repeated failures remain visible.

The release-policy decision includes that measurement, a funded per-attempt limit, and a recurring evaluation budget. Test a provider-price increase or failed-attempt scenario before committing the budget. If the required checks exceed the allowance, keep the attempt incomplete and change funding or future policy through its proper authority. Do not remove a check midway through a failing run to make the bill fit.

Token trading receipts are not required to fund the first rehearsal. Any actual compute source can fund it under a bounded authorization. Record that source without assuming it is a founder contribution.

Council qualification

An agent council is a group of authorized AI evaluators assigned to a question that requires judgment. Qualify each council for its intended work before granting it authority. Its benchmark should include real changes, planted defects, harmless changes, ambiguous policy cases, false positives, and attempts to influence its own permissions or evaluation rules.

Qualification can run offline. Record missed defects, false rejections, abstentions, cost, and repeatability. Use cases the council has not simply memorized from its qualification training. Different model brands do not prove independent reasoning or independent key control.

Governance grants the assignment after the council meets the adopted criteria. The assignment defines the agreement rule, treatment of disagreement, deadline, replacement route, and appeal authority. The council cannot expand its own powers or lower its own qualification standard. A replacement council can offer an explicitly accepted cost or performance tradeoff while still meeting mandatory criteria.

Treat source code, submitted reports, and tool output as untrusted input. A sentence inside a contribution cannot change a council's system instructions or grant a tool permission. Test those attacks against the actual tools and decision interface before the council receives authority.

The decision interface separates trusted instructions from submitted material. Tools produce a structured inventory of changes and evidence references. The council can inspect the underlying raw diff and source as untrusted attachments. A summary is a navigation aid and cannot replace inspection of a sensitive change. Tool output can also contain hostile text, so its format alone does not make it safe.

Require a typed output containing the assignment, exact target, question, disposition, required checks or corrections, evidence references, reasons, and permitted effect. Reject invalid fields and effects outside the assignment. The council has no general shell, treasury, or release permission merely because its output can influence a decision.

Assign benchmark maintenance to an evaluation maintainer distinct from the council being qualified. That role rotates planted defects, creates fresh variants, controls private benchmark access, and records which cases have been exposed. Requalify after a material model, prompt, tool, or council-configuration change. Budget benchmark maintenance and requalification as recurring operations. During founding operation these roles can share an operator, which provides task separation without proving independent control.

Human specialists can receive defined audit or investigation assignments. A human admission exam and a fixed number of human comparisons are not prerequisites for the automated release path.