Reporting that refuses when the evidence cannot support a conclusion
A synthetic-only draft demonstration; no scientific method or promotion rule is selected
Fail-closed evidence validation
Preventing incomplete evaluations from producing unsupported conclusions
What is this? A prototype verifier that recomputes an evaluation’s published result from its frozen evidence bytes and refuses when the evidence is mutated, mismatched, or missing a required prerequisite.
What is it for? It prevents an incomplete study from being mistaken for a valid model result.
How is it used? The verifier checks whether the statistical method, replication requirements, and minimum useful improvement have been defined. If any required element is missing, it returns NOT_EVALUABLE — no scientific conclusion can be drawn — and issues no model recommendation.
Why does it matter? A trustworthy evaluation system must know when not to decide. This behavior helps model-testing teams avoid false confidence and creates an auditable record of what remains unresolved.
Evidence and status Synthetic prototype demonstrating contract validation and independent refusal behavior. It does not measure a real model.
Examine the evidence contract and verifier →
- Status: Prototype
- Evidence: Synthetic
- Decision:
NOT_EVALUABLE— no scientific conclusion permitted - Demonstrates: Independent evaluation reporting that refuses to report a result rather than guess when the evidence is insufficient
- Does not demonstrate: Performance of any real model
- Relevant to: Model testing, challenger evaluation, and model-risk review
The problem
This prototype tests whether an evaluation system can decline to make a model decision when the study design is incomplete, instead of manufacturing a pass-or-fail result.
What I built
A draft evidence contract (v2, 2.0.0-draft.1) that binds a complete synthetic evaluation record — registration, schedule, attempts, observations, analysis, decision, and report — together with a separate verifier that recomputes the published result from the frozen evidence bytes and refuses on any mutation, mismatch, or missing prerequisite.
What this demonstrates
Using synthetic data, the report below and the verifier confirm that the statistical method, required repetitions, and minimum useful improvement have not yet been approved. The system therefore returns NOT_EVALUABLE — no scientific conclusion can be drawn from this evidence — instead of a pass-or-fail result.
What this does not demonstrate
This page is a structural contract demonstration, not model evidence; it does not measure the performance of any real model. The contract and verifier are unreleased repository-only drafts outside repository package version 0.2.0. The inferential basis and enforcing test (the test that determines whether evidence supports a decision), replication policy (how many repeated trials are required), and practical-benefit threshold (the minimum improvement worth changing models for) remain undecided and unauthorized, so the only permitted outcome is NOT_EVALUABLE.
Why it matters
In a company model-testing workflow, the same pattern can prevent an incomplete experiment from triggering a change from the current model to a challenger. It also creates an auditable record explaining why no decision was made and what evidence is still required.
Technical details
The report below is the machine-verified artifact. Each time this page renders, the verifier (verify_exact_bytes) recomputes it from the frozen synthetic evidence bytes; the text that follows is generated, not hand-written.
Synthetic Contract B v2 refusal example
NOT_EVALUABLE — Contract structure and independent refusal behavior only; no model or population claim.
Registration
- Receipt:
sha256:6c19e7043ea05efbc0961dd1ea0c52ab6a5dd92fe2dd29fe33d7c7f15f72e8ae - Informational verification result:
sha256:39f71065f2c014d5a423b25d55ed54414e28b811161b103b98283c0d3ad20ad1 - Provenance:
copied-attested-synthetic
Design and data integrity
- Campaign type:
synthetic_contract_check - Scheduled attempts:
4 - Observed attempts:
4 - Gateway retry attempts:
0 - Method authority:
held - Science protocol:
sha256:e3aca988595623675cae6147cfff0cbcc38856cb6855640cbbddf01223658d51 - Task-set reveal:
sha256:a6c2e80f695ea0a988d2f9139832ebc03ee88013d59abe5ef6711f11659fb33e - Assignment schedule:
sha256:30e1fd2418b526edb48be1f8cc62a61ad4404a2c56ed01b419bca2679b2c6122
Analysis
Status:
not_evaluableStatistics emitted:
0Reason codes:
METHOD_AUTHORITY_HELD— No inferential method has been accepted for enforcing use.R1_R2_R4_HELD— The inferential basis and test, replication policy, and practical-benefit threshold remain undecided and unauthorized.
No enforcing test, promotion rule, replicate policy, margin, or minimum practical benefit is installed by this draft.
Provenance and independent verification
- Gateway invocation-open receipts:
4 - Gateway call receipts:
4 - Gateway invocation-closure receipts:
4 - Editing-harness descriptor-validation receipts:
4 - Evaluation-harness completion receipts:
4 - Provenance upgrades:
0 - Verified evidence:
sha256:7554c3e4336be0a47371aa6b4c16ae17c71f104586b993580880582dd1873911 - Contract schema:
agentic-coding-evaluation-lab.evidence.v2.0.0-draft.1 - Contract schema bytes:
sha256:2b9a6befee9cf96bcef309aface080bf536c01b59e6a4351498c41fd4c4e4c23 - Verifier implementation:
agentic-coding-evaluation-lab.contract-v2-verifier.v2.0.0-draft.1 - Verifier/report/safety implementation bundle:
sha256:c1a01cebecbb685c158e7fa0a6fe5084fcbc08c54b3b41d169e9ce7c6a5f9899 - Recomputed analysis result:
sha256:32d86d273c58dda8e9aa411e2b2d46efa75a332e8ad42e84706b6e6d5f72e1d4 - Recomputed decision receipt:
sha256:e8fd7dda6a54541f0b9a28a7318421d867fa080eedf6b5fcad6be60fb755ce2a
Limitations
- Synthetic structure only; no real request, response, task, grader, model, or deployment observation.
- No inferential basis, enforcing test, replicate policy, margin, or minimum practical benefit is selected.
- Copied verifier metadata is informational and is not independent cryptographic verification.
- The registry verification result refuses under D2; no operational signer identity, issuer, trust policy, signature, or trusted timestamp is claimed.
- Editing-harness, evaluation-harness, and gateway owner receipt schemas and provenance classes are synthetic unratified fixtures, not operational adapters.
- Operational adoption must validate exact owner receipts against frozen owner schemas and must not trust copied source envelopes as verification inputs.