Agentic-Coding Evaluation Lab Methodology Report

This page is an automatically rendered view of a single validated, public-safe evidence file — a demonstration of the reporting layer, not a hand-written document. Before anything renders, the evidence is independently re-validated and re-scanned, and rendering refuses if any required honesty label has drifted.

What this demonstrates: deterministic report rendering, label checking, and refusal behavior on one validated evidence file.

What this does not demonstrate: any claim beyond that one evidence file. The campaign header below records the file’s provenance; every example report published on the public site is synthetic and demonstrates the reporting layer, not model performance.

Campaign Header

campaign_id: synthetic-demo-001
created_at: 2026-07-09T00:00:00Z
evidence: evidence/sample-lab-report.json

Note

Provenance: synthetic
Sanitization status: public-safe
Sanitization checked: true
Redactions required: false

Capability Table

Role	Successes / trials	Rate	Wilson CI
incumbent	4/6	66.7%	95.0% CI [0.299988, 0.903231]
candidate	5/6	83.3%	95.0% CI [0.436491, 0.969947]

Paired Delta

Point estimate	Bootstrap CI	Seed	Iterations	Alpha
0.166667	95.0% CI [0, 0.666667]	12345	2000	0.05

Decision

Decision: false
Label: enforcing:superiority-by-margin
Rule: rule=bootstrap_ci_low_gt_margin
Margin: 0.1

The point estimate clears the margin, but the two-stage bootstrap lower bound is not strictly above the margin.

Enhanced Estimators

Enhanced estimators are additive diagnostics for new analyses. They are explicitly non-gating here and do not strengthen or relabel the primary result.

Estimator	Label	Status	Rendered values	Gate role
Two-stage bootstrap	enhanced:two-stage-bootstrap	reported	estimate 0.166667; 95.0% CI [0, 0.666667]; seed 12345; iterations 2000; alpha 0.05	non-gating additive estimator
Wilcoxon signed-rank	enhanced:wilcoxon-signed-rank	reported	statistic 0; p_value 1; n 1	non-gating additive estimator
GLMM logistic	enhanced:glmm-logistic	optional-dependency-missing	formula n/a; backend n/a	non-gating additive estimator

Sign Test

The sign test is two-sided, reported-only, and does not gate.

Wins	Losses	Ties	p_value	alternative	reported_only	Gate role
1	0	2	1	alternative:“two-sided”	reported_only:true	does not gate

Raw Outcomes Summary

Tasks: 3
Replicates: 6
Task classes: fix_failing_test=1, multi_file_refactor=1, small_edit=1

Task class	Tasks	Replicates	Incumbent successes	Candidate successes
fix_failing_test	1	2	1	1
multi_file_refactor	1	2	1	2
small_edit	1	2	2	2

Reproducibility Manifest

Seeds

Name	Seed
bootstrap	12345
power_simulation	12345

Task Set

task_set_hash	task_count	replicates_per_task	fixture_policy	task_classes
sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa	3	2	synthetic	small_edit, multi_file_refactor, fix_failing_test

Models

Role	Label	Version	Quantization
incumbent	incumbent-default	synthetic-v1	n/a
candidate	candidate-a	synthetic-v1	n/a

Cost

Currency	Total	Basis
USD	0	synthetic local example

Preflight Results

Status: passed
Checked at: 2026-07-09T00:00:00Z

Model role	Resolved	Model version	Notes
incumbent	true	synthetic-v1	synthetic
candidate	true	synthetic-v1	synthetic

Core

Core version	Core content hash
0.2.0	sha256:7ec59090e3196b225ca2d68bc76cb6703b867f259f87e9d9e2bcafce896bf6ca