Experimental inference note — exact computation is not exact inference
Synthetic decision input on estimands, candidate tests, repeatability, dependence, and feasibility. No method is selected.
- Status: Executable methods review — synthetic decision input
- Evidence: Synthetic only — no live model outputs and no campaign data
- Decision: None — this note informs a pending methods ruling and selects nothing
- Demonstrates: That a bit-exact, fully reproducible calculation can still exceed its nominal error rate — reproducibility is not validity
- Does not demonstrate: Performance of any real model, or a validated statistical design
- Relevant to: Statistical method review for model evaluation
Before this project commits to a statistical rule for deciding between models, the candidate rules themselves are being tested — on invented data, with every calculation committed and byte-checked. The central finding so far: a rule can be perfectly reproducible and still be wrong more often than it claims. That is why no rule has been adopted yet.
The evaluation lab’s rulings R1 (the inferential basis and enforcing test — the statistical footing and the test that determines whether evidence supports a decision), R2 (replicate and nondeterminism semantics — how many repeated trials are required and how run-to-run randomness is treated), and R4 (the margin — the minimum improvement worth acting on) are held as one coupled decision. Held means not decided: the rulings are still awaiting an explicit decision, in either direction. Nothing in this note selects, installs, or enforces an inferential basis, a test construction, a replicate policy, a margin, or an MPIB value (the minimum practically important benefit — the smallest improvement worth changing models for). Every preference expressed here is labeled as a recommendation to the coupled R1/R2/R4 ruling; only that ruling — an operator signature, not this package — can select. Until a rule is selected, every simulation below compares candidate rules; none of them is an operating characteristic of an accepted enforcing rule.
This note is design-only and synthetic-only: no live model outputs, no campaign data, and no claim about any real system. Every package-derived number is produced by committed, seeded, repository-only experimental code — analysis/run_method_tranche.py, analysis/_method_tranche/paired_trinomial.py, and analysis/_method_tranche/dependence.py — and is read at render time from the committed artifact analysis/method-tranche-results.json. The externally supplied comparison value is labeled as such and recorded in that artifact; it is not a package result. These modules are deliberately excluded from the repository-versioned 0.2.0 statistical core while the method decision remains open. This document refuses to render if the artifact’s honesty labels or decision-input citations have drifted.
Deterministic enumeration can reproduce a reference distribution and every reported probability bit-for-bit while the resulting rule still exceeds its nominal Type I error under the stated composite-null model. Exact arithmetic is therefore a reproducibility property, not a validity guarantee.
The calculations below also do not validate the intended class-balanced task design. They use a common i.i.d. paired-trinomial model and a finite nuisance grid. That mismatch is a result of this review, not a detail to hide.
What this tranche is
This is the interim decision-input tranche — the first installment — of a commissioned public method package. It covers the commission’s required content 1 (inferential- basis comparison), 3 (assignment unit and mechanism), 4 (assumption audit), 5 (the weak-null problem and the enumerator/exactness distinction), 6 (the three candidate constructions), 8 (superiority at zero versus MPIB), and 9 (replicate and nondeterminism semantics), together with six scenario families: null-boundary behavior, heterogeneous effects, margin feasibility (with the falsifiable cross-check), superiority/MPIB operating characteristics, nondeterminism-floor propagation, and cross-task dependence under session layouts. The replicate-lattice demonstration required by the commission is included with the weak-null section.
The complete package will add the remaining content: the full formal estimand treatment, the finite-sample bounded-mean (betting/confidence- sequence) constructions of Waudby-Smith and Ramdas, and full report and figure treatment. Nothing in this tranche presumes what that bounded-mean evaluation will show; in particular, nothing here presumes it overturns — or fails to overturn — the feasibility picture presented below.
No numeric gate is recommended anywhere in this tranche — no margin, no MPIB value, no alarm threshold, no sample size. The scenario families state consequences; gates come later, if at all, and only after their operating- characteristic scenarios have been reviewed through the coupled ruling.
Notation, model, and provenance of package-derived numbers
Per task \(i\), the paired difference of aggregated task successes is \(d_i \in \{-1, 0, +1\}\) with \(P(d_i = +1) = p_+\), \(P(d_i = -1) = p_-\), and \(P(d_i = 0) = 1 - p_+ - p_-\). Two derived parameters index every scenario: the mean difference \(\Delta = p_+ - p_-\) and the discordance probability \(\pi_d = p_+ + p_-\). The suite statistic is \(S = \sum_i d_i = n_+ - n_-\). Throughout, \(\alpha = 1/20\) one-sided, and suite sizes \(N \in \{20, 40, 60\}\) are examined.
This trinomial model is itself an assumption — an i.i.d. (or, in the heterogeneity family, independent-but-heterogeneous) sampling model for per-task differences. Section [The two inferential bases] states which basis gives such a model a referent. Every probability below is either
- deterministically enumerated: an integer dynamic program (DP) over the outcome lattice computes the finite distribution with exact
fractions.Fractionarithmetic — no Monte Carlo iterations, no seeds, no tail-simulation error (the committed JSON records the exact rationals); or - seeded Monte Carlo: the cross-task-dependence family simulates a stated synthetic mixture model with pinned seeds (base seed 20260713, one derived seed per cell) and reports binomial Monte Carlo standard errors.
Reproduction, from the repository root:
PYTHONPATH=src python3 -m analysis.run_method_tranche --check # refuses if committed bytes drift
python3 -m unittest discover -s tests -v # includes byte-level reproducibility tests for the artifactTwo claims are kept separate everywhere in this note — in prose, in table captions, and in the committed code’s comments and test names.
- Computation claim. “Deterministic integer-DP enumeration” says the reference or sampling distribution was computed exactly, with rational arithmetic, eliminating simulation error. It is a statement about arithmetic.
- Inference claim. “Finite-sample valid at level \(\alpha\)” says a rule’s rejection probability never exceeds \(\alpha\) under the null, under stated sampling assumptions. It is a statement about inference.
The first never implies the second. Fisher-style randomization tests are finite-sample exact for compatible sharp nulls; the weak (composite average-effect) null \(H_0\!: \Delta_{\text{suite}} \le \delta_0\) requires additional construction or assumptions (Wu and Ding 2021; Zhao and Ding 2021). The null-boundary family below demonstrates the gap numerically: a shifted sign-flip rule whose reference distribution is enumerated exactly — exact arithmetic — nevertheless exceeds the nominal level on the composite boundary at \(\delta_0 = 1/10\).
The two inferential bases (required content 1)
The candidate estimand under review — not an accepted or recommended choice — is the equal-class-weighted mean difference in task success over the frozen 40-task suite, with equal task weight inside each of four registered classes, per-task success aggregated over registered replicates, difference between arms. With complete data and equal class sizes (ten tasks per class) the equal-class- weighted mean equals the unweighted task mean \(\Delta_{\text{suite}} = \tfrac{1}{40}\sum_{i=1}^{40} d_i\) of the per-task differences in aggregated success; missingness breaks that identity, follows the registered attrition rule, and is never silently dropped. The candidate scientific null under review is the weak (composite average-effect) null \(H_0\!: \Delta_{\text{suite}} \le \delta_0\) — a statement about the suite mean, not a sharp task-level statement. None of this is yet accepted: R1 is held, and this section is input to it.
Two inferential bases must be compared without presupposing either:
| Task superpopulation over four classes | Frozen-suite census | |
|---|---|---|
| Target | The superpopulation mean difference \(\Delta_{\text{super}} = \tfrac{1}{4}\sum_k E[\delta(T) \mid \text{class } k]\), where tasks are treated as drawn (stratified by class) from a larger population of tasks of the four registered kinds | The suite quantity \(\Delta_{\text{suite}}\) itself — a computed, described number over exactly the frozen tasks |
| Uncertainty admitted | Between-task variation is sampling variation; plus any registered stochastic components | No task-sampling uncertainty at all. Residual uncertainty, if any, comes only from registered stochastic components (stochastic decoding over a frozen seed schedule; measured apparatus nondeterminism). Under fully deterministic service there may be none to model |
| Assumptions required | A stated and defended sampling model: the frozen tasks behave like a stratified draw from a definable population; per-task differences independent across tasks (audited below) | Correct computation and the registered attrition rule; nothing else |
| Claim licensed | An inferential statement about tasks of these kinds, with size/power meaningfully defined | A descriptive statement about exactly these tasks, with no generalization and no error rates to control |
Two consequences shape everything downstream:
- Every operating characteristic in this note is basis-dependent. Size and power are probabilities under a sampling model. The trinomial model above is a superpopulation-style model (or a census model with registered stochastic components). Under a pure census basis with fully deterministic service, “Type I error” has no referent: there is nothing random, and the suite mean is simply a number to report.
- The census report is unconditional. Whatever basis the coupled ruling selects, every downstream report must carry the frozen-suite census descriptively alongside any inferential result. This tranche treats that commitment as binding on the package regardless of the ruling’s outcome.
Every enumerated operating characteristic in this tranche uses either a common i.i.d. paired-trinomial model or one of five hand-specified heterogeneous diagnostics. None is a proof for independent non-identically distributed tasks, equal class weighting, or a class-stratified test. Consequently, no rule evaluated here is yet eligible to enforce the intended four-class estimand. A later package must supply a stratified/non-i.i.d. guarantee or narrow the inferential claim.
Report the frozen-suite census descriptively in every downstream report — unconditionally. If an inferential layer is wanted at all, license it explicitly: either (a) adopt the task-superpopulation-over-four-classes basis, stating and defending the stratified sampling model and the independence assumptions audited below, or (b) remain on the census basis and confine inference to registered stochastic components (seed schedules, measured apparatus nondeterminism), accepting that with fully deterministic service there is nothing left to test and description is complete. What the package cannot honestly offer is task-level inference without the superpopulation assumptions — that combination is what the operator’s hold identified as unsupported.
Assignment unit and mechanism (required content 3)
The design is paired within task: both arms run every task. What is randomized (or balanced) is arm order at the registered assignment unit — not task membership, and not which arm a task receives.
What the mechanism licenses, exactly:
- Arm-order randomization controls period and order effects — a confound-control function. Within-pair differencing cancels a session main effect (anything that shifts both arms of a pair equally, such as shared warm-up state at that point in a session) and balances order- and cache-warmth effects across arms in expectation.
- Arm-order randomization does not generate the randomization distribution of the model contrast. Permuting arm order does not resample tasks and does not flip the sign of a task’s true effect; there is no design-induced reference distribution for \(S\) or for any statistic of the \(d_i\). Any sign-flip or permutation reference distribution must therefore be justified by an explicitly stated assumption — for example, symmetry or exchangeability of paired differences under the null — and never attributed to the randomization actually performed.
- The mechanism does not cancel a session-by-arm interaction: anything that pushes the two arms of a pair apart in a session-correlated way survives differencing and induces correlated per-task differences. The cross-task-dependence family quantifies the consequence.
Accordingly, every reference or sampling distribution used in this note is licensed by a stated assumption, never by the assignment mechanism:
| Rule / computation | Reference distribution generated by | Stated assumption that licenses it |
|---|---|---|
| Grid-boundary-calibrated rule \(S \ge c\) | Integer-DP enumeration of the common-trinomial suite distribution on a finite null-boundary grid | Common i.i.d. paired-trinomial sampling, independence across tasks, and a claim limited to the enumerated nuisance grid; not the intended stratified design |
| Shifted sign-flip rule | Integer-DP enumeration of \(\sum_i \varepsilon_i \lvert d_i - \delta_0\rvert\) over uniform \(\varepsilon \in \{-1,+1\}^N\) | Symmetry of the centered differences \(d_i - \delta_0\) under the null (a sharp/symmetry hypothesis) |
| Normal-approximation \(z\) rule | Asymptotic normal reference for the studentized mean | Independence across tasks plus large-\(N\) asymptotics |
| Conditional (McNemar-style) constructions | Binomial reference conditional on the discordant count | Common discordance parameter / exchangeability of discordant pairs |
| Cross-task-dependence Monte Carlo | Seeded simulation of a stated session-mixture model | The synthetic mixture model itself (stated in full in analysis/_method_tranche/dependence.py) |
The weak null, the enumerator, and the replicate lattice (required content 5)
\(H_0\!: \Delta_{\text{suite}} \le \delta_0\) is a composite average-effect (weak) null: it constrains a mean, not each task’s effect. Sharp-null randomization inference does not automatically transfer to it. A sign-flip test is finite-sample exact when the null makes the flipped sample distributionally equivalent to the observed one — a sharp symmetry statement. Under the weak null, task effects may be heterogeneous with mean at the boundary, and no distributional symmetry of \(d_i - \delta_0\) follows; The cited studentized/prepivoted Fisher-randomization results (Wu and Ding 2021; Zhao and Ding 2021) concern randomized treatment-assignment designs. This study runs both systems on every task and randomizes only arm order, so those theorems do not transfer automatically. A task-sampling or distributional theorem specific to this design is still required.
The integer DP used throughout is a legitimate deterministic enumerator of whatever reference distribution is selected — and the preferred one, whenever the selected reference distribution is itself legitimate under the selected basis, because it removes Monte Carlo iterations, seeds, and tail simulation error. What it does not do is make any rule exact for the weak null. Computation claims and inference claims are stated separately in every sentence and caption of this note; the numerical demonstration is in the null-boundary family below.
The replicate lattice, and the retired \(\delta_0 = 1/10\)
With binary outcomes and \(m\) replicates per arm, per-task success proportions live on \(\{0, 1/m, \dots, 1\}\) and per-task mean differences on the lattice \(\{-1, \dots, -1/m, 0, 1/m, \dots, 1\}\). A constant task-level shift of \(1/10\) is off this lattice for \(m \in \{1, 2, 3\}\): the nearest attainable point is exactly \(1/10\) away in every case. (\(\delta_0 = 1/10\) is retired as an enforcing or powered-promotion target; this family studies lattice mechanics only, not any margin’s justification.)
| Replicates m | Per-task mean-difference lattice | Shift 1/10 attainable? | Min |gap| (exact) |
|---|---|---|---|
| 1 | -1, 0, +1 | no | 1/10 |
| 2 | -1/1, -1/2, 0/1, 1/2, 1/1 | no | 1/10 |
| 3 | -1/1, -2/3, -1/3, 0/1, 1/3, 2/3, 1/1 | no | 1/10 |
Lattice membership computed exactly by the committed code. A statement about lattice geometry only — it neither justifies nor criticizes any margin choice, and no margin is recommended here.
How each candidate construction handles the off-lattice shift: the grid-boundary-calibrated rule and the \(z\) rule address the mean, and a mean of \(1/10\) is attainable as an average across tasks even though no single task can realize it. The sign-flip construction’s implicit sharp model — a constant task-level shift of exactly \(\delta_0\) — has no on-lattice realization at \(\delta_0 = 1/10\) for \(m \le 3\): the sharp hypothesis it is exact for cannot even be instantiated on the replicate lattice. This is the lattice half of the do-not-sign finding carried in the held R1 ruling; the symmetry half is demonstrated numerically below.
Scope of this family in the tranche. Only lattice-membership mechanics are computed here. No operating characteristic in this tranche is computed on the \(m \in \{2, 3\}\) replicate-proportion lattices: every enumerated size and power in this note uses one replicate per arm (\(d_i \in \{-1, 0, +1\}\)). Lattice-valued operating characteristics are deferred to the complete package.
Assumption audit for conditional, sign, and McNemar constructions (required content 4)
Every conditional, sign-test, or McNemar-style construction considered by the package carries this audit. Consequences are quantified in the named scenario families.
| Assumption | Exact form | Scope | Plausibility | Consequence of failure |
|---|---|---|---|---|
| Common \(\theta\) / exchangeability | Discordant pairs share a common \(\theta = P(d_i = +1 \mid d_i \ne 0)\), or are exchangeable, so the conditional binomial reference applies | Suite-wide, unless a stratified variant is used | Doubtful suite-wide: four registered task classes plausibly differ in both \(\pi_d\) and \(\theta\) | The conditional binomial is the wrong reference; with \(\delta_0 > 0\) the induced \(\theta_0 = (\pi_d + \delta_0)/(2\pi_d)\) depends on the nuisance \(\pi_d\), so a composite-null conditional test inherits the same least-favorable-configuration problem the unconditional rule is calibrated for |
| Class/stratum homogeneity | Task-level parameters constant within (or across) the four registered classes; equal class weights | Suite-wide vs within-class | Within-class more plausible than suite-wide; not testable at 10 tasks per class with useful power | Suite-wide constructions average over class heterogeneity (heterogeneity family); stratified variants shrink per-stratum counts to \(\approx 10\), where the null-boundary family shows severe discreteness |
| Independence across tasks | \(d_1, \dots, d_N\) independent | All unconditional and conditional constructions here | Load-bearing and unproven. The serving stack can remember prior work through at least batch composition and prefix-cache state; a session-by-arm interaction survives pairing | Size distortion quantified in the cross-task-dependence family: at \(\rho = 0.10\) with all pairs in one session, the boundary-calibrated rule is clearly above nominal level; the \(\rho = 0.05\) point estimate is above nominal but its Monte Carlo uncertainty includes 0.05 |
| Effect homogeneity | Task effects constant (sharp constructions) or irrelevant beyond the mean (weak-null constructions) | Sharp constructions assume it; weak-null constructions must survive without it | Implausible: task classes are registered precisely because tasks differ | Sharp-null tests can misbehave under heterogeneity with the mean at the boundary (heterogeneity family); at \(\delta_0 = 1/10\) the sign-flip rule already exceeds nominal level at the homogeneous boundary |
| Missingness | Outcomes missing only per the registered attrition rule, ignorably with respect to arm | All constructions | Design-controllable but not automatic | Missingness breaks the equal-class-weight identity with the unweighted mean; arm-correlated attrition biases the paired-difference distribution in a way no reference distribution repairs — it must be reported, never silently dropped |
Concrete dependence mechanisms. Three named mechanisms make the independence threat material rather than hypothetical: (i) batch composition — co-batched work can influence numerics unless the serving stack is batch-invariant; (ii) prefix-cache state — cache hits make a task’s compute path depend on what preceded it; in vLLM v0.25.0 the CacheConfig.enable_prefix_caching field default is True, and effective behavior depends on model support, so possible history-dependent numerics are an empirical threat to observe, not a fact inferred from the default; (iii) session-by-arm interaction — the registered within-task AB/BA design cancels a session main effect and balances order/cache-warmth effects, but it does not cancel a session-by-arm interaction, which induces correlated per-task differences.
Required observations. The evidence contract must record, per run: batch_invariant, enable_prefix_caching, session_id, and task_pair_id. The indivisible co-location unit is the task pair, not the campaign: both arms for one task_pair_id must remain within one session_id — no task pair may cross a session boundary — while different pairs may be randomized across sessions. These four fields are what later allow the dependence assumptions above to be checked rather than assumed.
The three candidate constructions (required content 6)
All three constructions are evaluated as candidates for the coupled ruling; none is selected here. Each is assessed on its guarantee (with source), the basis under which the guarantee holds, applicability to binary outcomes with 2–3 replicates on a 40-task suite, the claim narrowing it implies, and its compatibility with the DP enumerator.
Construction 1 — randomization test aimed at the weak null
Studentized (prepivoted) sign-flip/permutation ideas aimed at a weak null.
- Guarantee: not established for this design. Wu and Ding (2021) and Zhao and Ding (2021) establish sharp-null exactness and weak-null asymptotics for randomized treatment-assignment experiments. Here both systems run on every task and only arm order is randomized. No theorem in this tranche maps that mechanism to the proposed task-level reference distribution.
- Basis: potentially a defended task-superpopulation sampling model, but an applicable theorem and its independence/heterogeneity conditions must be supplied before this is more than a research direction.
- Applicability at \(N = 40\), binary, 2–3 replicates: strained. The outcome lattice is coarse, studentization is unstable at small counts, and no finite-sample size guarantee applies at the boundary; the asymptotic regime the guarantee lives in is not obviously reached.
- Claim narrowing: none — it targets the weak null as stated.
- DP compatibility: computationally compatible. Under hypothetical sign flips the residuals \(\lvert d_i - \delta_0 \rvert\) take at most three values, so any statistic that depends on the flip vector only through per-group flip counts — including studentized statistics — can be enumerated deterministically over polynomially many count configurations rather than \(2^N\) flips: \(O(N^3)\) in general for \(\delta_0 > 0\) (at most \((n_+ + 1)(n_0 + 1)(n_- + 1)\) per-group flip-count triples), and \(O(N^2)\) at \(\delta_0 = 0\), where the zero-residual group drops out.
Construction 2 — grid-boundary calibration under a common i.i.d. model
The grid-boundary-calibrated rule evaluated throughout this note: reject when \(S \ge c\), with \(c\) calibrated so that the supremum of the exact rejection probability over a grid on the null boundary \(\Delta = \delta_0\), \(\pi_d \in \{\delta_0 \vee 0.01, \dots, 1\}\) (step \(1/100\)), is at most \(\alpha\). The least-favorable configuration (LFC) is found by grid supremum, never assumed — and, as the null-boundary family shows, it is usually not at maximum discordance.
- Guarantee: finite-sample calibrated (conservative) at level \(\alpha\) only on the enumerated nuisance grid, under the common i.i.d. paired-trinomial sampling model — by construction, since every boundary distribution is enumerated exactly. Source: derivation in this package (
lfc_calibrateinanalysis/_method_tranche/paired_trinomial.pyand this note). Interior null points on each grid line are covered by a derivation, not only by spot checks: for fixed \(\pi_d\), \(P(S \ge c)\) is nondecreasing in \(\Delta\) — increasing \(\Delta\) at fixed \(\pi_d\) moves cell probability mass from the \(-1\) cell to the \(+1\) cell with the \(0\) cell unchanged, so a direct coupling (one shared uniform draw per task) makes \(S\) pointwise no smaller — hence the boundary size bound extends to every \(\Delta \le \delta_0\) on that grid line (the argument is stated withlfc_calibrateand verified exactly in the committed tests; the computed near-boundary offsets are consistent with it). The honest caveat that remains: adequacy of the \(1/100\) grid in \(\pi_d\) for the continuous boundary is a design judgment the enumeration does not settle. - Basis: a common-distribution task-superpopulation model. The current derivation does not cover the intended equal-class-weighted stratified superpopulation or independent non-identically distributed tasks.
- Applicability: computationally direct for this outcome type and these suite sizes, but scientifically incomplete for the intended class-balanced design and continuous nuisance boundary. Conservatism on the examined grid is quantified below.
- Claim narrowing: none in the null. The examined rule is conservative at several grid points, but the non-level-matched comparisons below do not identify the cause or a general power cost.
- DP compatibility: total — calibration and every operating characteristic are DP enumerations.
Construction 3 — sharp-null / symmetry estimand
Deliberately narrow the scientific claim to one for which a sign-flip reference is distributionally justified: an explicitly assumed joint sign symmetry of the centered paired differences.
- Guarantee: finite-sample exact for the assumed joint sign-symmetry model. This is a distributional symmetry guarantee, not a Fisher randomization guarantee generated by this study’s arm-order assignment.
- Basis: the symmetry assumption can be stated under either basis, but note the assignment mechanism does not generate it (Section [Assignment unit and mechanism]); it must be assumed about the differences themselves.
- Applicability: mechanically fine at \(N = 40\); the reference distribution enumerates exactly.
- Claim narrowing: severe, and this is the construction’s defining cost. A rejection licenses only “task-level effects are not symmetrically null”; it says nothing about the suite mean exceeding any margin, and a shifted variant at \(\delta_0 = 1/10\) posits a sharp hypothesis with no on-lattice realization for \(m \le 3\) (lattice table above). The scientific claim becomes strictly smaller than the held R1 estimand language.
- DP compatibility: total (
signflip_pvalueis an integer-DP enumeration).
No construction in this tranche is ready to enforce the intended class-balanced design. Construction 1 lacks an applicable theorem; construction 2 is finite-sample calibrated only for a common i.i.d. model on a finite nuisance grid; construction 3 is coherent only under an assumed symmetry model that narrows the claim. The complete package must either close those gaps or recommend descriptive census reporting with no inferential gate. This tranche exposes the trade; it selects nothing.
Scenario family: null-boundary behavior
Type I behavior at and near the boundary of the composite null, where validity is most stressed. The grid-boundary-calibrated critical values and their boundary-grid suprema, for each \(N\) and each \(\delta_0 \in \{0, 1/10\}\):
| N | delta0 | Critical value c | Sup boundary size (grid) | Least-favorable pi_d (found) |
|---|---|---|---|---|
| 20 | 0/1 | 9 | 0.02408 | 24/25 |
| 20 | 1/10 | 11 | 0.02204 | 24/25 |
| 40 | 0/1 | 11 | 0.04512 | 97/100 |
| 40 | 1/10 | 15 | 0.04325 | 97/100 |
| 60 | 0/1 | 14 | 0.04623 | 1/1 |
| 60 | 1/10 | 19 | 0.04919 | 49/50 |
Calibration by deterministic enumeration over the boundary grid. The supremum is a computed property of the stated trinomial model on that grid; it is not an exactness claim for the weak null on the continuous boundary.
Findings:
- The least-favorable \(\pi_d\) is a lattice-parity effect near, but not always at, 1 (24/25, 97/100, 1, 49/50 across the six calibrations). Whether boundary mass lands usably at the critical value depends on the parity of attainable \(S\) values, so the supremum moves between interior grid points and the endpoint. The grid supremum was attained at an interior grid point, not at maximum discordance, in five of the six calibrations — assuming the maximum-discordance point is least favorable would have misplaced the supremum on this grid.
- Discreteness makes the calibrated rule under-shoot \(\alpha\), severely at \(N = 20\) (sup size \(\approx 0.024\), half the nominal level).
Boundary size of the three candidate rules, across discordance levels. Sign-flip boundary sizes are committed on their own grid for every \(N \in \{20, 40, 60\}\); the \(N = 40\) table is rendered here, and the \(N \in \{20, 60\}\) grids are summarized in the findings below.
Boundary \(\Delta = \delta_0 = 0/1\), \(N = 40\), \(\alpha = 1/20\):
| pi_d | Grid-boundary-calibrated rule | Normal-approx z rule | Shifted sign-flip rule |
|---|---|---|---|
| 1/10 | 0.00000 | 0.06046 | — |
| 3/10 | 0.00120 | 0.05311 | 0.02664 |
| 1/2 | 0.00916 | 0.05129 | 0.03264 |
| 7/10 | 0.02327 | 0.05552 | 0.03500 |
| 9/10 | 0.03986 | 0.05656 | — |
| 1/1 | 0.04035 | 0.04035 | 0.04035 |
Boundary \(\Delta = \delta_0 = 1/10\), \(N = 40\), \(\alpha = 1/20\):
| pi_d | Grid-boundary-calibrated rule | Normal-approx z rule | Shifted sign-flip rule |
|---|---|---|---|
| 1/10 | 0.00000 | 0.01550 | — |
| 3/10 | 0.00106 | 0.05488 | 0.04649 |
| 1/2 | 0.00816 | 0.05376 | 0.05232 |
| 7/10 | 0.02165 | 0.05826 | 0.05231 |
| 9/10 | 0.03804 | 0.05723 | — |
| 1/1 | 0.03859 | 0.07506 | 0.03859 |
All entries are exact enumerations (rationals in the committed artifact) under the stated i.i.d. trinomial boundary model. “Exceeds 0.05” is a computed property under that model, not a statement about any real system.
Findings, all at \(N = 40\):
- The calibrated rule is very conservative at moderate discordance: size 0.00120 at \(\pi_d = 0.3\) (\(\delta_0 = 0\)) against a nominal 0.05. This quantifies the rule’s local conservatism but does not, by itself, identify which part of the construction caused it.
- The normal-approximation \(z\) rule over-rejects at the boundary: 0.06046 at \(\pi_d = 0.1\) and 0.05311 at \(\pi_d = 0.3\) (\(\delta_0 = 0\)); at the shifted boundary it reaches 0.07506 at \(\pi_d = 1\). Its guarantee is asymptotic, and \(N = 40\) with low discordance is not the asymptotic regime.
- Enumeration is not exactness, demonstrated: the shifted sign-flip rule — whose reference distribution is enumerated deterministically, with zero simulation error — exceeds the nominal level on the composite boundary at \(\delta_0 = 1/10\): size 0.05232 at \(\pi_d = 1/2\) and 0.05231 at \(\pi_d = 7/10\). This occurs at the homogeneous boundary: no heterogeneity is needed, because \(d_i - \delta_0\) is simply not symmetric under the trinomial boundary model. The exceedance is not an \(N = 40\) artifact: the committed grids show maxima 0.05652 at \(N = 20\) (\(\pi_d = 7/10\)) and 0.05073 at \(N = 60\) (\(\pi_d = 7/10\)). At \(\delta_0 = 0\), where the boundary model is symmetric, the same enumeration stays below nominal at every committed suite size (maxima 0.02962 / 0.04035 / 0.04623 at \(N = 20/40/60\)) — exactly the sharp-versus-weak distinction in operation.
- Interior of the null: at every computed offset below the boundary (\(\Delta = \delta_0 - 1/20\) and \(\delta_0 - 1/10\)), both the calibrated rule and the \(z\) rule reject less than at the boundary in every cell (committed in the artifact’s
near_boundary_rejectionrecords) — for the calibrated rule this is also implied by the fixed-\(\pi_d\) monotonicity derivation stated with construction 2.
Scenario family: heterogeneous effects
Task-level effects vary across tasks while the suite mean sits exactly at the null value — the regime where sharp-null tests are known to be able to over-reject weak nulls. Five committed 40-task profiles (blocks give \((\text{tasks} \times (p_+, p_-))\)):
| Profile | delta0 | Blocks (tasks × (p+, p-)) | Grid-boundary-calibrated | Shifted sign-flip | Normal-approx z |
|---|---|---|---|---|---|
| hom-boundary-0.10 | 1/10 | 40×(1/5, 1/10) | 0.00106 | 0.04649 | 0.05488 |
| het-two-block-0.10 | 1/10 | 20×(9/20, 1/20) + 20×(1/20, 1/4) | 0.00087 | 0.03192 | 0.03331 |
| het-extreme-0.10 | 1/10 | 24×(1/2, 0/1) + 16×(0/1, 1/2) | 0.00034 | 0.01152 | 0.01199 |
| hom-boundary-zero | 0/1 | 40×(3/20, 3/20) | 0.00120 | 0.02664 | 0.05311 |
| het-two-block-zero | 0/1 | 20×(7/20, 1/20) + 20×(1/20, 7/20) | 0.00134 | 0.01723 | 0.03097 |
Rejection probabilities at the suite-mean null value, computed by exact enumeration under each stated heterogeneous model (\(N = 40\), \(\alpha = 1/20\)). Computed operating characteristics of candidate rules — no validity claim for any null is implied.
Findings, reported as computed:
- No examined profile makes the grid-boundary-calibrated rule anti-conservative (maximum rejection 0.00134 across all five profiles).
- The \(z\) rule mildly over-rejects at the homogeneous \(\delta_0 = 1/10\) boundary (0.05488) — but in these profiles heterogeneity makes all three rules more conservative, not less: the \(z\) rule falls to 0.03331 (two-block) and 0.01199 (extreme); the sign-flip falls from 0.04649 (homogeneous) to 0.03192 and 0.01152; at \(\delta_0 = 0\) the \(z\) rule falls from 0.05311 to 0.03097.
- Honest limitation: these block profiles do not exhaust the space of heterogeneous configurations, and the absence of anti-conservatism here is a property of the examined profiles, not a validity result. The demonstrated weak-null failure of the sign-flip rule in this model family is at the homogeneous shifted boundary (previous family), where symmetry — not homogeneity — is what breaks.
Scenario family: margin feasibility, and the falsifiable cross-check
The feasibility frontier — power as a function of true effect \(\Delta\), discordance \(\pi_d\), and \(N\) — is committed for every construction evaluated in this tranche: the grid-boundary-calibrated rule in the table below, and the shifted sign-flip and normal-approximation \(z\) rules in the comparator tables that follow. The bounded-mean methods are the one construction family whose frontier is deferred — together with the rest of required content 7 — to the complete package. Both frontiers cover the superiority null at zero and the retired \(1/10\) margin. (\(\delta_0 = 1/10\) appears because its feasibility must be studied; it remains retired as an enforcing or powered-promotion target, and nothing here re-installs it.)
| True Δ | pi_d | N=20, δ0=0/1 (c=9) | N=20, δ0=1/10 (c=11) | N=40, δ0=0/1 (c=11) | N=40, δ0=1/10 (c=15) | N=60, δ0=0/1 (c=14) | N=60, δ0=1/10 (c=19) |
|---|---|---|---|---|---|---|---|
| 3/20 | 1/5 | 0.0031 | 0.0001 | 0.0488 | 0.0013 | 0.0857 | 0.0026 |
| 3/20 | 3/10 | 0.0100 | 0.0008 | 0.0881 | 0.0056 | 0.1346 | 0.0102 |
| 3/20 | 2/5 | 0.0214 | 0.0027 | 0.1228 | 0.0137 | 0.1720 | 0.0224 |
| 3/20 | 1/2 | 0.0354 | 0.0063 | 0.1515 | 0.0246 | 0.2006 | 0.0369 |
| 1/5 | 1/5 | 0.0100 | 0.0006 | 0.1608 | 0.0079 | 0.3056 | 0.0221 |
| 1/5 | 3/10 | 0.0251 | 0.0025 | 0.2174 | 0.0227 | 0.3498 | 0.0505 |
| 1/5 | 2/5 | 0.0451 | 0.0068 | 0.2552 | 0.0424 | 0.3741 | 0.0803 |
| 1/5 | 1/2 | 0.0667 | 0.0139 | 0.2817 | 0.0633 | 0.3897 | 0.1073 |
| 1/4 | 3/10 | 0.0560 | 0.0069 | 0.4303 | 0.0734 | 0.6513 | 0.1761 |
| 1/4 | 2/5 | 0.0875 | 0.0159 | 0.4467 | 0.1096 | 0.6314 | 0.2183 |
| 1/4 | 1/2 | 0.1170 | 0.0285 | 0.4558 | 0.1406 | 0.6176 | 0.2485 |
| 3/10 | 3/10 | 0.1133 | 0.0171 | 0.6913 | 0.1926 | 0.9000 | 0.4368 |
| 3/10 | 2/5 | 0.1568 | 0.0342 | 0.6654 | 0.2386 | 0.8520 | 0.4541 |
| 3/10 | 1/2 | 0.1920 | 0.0548 | 0.6475 | 0.2705 | 0.8183 | 0.4630 |
| 7/20 | 2/5 | 0.2609 | 0.0683 | 0.8538 | 0.4391 | 0.9678 | 0.7296 |
| 7/20 | 1/2 | 0.2960 | 0.0990 | 0.8168 | 0.4528 | 0.9414 | 0.7024 |
| 2/5 | 2/5 | 0.4044 | 0.1275 | 0.9648 | 0.6826 | 0.9979 | 0.9281 |
| 2/5 | 1/2 | 0.4290 | 0.1686 | 0.9310 | 0.6609 | 0.9893 | 0.8879 |
Exact enumerated power of the grid-boundary-calibrated rule under the stated i.i.d. trinomial model. “Exact” describes the arithmetic; whether any rule should gate anything is the held ruling’s question, not this table’s.
Findings:
- With the retired \(1/10\) margin, power is poor at the examined grid points. At \(N = 40\) (\(c = 15\)): 0.1926 at \((\Delta, \pi_d) = (0.30, 0.30)\) and only 0.6826 even at \((0.40, 0.40)\). At \(N = 60\) (\(c = 19\)): 0.4368 at \((0.30, 0.30)\). At \(N = 20\): 0.0171 at \((0.30, 0.30)\) — essentially no power. This independently reproduces, from this package’s own committed code, the decision-input finding that a confirmatory \(0.10\) margin is infeasible at \(N = 40\) under the examined paired-binary model.
- The superiority-at-zero columns are analyzed in the next family.
Comparator frontiers: shifted sign-flip and normal-approximation rules
The same frontier, committed for the other two constructions evaluated in this tranche, on the same alternatives and the same \((N, \delta_0)\) grid:
Shifted sign-flip rule:
| True Δ | pi_d | N=20, δ0=0/1 (c=9) | N=20, δ0=1/10 (c=11) | N=40, δ0=0/1 (c=11) | N=40, δ0=1/10 (c=15) | N=60, δ0=0/1 (c=14) | N=60, δ0=1/10 (c=19) |
|---|---|---|---|---|---|---|---|
| 3/20 | 1/5 | 0.1831 | 0.0847 | 0.5763 | 0.1373 | 0.8054 | 0.1972 |
| 3/20 | 3/10 | 0.1753 | 0.0990 | 0.4317 | 0.1366 | 0.6284 | 0.1733 |
| 3/20 | 2/5 | 0.1570 | 0.0968 | 0.3606 | 0.1300 | 0.5110 | 0.1501 |
| 3/20 | 1/2 | 0.1462 | 0.0926 | 0.3087 | 0.1202 | 0.4400 | 0.1386 |
| 1/5 | 1/5 | 0.3704 | 0.1958 | 0.9241 | 0.4069 | 0.9961 | 0.5514 |
| 1/5 | 3/10 | 0.3139 | 0.1957 | 0.6913 | 0.3181 | 0.8800 | 0.4373 |
| 1/5 | 2/5 | 0.2625 | 0.1748 | 0.5740 | 0.2739 | 0.7583 | 0.3492 |
| 1/5 | 1/2 | 0.2330 | 0.1577 | 0.4888 | 0.2382 | 0.6662 | 0.3017 |
| 1/4 | 3/10 | 0.5124 | 0.3530 | 0.9045 | 0.5920 | 0.9863 | 0.7710 |
| 1/4 | 2/5 | 0.4069 | 0.2928 | 0.7812 | 0.4843 | 0.9244 | 0.6215 |
| 1/4 | 1/2 | 0.3480 | 0.2514 | 0.6801 | 0.4092 | 0.8508 | 0.5291 |
| 3/10 | 3/10 | 0.7625 | 0.5836 | 0.9974 | 0.8890 | 1.0000 | 0.9705 |
| 3/10 | 2/5 | 0.5851 | 0.4557 | 0.9258 | 0.7207 | 0.9886 | 0.8622 |
| 3/10 | 1/2 | 0.4879 | 0.3763 | 0.8416 | 0.6122 | 0.9550 | 0.7592 |
| 7/20 | 2/5 | 0.7781 | 0.6577 | 0.9886 | 0.9078 | 0.9996 | 0.9792 |
| 7/20 | 1/2 | 0.6426 | 0.5292 | 0.9442 | 0.8020 | 0.9923 | 0.9184 |
| 2/5 | 2/5 | 0.9490 | 0.8744 | 1.0000 | 0.9939 | 1.0000 | 0.9997 |
| 2/5 | 1/2 | 0.7944 | 0.6979 | 0.9886 | 0.9324 | 0.9995 | 0.9856 |
Normal-approximation z rule:
| True Δ | pi_d | N=20, δ0=0/1 (c=9) | N=20, δ0=1/10 (c=11) | N=40, δ0=0/1 (c=11) | N=40, δ0=1/10 (c=15) | N=60, δ0=0/1 (c=14) | N=60, δ0=1/10 (c=19) |
|---|---|---|---|---|---|---|---|
| 3/20 | 1/5 | 0.5141 | 0.0848 | 0.7879 | 0.1638 | 0.8986 | 0.2261 |
| 3/20 | 3/10 | 0.3955 | 0.1021 | 0.5645 | 0.1570 | 0.7078 | 0.1757 |
| 3/20 | 2/5 | 0.3190 | 0.1098 | 0.4472 | 0.1331 | 0.5994 | 0.1618 |
| 3/20 | 1/2 | 0.2693 | 0.1150 | 0.3881 | 0.1228 | 0.5115 | 0.1511 |
| 1/5 | 1/5 | 0.7939 | 0.1958 | 0.9921 | 0.4069 | 0.9998 | 0.6766 |
| 1/5 | 3/10 | 0.5874 | 0.1992 | 0.8084 | 0.3572 | 0.9199 | 0.4406 |
| 1/5 | 2/5 | 0.4663 | 0.1931 | 0.6616 | 0.2796 | 0.8232 | 0.3663 |
| 1/5 | 1/2 | 0.3885 | 0.1896 | 0.5745 | 0.2418 | 0.7299 | 0.3214 |
| 1/4 | 3/10 | 0.7924 | 0.3551 | 0.9614 | 0.6426 | 0.9933 | 0.7772 |
| 1/4 | 2/5 | 0.6324 | 0.3140 | 0.8456 | 0.4937 | 0.9519 | 0.6362 |
| 1/4 | 1/2 | 0.5265 | 0.2927 | 0.7531 | 0.4131 | 0.8903 | 0.5516 |
| 3/10 | 3/10 | 0.9645 | 0.5836 | 0.9999 | 0.8890 | 1.0000 | 0.9861 |
| 3/10 | 2/5 | 0.7954 | 0.4740 | 0.9564 | 0.7336 | 0.9938 | 0.8678 |
| 3/10 | 1/2 | 0.6712 | 0.4236 | 0.8884 | 0.6152 | 0.9708 | 0.7768 |
| 7/20 | 2/5 | 0.9260 | 0.6663 | 0.9954 | 0.9207 | 0.9998 | 0.9796 |
| 7/20 | 1/2 | 0.8060 | 0.5751 | 0.9649 | 0.8038 | 0.9958 | 0.9269 |
| 2/5 | 2/5 | 0.9964 | 0.8744 | 1.0000 | 0.9939 | 1.0000 | 0.9999 |
| 2/5 | 1/2 | 0.9127 | 0.7321 | 0.9939 | 0.9334 | 0.9998 | 0.9876 |
Exact enumerated rejection probabilities under the same stated i.i.d. trinomial alternatives — computed operating characteristics of candidate rules. Not a like-for-like ranking of level-\(\alpha\) rules: on the composite boundary at \(\delta_0 = 1/10\) the shifted sign-flip rule exceeds the nominal level (committed grid maxima 0.05652 / 0.05232 / 0.05073 at \(N = 20/40/60\) — null-boundary family), and the \(z\) rule exceeds it too (up to 0.07506 at \(N = 40\), \(\pi_d = 1\); 0.05488 already at \(\pi_d = 3/10\)). Their power must be read next to those boundary sizes, not against the calibrated rule’s.
Findings:
- A large, non-level-matched power gap. At \(N = 40\), \(\delta_0 = 1/10\), \((\Delta, \pi_d) = (0.30, 0.30)\): the grid-boundary-calibrated rule has power 0.1926 versus 0.8890 for both comparators. At this alternative’s discordance the sign-flip boundary size is 0.04649, the \(z\) rule’s is 0.05488, and the calibrated rule’s is 0.00120. Because the candidates are not level-matched over the same null, these numbers do not identify the cause of the power gap. They are consistent with conservatism from grid-wide protection but cannot attribute the gap to it. (At alternatives with \(\pi_d = \Delta\), \(p_- = 0\) and both comparator rules reduce to thresholds on the count of positive differences, which is why their powers can coincide exactly.)
- The examined-model feasibility direction is unchanged. Even rules with no finite-sample guarantee for the composite null leave the retired \(0.10\) margin under-powered at \(N = 20\) (0.5836 at \((0.30, 0.30)\) for both comparators), and at \(N = 40\) the high-power entries in this family belong only to rules whose committed boundary size exceeds the nominal level somewhere on the composite boundary. This is not a universal feasibility result for stratified, bounded-mean, or multi-replicate methods.
The falsifiable cross-check (reported, never tuned)
The commission names a comparison value: power 0.193 for a construction described there as an exact LFC-calibrated test of \(H_0\!: \Delta \le 0.10\) at \(N = 40\), one replicate, \(\alpha = 0.05\) one-sided, true \(\Delta = 0.30\), \(\pi_d = 0.30\), with the comparison construction described as calibrated at the maximum-discordance boundary point. This tranche derived the operating point independently — the decision-input artifacts were not opened, and no code was adjusted toward the target:
| Quantity | Value |
|---|---|
| Design | N=40, δ0=1/10, α=1/20, true Δ=3/10, π_d=3/10 |
| Critical value (grid-supremum calibration) | 15 |
| Critical value (max-discordance-only recomputation, committed) | 15 |
| Boundary size at π_d = 1 (max-discordance-only recomputation) | 0.03859 |
| Independently derived power (float of the exact rational) | 0.19255175473517602 |
| Comparison value (from the cited decision inputs) | 0.193 |
| Absolute discrepancy | 0.000448 |
The committed artifact records the derived power as an exact rational; the enumeration is deterministic. Deterministic derivation of the number is a computation claim only.
Two differences from the comparison construction, reported as found:
- Calibration route. This package calibrates by grid supremum over the whole null boundary and finds the supremum at \(\pi_d = 97/100\) — not assumed at maximum discordance. Recomputing the commission-described maximum-discordance-only calibration from this package’s own committed code (
lfc_calibraterestricted to \(\pi_d = 1\); the artifact’smax_discordance_only_calibrationrecord) also yields the critical value \(c = 15\), with boundary size 0.03859 at \(\pi_d = 1\). The two routes agree on \(c = 15\) at this design point; nothing is asserted about the contents of the unconsulted decision inputs. - The value. The derived power is 0.19255 (exact rational in the artifact), versus the cited 0.193: an absolute discrepancy of \(4.5 \times 10^{-4}\). This is consistent with 0.193 being a three-decimal rounding of the same operating point (0.19255 rounds to 0.193), but it is reported here as a disagreement of \(4.5 \times 10^{-4}\), not smoothed over and not tuned away.
The decision-input artifacts are cited by name and SHA-256 only, per the public boundary; their contents are not reproduced here and were not consulted in deriving the numbers above:
science-layer-consult-r1-r5.md— SHA-25660409acec91e6aa02d4d39f8cfd4bc1f43b17031b1752da24f2f2e67f8aa896asuite_power.py— SHA-256bb534d7d908f64e01e45d2a039806fba22b54c994af9beb8713395f9ec53ff5a
Scenario family: superiority at zero, and the MPIB rule (required content 8)
Per the held R4 amendment requirement, this scenario evaluates a candidate separation between a superiority null at zero (\(H_0\!: \Delta \le 0\)) and a separately stated minimum practically important benefit (MPIB) rule. Statistical evidence of superiority and the judgment of practical importance are different questions; folding an unjustified margin into the test’s null conflates them. This is an illustrative candidate joint rule, not an accepted gate, and the package recommends neither its test nor any MPIB value.
The two illustrative components, stated separately:
- Candidate superiority component: under the common i.i.d. model and finite nuisance grid, reject \(H_0\!: \Delta \le 0\) at \(N = 40\) when \(S \ge 11\) (grid-supremum boundary size 0.04512, attained at \(\pi_d = 97/100\)).
- Candidate practical-benefit component: additionally require the point estimate \(\bar d = S/40\) to be at least a candidate MPIB. The MPIB grid below states consequences only; no value or component is recommended.
Power of the candidate superiority component alone (from the \(N=40\), \(\delta_0 = 0\) column of the frontier table): 0.6913 at \((\Delta, \pi_d) = (0.30, 0.30)\), 0.2174 at \((0.20, 0.30)\), 0.9648 at \((0.40, 0.40)\); at \(N = 60\) (\(c = 14\)) the rule reaches 0.9000 at \((0.30, 0.30)\). Boundary sizes across \(\pi_d\) appear in the null-boundary family.
Joint behavior of test-plus-MPIB-rule at \(N = 40\), \(\pi_d = 0.3\), across the candidate MPIB grid:
| True Δ | Candidate MPIB | Point-estimate threshold (S ≥) | Joint threshold (S ≥) | P(joint pass) | P(point-estimate component alone) |
|---|---|---|---|---|---|
| 0/1 | 1/20 | 2 | 11 | 0.0012 | 0.3318 |
| 0/1 | 1/10 | 4 | 11 | 0.0012 | 0.1551 |
| 0/1 | 3/20 | 6 | 11 | 0.0012 | 0.0555 |
| 0/1 | 1/5 | 8 | 11 | 0.0012 | 0.0150 |
| 0/1 | 1/4 | 10 | 11 | 0.0012 | 0.0030 |
| 1/10 | 1/20 | 2 | 11 | 0.0281 | 0.7692 |
| 1/10 | 1/10 | 4 | 11 | 0.0281 | 0.5578 |
| 1/10 | 3/20 | 6 | 11 | 0.0281 | 0.3285 |
| 1/10 | 1/5 | 8 | 11 | 0.0281 | 0.1511 |
| 1/10 | 1/4 | 10 | 11 | 0.0281 | 0.0529 |
| 1/5 | 1/20 | 2 | 11 | 0.2174 | 0.9797 |
| 1/5 | 1/10 | 4 | 11 | 0.2174 | 0.9206 |
| 1/5 | 3/20 | 6 | 11 | 0.2174 | 0.7810 |
| 1/5 | 1/5 | 8 | 11 | 0.2174 | 0.5589 |
| 1/5 | 1/4 | 10 | 11 | 0.2174 | 0.3182 |
| 3/10 | 1/20 | 2 | 11 | 0.6913 | 1.0000 |
| 3/10 | 1/10 | 4 | 11 | 0.6913 | 0.9994 |
| 3/10 | 3/20 | 6 | 11 | 0.6913 | 0.9914 |
| 3/10 | 1/5 | 8 | 11 | 0.6913 | 0.9447 |
| 3/10 | 1/4 | 10 | 11 | 0.6913 | 0.8041 |
Exact enumerated pass probabilities under the stated trinomial model (\(N = 40\), \(\pi_d = 3/10\), superiority critical value 11). The grid states consequences only; the MPIB value is an operator input and none is recommended.
Findings:
- The candidate significance component is the binding threshold for every examined MPIB up to 0.25 at this design point: every MPIB in \(\{0.05, \dots, 0.25\}\) gives a point-estimate threshold \(\lceil 40 \cdot \text{MPIB} \rceil \in \{2, \dots, 10\}\), all below the superiority critical value 11 — so the joint pass probability is constant across the whole grid and equals the reject-at-zero power (0.0012 at \(\Delta = 0\); 0.0281 at 0.1; 0.2174 at 0.2; 0.6913 at 0.3).
- The point-estimate component alone is not a substitute for a calibrated test: used without the candidate superiority component it false-passes at \(\Delta = 0\) with probability 0.3318 / 0.1551 / 0.0555 / 0.0150 / 0.0030 for MPIB 0.05 / 0.10 / 0.15 / 0.20 / 0.25. The scenario separates the two questions so a later ruling can evaluate them without conflation; it does not install either component.
Replicate and nondeterminism semantics (required content 9), with the floor-propagation family
The following candidate replicate semantics are decision input to held R2; they are not an accepted policy. Every scenario in this note is constructed consistently with them:
- Greedy/deterministic intent. Repeats are an apparatus-repeatability diagnostic, not extra inferential task \(N\). Greedy serving is not assumed bitwise deterministic: the candidate policy would require batch-invariance evidence where available (recorded in
batch_invariant), or an E1-measured-and-bounded nondeterminism floor \(\varepsilon\) (E1, specified below, is the planned study measuring how much the apparatus itself varies between identical runs). The apparatus alarm threshold comes from that measured floor, never from zero. - Stochastic intent. Under the candidate policy, seeds would be randomly drawn once and then frozen; the frozen seed schedule is part of the tested system, and task-level outcomes aggregate over it.
- In both candidate modes, replicates never inflate inferential task \(N\). Batch composition, arm order, process restarts, and pod instantiation are design factors, not unnamed replicates. In every construction above, the replicate count \(m\) changes the per-task outcome lattice and the meaning of per-task success — never the task count \(N\) that enters the reference distribution.
- Interaction with the candidate constructions. All three constructions consume per-task aggregated outcomes, so all three inherit these semantics identically: under greedy intent their inputs are single measured outcomes (with repeats policing the apparatus), and under stochastic intent their inputs are aggregates over the frozen schedule. A nonzero floor changes the distribution of those inputs — quantified next — and the lattice family already showed how \(m\) changes what per-task values are attainable.
Floor propagation (synthetic floor values)
A measured floor \(\varepsilon\) (each arm’s recorded binary outcome independently flipped with probability \(\varepsilon\), the stated synthetic channel model) propagates three ways: it attenuates the mean difference — linearly, \(\Delta' = (1 - 2\varepsilon)\Delta\) under this channel — it perturbs the effective outcome cells, and it sets the apparatus alarm threshold. Base synthetic cells: \((p_{11}, p_{10}, p_{01}, p_{00}) = (2/5, 3/10, 0, 3/10)\), i.e. \(\Delta = 0.3\), \(\pi_d = 0.3\).
| Floor ε | Linear attenuation (1−2ε)Δ of Δ=0.3 | Cell-transform implied Δ′ | Implied π_d′ | Superiority power at c=11 | Repeat-disagreement prob. | Floor-derived alarm k (40/80/120 pairs) | From-zero alarm k (40/80/120 pairs) |
|---|---|---|---|---|---|---|---|
| 1/100 | 0.2940 | 0.2940 | 0.3079 | 0.6582 | 0.0198 | 3 / 5 / 6 | 1 / 1 / 1 |
| 1/50 | 0.2880 | 0.2880 | 0.3157 | 0.6259 | 0.0392 | 5 / 7 / 9 | 1 / 1 / 1 |
| 1/20 | 0.2700 | 0.2700 | 0.3380 | 0.5339 | 0.0950 | 8 / 13 / 18 | 1 / 1 / 1 |
Exact rational arithmetic under the stated synthetic floor model; the alarm threshold is the smallest disagreement count whose false-alarm probability under the floor is ≤ 1/20 (exact binomial tail). Computed model properties — whether the floor model fits a given apparatus is an empirical question for E1, and no threshold is recommended here.
Findings:
- Attenuation is linear in the mean; discordance inflates. Under the stated per-arm flip channel the mean difference attenuates linearly: \(\Delta' = (1 - 2\varepsilon)\Delta\) = 0.2940/0.2880/0.2700 at \(\varepsilon = 0.01/0.02/0.05\) — exactly the cell-transform implied \(\Delta'\) column (the two committed columns agree identically; the identity is verified exactly in the committed tests). The quadratic factor \((1 - 2\varepsilon)^2\) = 0.9604/0.9216/0.8100 (committed as
covariance_attenuation_factor) is the attenuation factor a covariance-type quantity would suffer under the same channel; it is not the attenuation of \(\Delta\), and no stated model in this package attenuates \(\Delta\) quadratically. Meanwhile discordance inflates to \(\pi_d' = 0.3079/0.3157/0.3380\) — the floor manufactures spurious discordance, changing the effective cells every construction consumes. - Power erosion. Superiority power at \(c = 11\) falls from 0.6913 (no floor) to 0.6582/0.6259/0.5339.
- Alarm thresholds must come from the floor. With per-pair repeat disagreement probability \(2\varepsilon(1-\varepsilon)\) = 0.0198/0.0392/0.0950, the floor-derived alarm thresholds at \(\alpha = 1/20\) for 40/80/120 repeat pairs are 3/5/6 (\(\varepsilon=0.01\)), 5/7/9 (\(\varepsilon=0.02\)), and 8/13/18 (\(\varepsilon=0.05\)). The from-zero policy’s threshold is 1 at every \(n\) — any single disagreement alarms — which under any nonzero floor mistakes routine floor-level disagreement for an apparatus fault. This is the contrast the held R2 amendments rule out: thresholds are derived from the measured floor, never from zero.
Scenario family: cross-task dependence and session layout
Cross-task independence is load-bearing for every construction above and is unproven. This family quantifies the cost of losing it, under a stated synthetic mixture model: with probability \(\lambda\) a pair’s difference copies a session-level sign (a session-by-arm interaction that pairing does not cancel), otherwise it is an independent trinomial draw at the \(\Delta = 0\) boundary with \(\pi_d = 0.3\); \(\lambda\) is solved in closed form so the within-session correlation equals a target \(\rho\). The model preserves \(\Delta = 0\) exactly and (documented property) inflates marginal discordance above \(\pi_d\). The design-effect identity \(\text{DEFF} = 1 + (m - 1)\rho\) for \(m\) pairs per session:
| rho | m = 10 pairs/session | m = 20 pairs/session | m = 40 pairs/session |
|---|---|---|---|
| 0.01 | 1.09 | 1.19 | 1.39 |
| 0.02 | 1.18 | 1.38 | 1.78 |
| 0.05 | 1.45 | 1.95 | 2.95 |
| 0.10 | 1.90 | 2.90 | 4.90 |
Closed-form variance-inflation identity of the stated exchangeable dependence model — a model property, not a test-validity claim.
Rejection rate at the \(\Delta = 0\) boundary (nominal \(\alpha = 0.05\)) of two candidate rules under this model, by \(\rho\) and session layout — 5000 seeded simulations per cell, Monte Carlo standard error in parentheses. “Grid-boundary-calibrated (\(c = 11\))” names the critical value computed by deterministic enumeration under cross-task independence; simulating it under dependence measures how far that independence-derived size travels, and claims no validity for the rule under dependence.
Grid-boundary-calibrated rule, c = 11 (critical value derived under independence):
| rho | 1 session × 40 pairs | 2 sessions × 20 pairs | 4 sessions × 10 pairs |
|---|---|---|---|
| 0.01 | 0.0074 (±0.0012) | 0.0034 (±0.0008) | 0.0030 (±0.0008) |
| 0.02 | 0.0110 (±0.0015) | 0.0080 (±0.0013) | 0.0040 (±0.0009) |
| 0.05 | 0.0550 (±0.0032) | 0.0302 (±0.0024) | 0.0120 (±0.0015) |
| 0.10 | 0.1562 (±0.0051) | 0.0806 (±0.0038) | 0.0368 (±0.0027) |
Normal-approximation z rule:
| rho | 1 session × 40 pairs | 2 sessions × 20 pairs | 4 sessions × 10 pairs |
|---|---|---|---|
| 0.01 | 0.0820 (±0.0039) | 0.0662 (±0.0035) | 0.0570 (±0.0033) |
| 0.02 | 0.1146 (±0.0045) | 0.0834 (±0.0039) | 0.0710 (±0.0036) |
| 0.05 | 0.2030 (±0.0057) * | 0.1196 (±0.0046) | 0.0846 (±0.0039) |
| 0.10 | 0.3410 (±0.0067) | 0.1994 (±0.0057) | 0.1260 (±0.0047) |
* independent verification cell (see below).
Seeded Monte Carlo under the stated synthetic mixture model; rejection rates are computed operating characteristics of candidate rules under that model and say nothing about any real serving stack.
Findings:
- Repeatability is not independence. A perfectly repeatable apparatus can still correlate task outcomes within a session; this family is why E1 must be an independence study and not only a repeatability study.
- The one-session layout is the dangerous one. For the \(z\) rule at nominal 0.05, one session of 40 pairs gives rejection rates 0.082/0.115/0.203/0.341 at \(\rho = 0.01/0.02/0.05/0.10\); splitting into four sessions of 10 pairs reduces this to 0.057/0.071/0.085/0.126.
- Dependence can break even the boundary-calibrated rule. The grid-boundary-calibrated rule’s computed size under independence at this \(\pi_d = 0.3\) boundary point is 0.0012; under the mixture model in one session its point estimate is 0.055 (MC SE 0.0032) at \(\rho = 0.05\) and 0.156 (MC SE 0.0051) at \(\rho = 0.10\). The \(\rho = 0.05\) Monte Carlo uncertainty includes 0.05, so that cell alone does not establish over-rejection; the \(\rho = 0.10\) cell is clearly above nominal. The rule’s calibration is conditional on independence, and this synthetic family shows the consequence of violating it.
- Independent verification cell. The commission carried a synthetic advisory diagnostic for comparison under a different dependence model. This package’s independently derived starred cell gives 0.203 ± 0.0057 (MC SE) at \(\rho = 0.05\) — higher than that external advisory approximation. Differences in the two synthetic dependence models (this one, for instance, inflates marginal discordance as documented) can plausibly account for the gap; per the falsifiable-cross-check discipline the disagreement is reported, not tuned to match.
What this tranche defers
- Full formal estimand treatment (required content 2): the candidate estimand and null statements above are provisional and compact; the complete package must settle their inferential basis, class weighting, missingness algebra, and licensed claim.
- Finite-sample bounded-mean methods (required content 7): the betting/confidence-sequence constructions of Waudby-Smith and Ramdas will be evaluated on the same dimensions as the three constructions above, including their cells of the feasibility frontier. Nothing in this tranche presumes their result in either direction; if they change the feasibility picture, the complete package will show exactly where and by how much, from committed code.
- Lattice-valued operating characteristics: sizes and powers computed on the \(m \in \{2, 3\}\) replicate-proportion lattices. This tranche demonstrates the lattice mechanics only; every operating characteristic here uses one replicate per arm.
- Figures and full report treatment for all scenario families.
Limitations, stated plainly
- Everything here is computed under stated synthetic models — the i.i.d./heterogeneous paired-trinomial families, the flip-channel floor model, and the session-mixture dependence model. No number describes any real system, campaign, or live run, and none of the models has been validated against a real apparatus (that is E1’s job, later and separately gated).
- LFC calibration searches a \(1/100\) grid on the null boundary; grid adequacy in \(\pi_d\) for the continuous boundary is a design judgment. (Interior null behavior on each grid line follows from the exact fixed-\(\pi_d\) monotonicity coupling stated with construction 2 and verified in the committed tests.)
- The dependence family’s rates carry Monte Carlo error (reported per cell); the enumerated families carry none.
- Small-\(N\) discreteness dominates several results (e.g., sup size 0.024 at \(N = 20\)); conclusions do not transfer across \(N\) without recomputation.
- The heterogeneity profiles are five committed configurations, not a search over the space of heterogeneous nulls.
- The i.i.d. calculations do not establish validity for equal class weights, class-stratified sampling, or independent non-identically distributed tasks.
- The randomized-experiment theorems cited for construction 1 do not map to this arm-order-only design without an additional theorem; construction 3 relies on assumed joint sign symmetry rather than assignment-generated Fisher exactness.
- The reported candidate powers are not a level-matched comparison and do not identify why their power differs.
- R1, R2, and R4 remain held. This tranche informs the coupled ruling and selects nothing; its recommendations are labeled as such and bind no one.
References
- Wu, J. and Ding, P. (2021). Randomization Tests for Weak Null Hypotheses in Randomized Experiments. arXiv:1809.07419.
- Zhao, A. and Ding, P. (2021). Covariate-adjusted Fisher randomization tests for the average treatment effect. arXiv:2010.14555.
- Waudby-Smith, I. and Ramdas, A. Estimating means of bounded random variables by betting. JRSS-B. doi:10.1093/jrsssb/qkad009 (evaluation deferred to the complete package).
- vLLM v0.25.0
CacheConfigreference —enable_prefix_cachingfield defaultTrue; effective behavior depends on model support and must be observed, not inferred from the default. - Decision-input artifacts, cited by name and SHA-256 only (contents neither quoted nor consulted for the derivations here):
science-layer-consult-r1-r5.md(60409acec91e6aa02d4d39f8cfd4bc1f43b17031b1752da24f2f2e67f8aa896a);suite_power.py(bb534d7d908f64e01e45d2a039806fba22b54c994af9beb8713395f9ec53ff5a).