Identity Domains

The normative comparability spec for the Agentic-Coding Evaluation Lab

Two scores produced on different serving stacks are not the same measurement, even for identical model weights. Comparing them as if they were silently turns an infrastructure change into a fake model result. An identity domain is this project’s guard against that mistake: a fingerprint of the declared serving stack that decides which runs may be compared at all. Hash equality is necessary, not sufficient, for numerical comparability.

Why identity domains

Serving stacks do not promise bitwise-identical numerics across engine versions, hardware, or launch configuration. A score produced on one stack and a score produced on another are not the same measurement, even for the same model weights. Treating them as comparable silently converts an infrastructure change into a fake model result.

The identity domain makes known stack boundaries explicit and machine-checkable. Its v1 preimage — the exact set of declared fields that get hashed — covers the serving engine and version, serving and base-image digests, engine wheel hash, GPU hardware, model artifacts, and a fixed launch field set. Run ids and timestamps are excluded by construction.

Equality is necessary, not sufficient, for numerical comparability. V1 does not record every source of variation. Prefix-cache state, batch context and composition, session position, decoding parameters, harness version, and other residual state can differ while the hash remains unchanged. Those factors require separate controls and measurement; a later wider field set requires a new schema version.

Identity-domain matching is one admission decision — a check that determines whether two runs may enter the same comparison at all — and it fails closed:

• Runs are not admitted as score-comparable when their identity_domain_sha256 values differ.
• A comparison across differing domains is refused unless an explicit bridge authorization — a deliberate, recorded human sign-off (an operator-attested exception with a bridge id and a reason) — is recorded downstream in the evidence.
• A run that cannot present a well-formed domain hash is not admitted at all. Non-evaluable is not a pass.

A bridge records an exception to the equality rule. It does not prove that two domains are numerically equivalent.

The evidence contract carries the admission verdict in every published manifest (manifest.identity_domain), so a reader can always see whether a reported comparison stayed inside one domain or crossed on a recorded bridge.

The canonical field set (v1)

The field set below is rendered directly from the package constants in src/eval_lab_methodology/identity_domain.py, which define the frozen v1 bytes, required fields, hashing algorithm, and conformance vector. The module’s broader historical prose is superseded by the interpretation correction in ERRATA.md; the v1 byte-level contract is unchanged.

Section	Required keys
(top level)	schema_version, reference_lane, runtime, image, hardware, model, launch
runtime	engine, engine_version
image	eval_serving_image_digest, base_image_digest, vllm_wheel_sha256
image.base_image_digest	version, manifest_list, linux_amd64
hardware	gpu_name, gpu_count, vram_total_mb, driver_version, cuda_version, compute_capability
model	repo, revision, tokenizer_sha256, chat_template_sha256, weights_fingerprint_sha256
launch	runner, attention_backend, distributed_executor_backend, tensor_parallel_size, pipeline_parallel_size, device_ids, dtype, activation_dtype, weight_dtype, kv_cache_dtype, quantization_method, max_model_len, trust_remote_code, batch_invariant, served_model_name

schema_version must equal agentic-coding-evaluation-lab.identity-domain.v1, and reference_lane must be a non-empty string.

Forbidden at every nesting level (run-scoped, excluded by construction): completed_at, run_id, started_at, timestamp, timestamps.

Structural validation is fail-closed: validate_identity_domain rejects a domain with a wrong schema version, a missing section or key, a non-mapping section, or a forbidden run-scoped key anywhere in the tree. Value semantics beyond structure belong to the downstream builders that attest them.

The canonicalization rule

Every implementation hashes the identity domain with exactly this rule — the prefix is required and the hex digits are lowercase:

"sha256:" + hashlib.sha256(
    json.dumps(
        payload, sort_keys=True, separators=(",", ":"), ensure_ascii=True
    ).encode("utf-8")
).hexdigest()

Hash strings match ^sha256:[0-9a-f]{64}$. Key order cannot change the hash (sort_keys=True), whitespace cannot change the hash (compact separators), and non-ASCII text is escaped deterministically (ensure_ascii=True), so the same domain always produces the same bytes.

The conformance vector

CONFORMANCE_IDENTITY_DOMAIN is a fully synthetic, public-safe fixture carrying the complete v1 field set; its digests are repeated-hex placeholders. The frozen constant CONFORMANCE_IDENTITY_DOMAIN_SHA256 is the cross-repo conformance value. The cell below recomputes the hash live and asserts that it equals the frozen constant — rendering this page is itself a conformance proof.

import json

from eval_lab_methodology import (
    CONFORMANCE_IDENTITY_DOMAIN,
    CONFORMANCE_IDENTITY_DOMAIN_SHA256,
    identity_domain_sha256,
    validate_identity_domain,
)

assert validate_identity_domain(CONFORMANCE_IDENTITY_DOMAIN) == []
recomputed = identity_domain_sha256(CONFORMANCE_IDENTITY_DOMAIN)
assert recomputed == CONFORMANCE_IDENTITY_DOMAIN_SHA256

print(json.dumps(CONFORMANCE_IDENTITY_DOMAIN, indent=2))
print()
print("CONFORMANCE_IDENTITY_DOMAIN_SHA256 =", recomputed)

{
  "schema_version": "agentic-coding-evaluation-lab.identity-domain.v1",
  "reference_lane": "vllm-0.25.0-dedicated",
  "runtime": {
    "engine": "vllm",
    "engine_version": "0.25.0"
  },
  "image": {
    "eval_serving_image_digest": "sha256:abababababababababababababababababababababababababababababababab",
    "base_image_digest": {
      "version": "v0.25.0",
      "manifest_list": "sha256:cdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcdcd",
      "linux_amd64": "sha256:efefefefefefefefefefefefefefefefefefefefefefefefefefefefefefefef"
    },
    "vllm_wheel_sha256": "sha256:1212121212121212121212121212121212121212121212121212121212121212"
  },
  "hardware": {
    "gpu_name": "NVIDIA L40S",
    "gpu_count": 1,
    "vram_total_mb": 46068,
    "driver_version": "550.90.07",
    "cuda_version": "12.4",
    "compute_capability": "8.9"
  },
  "model": {
    "repo": "example-org/synthetic-model",
    "revision": "0123456789abcdef0123456789abcdef01234567",
    "tokenizer_sha256": "sha256:3434343434343434343434343434343434343434343434343434343434343434",
    "chat_template_sha256": "sha256:5656565656565656565656565656565656565656565656565656565656565656",
    "weights_fingerprint_sha256": "sha256:7878787878787878787878787878787878787878787878787878787878787878"
  },
  "launch": {
    "runner": "dedicated-gpu",
    "attention_backend": "FLASH_ATTN",
    "distributed_executor_backend": "mp",
    "tensor_parallel_size": 1,
    "pipeline_parallel_size": 1,
    "device_ids": [
      0
    ],
    "dtype": "bfloat16",
    "activation_dtype": "bfloat16",
    "weight_dtype": "bfloat16",
    "kv_cache_dtype": "fp8",
    "quantization_method": null,
    "max_model_len": 32768,
    "trust_remote_code": false,
    "batch_invariant": true,
    "served_model_name": "synthetic-model"
  }
}

CONFORMANCE_IDENTITY_DOMAIN_SHA256 = sha256:174b6b7acf10e667605d2d4cf144e540ed61e020df9f2907ca089342ba72e462

Versioning scope

The recorded scope is version, hardware, model artifact, image, and the v1 launch field set. An engine bump, serving-image change, different GPU, or changed recorded launch field produces a new identity domain and therefore a re-baseline. A matching domain does not waive the separate residual-state, assignment, and calibration requirements stated above.

Local-lane convention (separate from normative v1)

Small, same-host campaigns may use the distinct downstream convention agentic-coding-evaluation-lab.identity-domain.local-v1. Its lane fingerprint contains the local serving engine and version, host hardware, and launch configuration. It deliberately excludes per-model identity so a candidate and incumbent measured in the same lane receive the same domain hash by construction; model identity remains explicit elsewhere in the evidence manifest.

This is a documented adapter convention, not an extension or reinterpretation of the normative v1 field set above. It is not accepted by validate_identity_domain, does not alter the canonical v1 constants or frozen conformance vector, and uses its own schema_version precisely so the two contracts cannot be confused. Implementations using the local convention retain the canonical JSON and sha256: hashing rule, and must fail closed when the engine version, hardware, or launch configuration is unavailable.

Downstream conformance

This page is the normative definition; the private components conform to it functionally. The private deployment’s serving entrypoint and session validator, and the evaluation lab’s admission gate, each carry parity tests that reproduce this vector — the same fixture, the same canonicalization rule, the same frozen hash. A drift in any implementation fails its parity test before it can produce a misleadingly “comparable” number.