Files
msd-core/tests/qa/smell-fingerprint.cjs
Tom Boucher 33fd203ccd test(#2966): loop QA walk — drive real scenarios across all five loop steps (#2976)
* test(#2966): loop QA walk — drive real scenarios across all five loop steps

Adds a headless walk that carries accumulating project state across
discuss -> plan -> execute -> verify -> ship against one temp project,
layered over the existing tests/helpers.cjs runGsdTools substrate.

Findings carry severity. A violation breaks a stated contract and fails
the build; a smell is legal under today's implementation but structurally
questionable, is recorded, and never reddens CI. Without that split an
oracle set derived from current behavior can only ever confirm current
behavior -- the harness could not say "this works and is still wrong".

The end-to-end test asserts the walk produces at least one smell: a QA
harness that reports nothing on a first run against a real engine is far
more likely mis-specified than the engine is perfect. It deliberately does
not pin smell ids or counts, which would re-freeze current behavior.

First run against the real engine: 0 violations, 3 smell classes --
init returns agents_dir outside the project tree; smart-entry emits prose
unconditionally so routing cannot be asserted; state-snapshot reports a
missing STATE.md through a payload key with exit 0.

Also fixes tests/fixtures/index.cjs: createFixture with git:true and
planning:false staged nothing, so the commit failed with "nothing to
commit". That combination was unreachable until greenfield needed it.

Extends RULESET.TESTS.feedback-loop-convergence from estimation to the
loop itself. Design lock: docs/adr/2966-loop-qa-walk.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(#2966): wire fault injection, make perturbations discriminating

Independent review found tests/qa/mutations.cjs entirely unwired: 462
lines exercised only by their own unit tests, with no mutation hook in
the scenario DSL and no scenario applying one, while the module header
and the ADR described fault injection in the present tense. Dead code
documented as live.

Adds a `mutate` step field, three perturbation scenarios, and a wiring
detector: a self-test scenario whose expectations are known-false and
which MUST fail. The previous anti-vacuity check asserted only that the
walk produced a smell, which passes on well-known engine behavior
regardless of whether the harness wiring works.

First perturbation attempt produced zero signal -- progress does not
structurally parse ROADMAP.md, so a corrupted roadmap sailed through. A
perturbation that cannot fail is the same defect in a new costume.
Probes now target roadmap get-phase, and each mutated step runs a clean
baseline first so `mutationObserved` records whether the corruption
changed anything at all.

Also clears four review findings: classify() returned PROSE for exit-0
with empty stdout; `warnings` was structurally unpopulatable on the
success path (execFileSync discards it) and is now documented as
error-path-only; read-only-idempotence passed vacuously when asked to
check idempotence without the data to check it; the ADR miscounted the
oracles.

Discrimination matrix across 8 mutations x 6 commands: bom,
duplicate-phase-id and escaped-pipes are absorbed silently by every
probed surface, and progress / smart-entry / roadmap validate never
reacted to any mutation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(#2966): add path-containment guard for scenario-supplied targets

Security review found scenario-supplied paths joined to the temp project
with no containment check. step.mutate.target and agent.write keys were
validated only as non-empty strings, so a target of ../../../../etc/hosts
reached fs.unlinkSync / fs.writeFileSync / fs.symlinkSync outside the
project. The symlink mutation was worst: it read the traversed file, wrote
a sibling copy, deleted the original and symlinked it back.

Not exploitable today -- all shipped scenarios target .planning/ROADMAP.md
and scenarios are repo-committed, not runtime input. Fixed anyway: it is a
live primitive any future scenario or copied helper can reach.

Adds tests/qa/paths.cjs with resolveWithin(): rejects absolute paths, NUL
bytes and empty input, normalizes separators unconditionally, and requires
containment by path segment so a sibling like <base>-evil is not treated as
inside. Non-existent targets resolve via nearest existing ancestor rather
than falling back to a lexical compare. Scenario load now rejects traversing
or absolute targets up front.

oracles.cjs previously carried its own copy of the containment logic; both
now share paths.cjs, since a duplicated containment check is exactly the
divergence class this repo calls out.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(#2966): complete trajectory corpus, report emission, boundary-aware oracle

Adds the remaining trajectories and drives all 11 mutations end-to-end.
20 scenarios, 72 steps, 0 violations, 25 smells.

Adds qa-report.json with per-step verdicts and a copy-pasteable repro
command, plus --keep / GSD_QA_KEEP=1 to preserve a failing tree. A repro
line for a tree that was not preserved is marked NOT RUNNABLE rather than
emitting a command pointing at a deleted directory.

monotonic-progress is now boundary-aware. Two scenarios had been trimmed
to stop the oracle complaining at a milestone rollover, which destroys the
signal the trajectory exists to produce. Evidence: counters legitimately
reset to zero at milestone complete, but the payload milestone_version
lags until a new ROADMAP.md is written. So the oracle now scopes by
milestone plus workstream, keeps a same-scope decrease as a violation, and
records a boundary crossing as a smell. Both scenarios walk the real
boundary again.

Standards review fixes: oracle findings now carry a structured subject so
tests assert on typed fields instead of substring-matching the free-form
detail string, resolveWithin throws a typed EPATHESCAPE error, and the
absolute-path predicate scenario.cjs had re-implemented now comes from
paths.cjs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(#2966): fix silently-vacuous fixtures and guard the class

Every fixture carried its #2371 provenance comment BEFORE the frontmatter
block, and extractFrontmatter returns {} when anything precedes the opening
---. So every scenario reading status/phase/name was operating on an empty
object and reporting green. Nine fixtures repositioned; the comment stays,
it just moves below the closing ---.

Both UAT fixtures lacked a parser-recognized result block, so
evaluateUatPassed saw checks.length===0 and could never return passed:true.
The uat-fail-then-remediate scenario could not have proven a remediation.
Its expect block only inspected blockers, which is empty before AND after,
which is why the corpus never noticed. Both fixtures now carry real result
blocks and the scenario asserts passed and no_uat_artifacts on each side of
the flip.

The actual deliverable is the guard: a fixture-integrity block asserting
every fixture with a frontmatter shape parses to a non-empty object, that
every fixture carries its provenance marker, and that the two UAT fixtures
produce opposite verdicts through the real evaluateUatPassed. The first
guard written required --- at byte 0, which would never have fired on the
regression it exists to prevent; it was rewritten and proven by deliberately
re-breaking a fixture.

No engine defect here. no_uat_artifacts means no parsed check items, not no
UAT files, and it was reporting correctly on fixtures that had none.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(#2966): make the walk report — smell ratchet, baseline, CI job

The harness computed smells into a gitignored qa-report.json that nothing
read. In CI it surfaced nothing at all: violations failed the build, but the
half of the tool that says "this works and is still wrong" was inert. A QA
tool nobody hears is decoration.

Adds a ratchet on the same idiom this repo already uses three times over
(the regression-test-name allowlist, the emitted-drift acks, the size
baseline): a committed smell-baseline.json, per-PR acknowledgment fragments
under tests/qa/smell-acks/, and a ratchet script wired into CI.

The design invariant is preserved exactly. A smell still never fails a build
on its own merits. What fails is an UNACKNOWLEDGED NEW smell -- the absence
of a decision -- leaving an author two honest exits: fix it, or record a
fragment with a real reason. An empty reason is rejected. The baseline is
shrink-only, so a fixed smell must prune its entry. Violations remain
unacknowledgeable.

Fingerprints are composed only from stable fields (oracle id, scenario,
argv, subject discriminator) -- never temp paths, timestamps or counts.
Verified byte-identical across two runs in separate temp dirs; an unstable
fingerprint would have false-positived every CI run.

CI gains a qa-loop-walk job that runs the suite and the ratchet, uploads the
report with `if: always()` (it matters most when it failed), and renders a
summary a reviewer reads without downloading anything.

Also fixes the report runner invoking main() unconditionally on require, so
importing it double-ran every scenario and clobbered its own output.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(#2966): every smell terminates in a defect or a fixed detector

The baseline accepted a smell with a free-text reason. That is a mechanism
for designing smells in -- an allowlist nobody revisits. The harness is
brand new, so nothing it found is inherited legacy; every finding is a
FIRST finding. Each must now terminate in exactly one of two states:

  REAL           -> an assigned defect, entry carries the issue number
  FALSE POSITIVE -> the detector is wrong and gets fixed, never baselined

There is no third "accepted with a good explanation" state, so the ratchet
now requires a positive-integer `issue` on every entry. A reason may remain
as a human note but can never substitute. `--update` refuses to invent
issue numbers: a new smell is written with `issue: null` and a TODO, and
the next plain run rejects it, forcing triage rather than accumulation.

Working the 21 existing entries through that rule found 16 were my own
detectors being wrong:

value-hygiene (10) flagged $.agents_dir, a field whose entire contract is
to point at the install tree outside any project. Fixed with a leaf-key
allowlist of contractually-external fields, verified as the only such key
in the init payload. Genuinely unexpected out-of-project paths still smell.

monotonic-progress (6) fired on legitimate boundary crossings -- milestone
v1.0 to v2.0, workstream beta to alpha -- and on one payload carrying no
scope fields at all, where a change cannot even be known. Scope changes now
reset silently and scope-less observations are skipped. The same-scope
decrease remains a violation; that is the real invariant and is regression-
guarded.

The five survivors are real and now tracked: soft-error-exit-zero (#2980),
untyped-success (#2979). Baseline 25 -> 5.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(#2966): keep the ratchet out of the tarball, unpin the qa CI job

The remote matrix returned failed -- 3 unique failures, identical on
node22 and node24, both root causes in this branch's own diff.

The ratchet lives under scripts/, which ships in the npm tarball, and it
requires three modules under tests/, which does not. In a published
install it is MODULE_NOT_FOUND at load. This is exactly the class the
#2858 guard was added to catch, and it caught it. Fixed the way #2858
fixed the same shape for its own repo-only CI script: a targeted files[]
negation, so the ratchet stays in the repo for CI and out of the tarball.
Not solved by moving or inlining the required modules -- the ratchet must
keep using the same code the harness uses, or the two drift.

Verified both directions: the script is no longer in the pack list, and
build-hooks.js, fix-slash-commands.cjs and gen-capability-registry.cjs are
all still shipped. Over-negating there would have broken installs, since
bin/install.js requires them.

The qa-loop-walk job also carried CI_REBASE_BASE_SHA copied from a
neighbouring job without the paired GSD_EMITTED_BASE, which the #2854
invariant forbids by name: diverging them makes the differential compare a
tree against a baseline from a different commit. The job runs only the qa
suite and the ratchet and invokes no emitted-attribution test, so it needs
no rebase-pinned base at all -- the step was removed rather than paired.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(#2966): stop monotonic-progress going blind on scope-less payloads

The full remote suite caught a false NEGATIVE I introduced while fixing a
false positive. Silencing the boundary-crossing noise had made the oracle
skip ANY observation lacking milestone fields -- so a minimal payload like
{total_summaries: n} produced no violation at all, and the oracle stopped
catching the exact defect it exists to catch. For a QA tool that is
strictly worse than the noise it replaced.

Scope is only indeterminate when the two observations DISAGREE about
having it:

  both scoped, same scope, decrease -> VIOLATION
  both scoped, different scope      -> reset silently
  NEITHER scoped, decrease          -> VIOLATION   (the regression)
  mixed                             -> skip the comparison

Implementing the mixed case surfaced a second blind spot: advancing the
reference point on a skipped pair lets a scope-less observation sitting
between two same-scope ones mask a real decrease. Mixed now leaves the
reference untouched. All four branches carry explicit coverage; only one
did before, which is why this shipped.

The self-test that failed was right and the code was wrong, so the code
moved. Corpus behavior is unchanged: still 5 smells, 0 new, 0 stale, 0
violations.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 16:13:28 -04:00

197 lines
11 KiB
JavaScript

'use strict';
/**
* smell-fingerprint.cjs — a STABLE identity for one QA-walk "smell" finding
* (see `oracles.cjs`'s `SEVERITY.SMELL`), across runs, temp dirs, and hosts.
*
* WHY THIS FILE EXISTS
* ────────────────────
* `scripts/qa-smell-ratchet.cjs` needs to answer "is this the SAME smell we
* already decided about, or a NEW one?" across two runs that never share a
* filesystem state — every run gets its own `mkdtemp`'d project directory
* (see `loop-walk.cjs`), so anything that leaks a temp path, a byte count, a
* timestamp, or free-form prose into the identity would make every run look
* "new" even when nothing about the underlying behavior changed. That would
* make the ratchet permanently red (or, worse, silently trained to be
* ignored) — exactly the failure mode `assertWithinAllowlist`
* (`scripts/lib/allowlist-ratchet.cjs`) exists to prevent for other guards.
*
* STABLE INPUTS ONLY
* ───────────────────
* `fingerprint(scenarioName, smell)` composes from exactly four stable
* fields:
* 1. `smell.id` — the oracle id (e.g. `"value-hygiene"`), a closed,
* versioned enum (`ORACLES` in `oracles.cjs`).
* 2. `scenarioName` — the scenario's own `name` field, author-chosen and
* fixed at scenario-authoring time.
* 3. `smell.argv` — the exact CLI invocation (`step.argv`, an array of
* literal argument strings) joined with a single
* space. This is deterministic: two runs of the same
* scenario always issue the same argv for the same
* step.
* 4. `smell.subject` — a STRUCTURAL discriminator extracted from the
* oracle's own `subject` payload (see `oracles.cjs`'s
* `OracleOutcome` typedef), via `subjectDiscriminator`
* below. Only whitelisted, provably-stable fields are
* read from `subject` — see that function's header.
*
* `smell.detail` (free-form prose, may embed a temp path or a byte count) is
* NEVER read here. Neither is anything derived from `Date.now()` /
* `Math.random()` / the process's own PID or hostname — this module performs
* NO I/O and reads NO ambient state, so `fingerprint()` is a pure function of
* its two arguments.
*
* WHY A HASH *AND* A READABLE COMPOSITE (not one or the other)
* ──────────────────────────────────────────────────────────────
* A pure hash (e.g. sha256 of the four fields) is technically sufficient as
* an identity, but it makes `tests/qa/smell-baseline.json` an opaque blob: a
* reviewer diffing a PR that touches the baseline sees `"a1b2c3d4e5f6"` roll
* to `"9f8e7d6c5b4a"` and cannot tell what changed without re-running the
* ratchet themselves. A pure readable composite (no hash) is diffable but
* fragile to injection: nothing stops a scenario named `"a|b"` combined with
* an argv token containing `|` from colliding with an unrelated
* oracle/scenario/argv/subject tuple that happens to serialize to the same
* joined string, and JSON.stringify-based hashing of an array does not have
* that ambiguity (array elements are individually length-prefixed by the
* serializer's own quoting/escaping rules).
*
* The chosen key is therefore BOTH, concatenated as `"<hash>::<readable>"`:
* - `<hash>` (first 12 hex chars of sha256 over an unambiguous JSON-array
* encoding of the four fields) is the collision-safe, canonical identity
* — this is the part any two runs of the same finding are GUARANTEED to
* agree on bit-for-bit, regardless of what punctuation an author put in
* a scenario name or an argv token.
* - `<readable>` (the four fields pipe-joined, human-legible) is what makes
* `smell-baseline.json` diffable in a PR review — a reviewer can read
* `value-hygiene|greenfield-happy-path|--json-errors init new-project|key=$.agents_dir`
* and immediately know what fired, without decoding a hash.
* Per the brief's "prefer the readable composite as the key if it is
* deterministic" guidance: the readable half is deterministic here (all four
* inputs are literal, author-controlled strings/arrays with no free
* variation run-to-run), so it is safe to keep in the identity rather than
* discarding it in favor of the hash alone.
*
* @module tests/qa/smell-fingerprint
*/
const crypto = require('node:crypto');
/**
* Deterministically stringify a plain JSON-ish value with object keys sorted
* recursively, so two structurally-equal objects with keys inserted in a
* different order always serialize identically. Arrays keep their order
* (order is semantically meaningful for an array; it is not for object keys).
*
* Used only on the small, already-whitelisted `subject.fromScope` /
* `subject.toScope` scope objects (see `progressScopeOf` in `oracles.cjs`) —
* never on an arbitrary/attacker-shaped value — so cycle-safety is
* deliberately NOT handled here (those objects are always flat
* `{milestoneVersion, milestoneName, workstream}` records).
*
* @param {unknown} value
* @returns {string}
*/
function stableStringify(value) {
if (value === null || typeof value !== 'object') return JSON.stringify(value);
if (Array.isArray(value)) return `[${value.map(stableStringify).join(',')}]`;
const keys = Object.keys(value).sort();
return `{${keys.map((k) => `${JSON.stringify(k)}:${stableStringify(value[k])}`).join(',')}}`;
}
/**
* Extract a stable, human-readable discriminator from an oracle's `subject`
* payload (see `oracles.cjs`'s `OracleOutcome` typedef for the per-oracle
* shapes). Reads ONLY fields that are, by construction, free of run-to-run
* variation:
*
* - `subject.key` (`value-hygiene`'s JSON-path leaf, e.g.
* `"$.agents_dir"` — a structural path, never
* the leaked VALUE at that path, which may be
* a temp-dir-dependent absolute path)
* - `subject.missing` (`read-only-idempotence`'s absent-input
* case — a fixed, closed set of field names)
* - `subject.fromScope`/`toScope` (`monotonic-progress`'s scope-boundary
* case — `{milestoneVersion, milestoneName,
* workstream}`, all literal/author-controlled
* strings; the sibling `from`/`to`/
* `fromIndex`/`toIndex` counters on that same
* subject are deliberately NEVER read here,
* since counters are exactly the kind of
* per-run-variable field the brief bans)
*
* `subject.argv` (present on `soft-error-exit-zero` / `untyped-success` /
* `contract-conflict`) is deliberately NOT read here — it duplicates
* `smell.argv`, which `fingerprint()` already folds in separately, so reading
* it again here would add nothing but risk drifting out of sync with that
* field. A `subject.mismatches` array (`read-only-idempotence`'s
* data-present-but-differs case) is also deliberately NOT read here — its
* entries embed live stat facts (`size`, `mtimeMs`) that vary by definition
* between runs, so no discriminator can be derived from it without violating
* the "no counts" rule; that oracle is a VIOLATION-only check today (never a
* SMELL — see `oracles.cjs`), so this gap is currently unreachable in
* practice, but is documented here rather than silently mishandled if that
* ever changes.
*
* @param {unknown} subject
* @returns {string}
*/
function subjectDiscriminator(subject) {
if (!subject || typeof subject !== 'object' || Array.isArray(subject)) return '(no-subject)';
const parts = [];
if (typeof subject.key === 'string') parts.push(`key=${subject.key}`);
if (typeof subject.missing === 'string') parts.push(`missing=${subject.missing}`);
if (subject.fromScope !== undefined) parts.push(`fromScope=${stableStringify(subject.fromScope)}`);
if (subject.toScope !== undefined) parts.push(`toScope=${stableStringify(subject.toScope)}`);
return parts.length ? parts.join(';') : '(subject-with-no-stable-discriminator)';
}
/**
* Compute the stable fingerprint for one smell finding within one scenario.
*
* PURITY GUARANTEE: this function performs no I/O, reads no ambient state
* (`Date.now()`, `Math.random()`, env vars, the filesystem), and its return
* value is a pure function of `scenarioName` and the four stable fields read
* off `smell` (see this file's header). Two calls with structurally-equal
* arguments — even across two separate process invocations, two different
* temp directories, two different hosts — MUST return byte-identical
* strings. This is proven empirically in `scripts/qa-smell-ratchet.cjs`'s
* `--update` flow (fingerprinting the same scenario corpus twice, in two
* separate `mkdtemp` runs, and diffing the resulting key sets) and in the
* VERIFY section of the PR that introduced this module.
*
* @param {string} scenarioName the owning scenario's `name` field.
* @param {{id: string, subject?: object, argv?: string[]}} smell one finding
* from a step's `smells` array, AUGMENTED with that step's own `argv` (the
* raw `runOracles` finding shape from `oracles.cjs` carries `id`/`detail`/
* `subject` only — `detail` is intentionally never read by this function;
* the caller is responsible for attaching the owning step's `argv`).
* @returns {string} `"<12-hex-char sha256>::<id>|<scenarioName>|<argv joined
* with a space>|<subject discriminator>"`.
* @throws {TypeError} when `scenarioName` is not a non-empty string or
* `smell.id` is not a non-empty string — a fingerprint computed from a
* malformed input would silently corrupt the whole ratchet's identity
* space, so this fails loudly instead.
*/
function fingerprint(scenarioName, smell) {
if (typeof scenarioName !== 'string' || scenarioName === '') {
throw new TypeError(`fingerprint: scenarioName must be a non-empty string, got ${JSON.stringify(scenarioName)}`);
}
if (!smell || typeof smell !== 'object' || typeof smell.id !== 'string' || smell.id === '') {
throw new TypeError(`fingerprint: smell.id must be a non-empty string, got ${JSON.stringify(smell && smell.id)}`);
}
const argvJoined = Array.isArray(smell.argv) ? smell.argv.join(' ') : '';
const subjectPart = subjectDiscriminator(smell.subject);
const readable = `${smell.id}|${scenarioName}|${argvJoined}|${subjectPart}`;
const hash = crypto
.createHash('sha256')
.update(JSON.stringify([smell.id, scenarioName, argvJoined, subjectPart]))
.digest('hex')
.slice(0, 12);
return `${hash}::${readable}`;
}
module.exports = { fingerprint, subjectDiscriminator, stableStringify };