* test(#2773): failing-first contract and premise tests for translated edge-probe input Locks the Step 5.5 contract that a response_language project must feed the edge probe an English translation of each requirement's text, and binds that advice to measured engine behavior: the same requirement classifies to zero shapes in Portuguese and to collection/adjacency/empty/ordering in English. Also pins the honest limit — the issue's own repro sentence classifies to [] in English too, so translation is necessary but not sufficient and the authored shapes override is the documented fallback. Red before the doc change; the assertions are all false today. Refs #2773 * fix(#2773): feed the spec-phase edge probe English-translated requirement text The shape cues in src/edge-probe.cts are English word-boundary regexes, so a project running with response_language set wrote its SPEC requirements into the Step 5.5 $REQS_JSON heredoc in that language, matched no cue, classified to zero shapes, and landed every row in the unclassified sentinel (#1110). The taxonomy contributed nothing and --auto left it all unresolved — the probe was a silent no-op for exactly the spec type it exists to harden. Step 5.5 now states that the $REQS_JSON payload is engine input rather than user-facing output, so the response_language rule does not govern it: each requirement's text carries a faithful English translation, the SPEC keeps its original language, and requirement ids are never translated or renumbered. The instruction sits before the heredoc on purpose — the downstream APPLICABLE=0 warning fires only when every requirement is unclassified, so a partly-classified non-English spec would otherwise slip through with no signal at all. Measured against the compiled engine: the same requirement returns [] in Portuguese and collection -> adjacency/empty/ordering in English. Also measured: the issue's own repro sentence returns [] in English too, so translation is necessary but not sufficient — the instruction therefore points at the authored shapes override for prose carrying no cue in any language rather than promising that translation restores classification. Doc scope only, per the triage disposition on the issue. The compiled engine is untouched; the lang-hint / per-language cue-set fix is a separate follow-up. Closes #2773 * fix(#2773): clean up the edge-probe temp file on the placeholder-guard exit path Surfaced by the isolated security review of this branch. Between the mktemp and the unconditional cleanup, Step 5.5 has two sibling guards that disagreed about their own invariant: the engine-failure guard runs rm -f "$REQS_JSON" before exiting, while the empty/placeholder guard directly above it exited without one. A spec run that tripped the placeholder check therefore stranded a temp file holding the SPEC's requirement text in TMPDIR, once per failed run. The added contract test walks the region between the mktemp and the unconditional cleanup and asserts no exit path leaves the file behind, so the two guards can no longer drift apart. Proven to bind: run against the pre-fix file the walker reports the leaking exit; against the fixed file it reports none. Refs #2773 * docs(#2773): record the edge probe's English-cue input constraint in the predicate store The co-change gate flagged CONTEXT.md (13 co-changes with spec-phase.md) and docs/CONFIGURATION.md (11) as candidate-missing-updates, and both were real gaps rather than incidental coupling. CONTEXT.md's EdgeCompletenessProbeModule entry documents the input contract for classifyShape but did not record that SHAPE_CUES are English word-boundary patterns — so the predicate store implied text was language-agnostic, which is what a future agent reads before touching this seam. docs/CONFIGURATION.md's response_language row is what a non-English project reads when it turns the setting on; it now names the one deliberate exception and links to the FEATURES.md explanation, so the interaction is discoverable from the config key rather than only from the workflow. CONTEXT-INDEX.json regenerated via gen-context-index.cjs --write. The drift-ack fragment is updated for the final byte range and now also records the placeholder-guard cleanup fix folded into the same block. Refs #2773 * fix(#2773): append the growth rationale to the existing spec-phase.md ack entry The remote runner caught this: emitted-attribution.test.cjs pins the 0000-legacy-migration.json spec-phase.md entry permanently (the #2914 migration regression test asserts the exact '31987 -> 31997' delta text survives), so removing it to avoid a duplicate-key collision with a new fragment broke that test instead of satisfying the ratchet. The entry is an accreting log, not a single-use slot — #2733, #3132 and #3102 were each appended to the same reason string by later PRs, which is how a shared growth key coexists with the rule that two ack sources may never name the same path. This appends the #2773 rationale the same way and drops the separate fragment, whose spec-phase.md key was the collision. Verified locally by reproducing both affected tests against the real fragment before re-dispatching: the pinned delta survives, grown[0].acked is true, staleAcks is empty, and all 35 entries still read as spent. Refs #2773 * docs(#2773): add a how-to for probing edges in a non-English project The phase gate's enablementSequence check caught a wrong call of mine. I had recorded that no how-to was owed because the user takes zero extra steps — the workflow translates the probe input itself. Written out, though, the sequence from off to value is two steps and step 1 depends on response_language, a setting owned by a different capability than the edge probe, which is exactly the condition the how-to test names. There is also real task content a reference table cannot carry: the three-way split between a few unclassified rows (the classifier's recall gap), every row unclassified (the probe could not read the spec at all), and the silent partly-classified case where the APPLICABLE=0 warning never fires. That last one is what a user would otherwise misread as a clean bill of health. Shaped after the resolve-edge-coverage-findings / resolve-unreachable-guard siblings and indexed from docs/README.md next to its closest relative. Refs #2773 * chore(#2773): backfill the changeset PR number pr:0 placeholder replaced with the real PR number now that #3713 exists. Refs #2773 --------- Co-authored-by: sim <sim@local>
This commit is contained in:
Tom Boucher
@@ -14,8 +14,13 @@ const { test } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const fc = require('fast-check');
|
||||
|
||||
const SPEC_PHASE_PATH = path.join(__dirname, '..', 'gsd-core', 'workflows', 'spec-phase.md');
|
||||
const EDGE_PROBE_REF_PATH = path.join(__dirname, '..', 'gsd-core', 'references', 'edge-probe.md');
|
||||
const { classifyShape, applicableCategories } = require(
|
||||
path.join(__dirname, '..', 'gsd-core', 'bin', 'lib', 'edge-probe.cjs'),
|
||||
);
|
||||
|
||||
function readSpecPhase() {
|
||||
return fs.readFileSync(SPEC_PHASE_PATH, 'utf8');
|
||||
@@ -299,3 +304,203 @@ test('#3132: specless-probe-fallback.md uses resolved+verification — not cover
|
||||
assert.match(content, /verification: explicit/,
|
||||
'specless-probe-fallback.md must reference "verification: explicit"');
|
||||
});
|
||||
|
||||
// ─── #2773: non-English requirements and the English-only shape cues ──────────
|
||||
//
|
||||
// `SHAPE_CUES` (src/edge-probe.cts) are English word-boundary regexes. A project that sets
|
||||
// `response_language` writes its SPEC Requirements in that language, so transcribing them
|
||||
// verbatim into the Step 5.5 `$REQS_JSON` heredoc classifies every requirement to zero shapes
|
||||
// -> every row becomes the `unclassified` sentinel (#1110) and the 8-category taxonomy
|
||||
// contributes nothing. Approved scope for #2773 is doc-only: Step 5.5 must instruct that the
|
||||
// probe's `text` field carries a faithful English translation (engine input, never
|
||||
// user-facing), while the SPEC itself stays in the original language.
|
||||
|
||||
test('#2773: Step 5.5 documents the English-translation step for response_language projects', () => {
|
||||
const block = extractStep55Block(readSpecPhase());
|
||||
assert.ok(block.length > 0, 'Step 5.5 block must be extractable from spec-phase.md');
|
||||
|
||||
assert.match(
|
||||
block,
|
||||
/response_language/,
|
||||
'Step 5.5 must name `response_language` — it is the setting that makes the requirement prose non-English',
|
||||
);
|
||||
assert.match(
|
||||
block,
|
||||
/translat/i,
|
||||
'Step 5.5 must instruct that the probe input carries a translation, not the original-language prose',
|
||||
);
|
||||
assert.match(
|
||||
block,
|
||||
/English/,
|
||||
'Step 5.5 must say the translation target is English (the cue set the classifier actually speaks)',
|
||||
);
|
||||
// The SPEC must NOT be anglicized — only the transient probe payload is translated.
|
||||
assert.match(
|
||||
block,
|
||||
/SPEC[^\n]*(original language|stays in|keeps)|(original language)[^\n]*SPEC/i,
|
||||
'Step 5.5 must state the SPEC keeps the original language — only the probe input is translated',
|
||||
);
|
||||
// Requirement ids are the join key for coverage rows; translating them breaks the mapping.
|
||||
assert.match(
|
||||
block,
|
||||
/`?id`?s?[^\n]*(unchanged|not translated|never translated|stable)/i,
|
||||
'Step 5.5 must state requirement ids are NOT translated or renumbered (coverage rows join on id)',
|
||||
);
|
||||
});
|
||||
|
||||
test('#2773: Step 5.5 names the authored shapes override as the zero-cue fallback', () => {
|
||||
const block = extractStep55Block(readSpecPhase());
|
||||
assert.ok(block.length > 0, 'Step 5.5 block must be extractable from spec-phase.md');
|
||||
|
||||
// Translation is necessary but NOT sufficient: prose carrying no cue in ANY language still
|
||||
// classifies to []. The engine already accepts an authored `shapes` override for exactly
|
||||
// that case, so the instruction must point at it rather than over-promise.
|
||||
assert.match(
|
||||
block,
|
||||
/`shapes`/,
|
||||
'Step 5.5 must name the authored `shapes` override as the fallback for prose that still classifies to zero',
|
||||
);
|
||||
});
|
||||
|
||||
test('#2773: the translation instruction precedes the $REQS_JSON write', () => {
|
||||
const block = extractStep55Block(readSpecPhase());
|
||||
assert.ok(block.length > 0, 'Step 5.5 block must be extractable from spec-phase.md');
|
||||
|
||||
const langIdx = block.search(/response_language/);
|
||||
const writeIdx = block.search(/>\s*"\$REQS_JSON"/);
|
||||
assert.ok(langIdx !== -1, 'Step 5.5 must mention response_language');
|
||||
assert.ok(writeIdx !== -1, 'Step 5.5 must write $REQS_JSON');
|
||||
assert.ok(
|
||||
langIdx < writeIdx,
|
||||
'the translation instruction must come BEFORE the $REQS_JSON write — the downstream `$APPLICABLE = 0` warning only fires when EVERY requirement is unclassified, so a partially-classified non-English spec would slip through silently',
|
||||
);
|
||||
});
|
||||
|
||||
test('#2773: translating a non-English requirement is what makes the cue matcher apply', () => {
|
||||
// Same requirement, two languages. This is the premise the Step 5.5 instruction rests on;
|
||||
// if a future SHAPE_CUES edit breaks it, the documented advice becomes false and this fails.
|
||||
const pt = 'O sistema mescla intervalos sobrepostos em uma lista ordenada';
|
||||
const en = 'The system merges overlapping intervals in a sorted list';
|
||||
|
||||
assert.deepEqual(classifyShape(pt), [], 'non-English prose matches no English cue — the silent no-op #2773 reports');
|
||||
assert.deepEqual(applicableCategories(classifyShape(pt)), [], 'zero shapes raise zero categories');
|
||||
|
||||
const enShapes = classifyShape(en);
|
||||
assert.ok(enShapes.includes('collection'), 'the English rendering must classify as a collection');
|
||||
const enCategories = applicableCategories(enShapes);
|
||||
for (const expected of ['adjacency', 'empty', 'ordering']) {
|
||||
assert.ok(enCategories.includes(expected), `translated requirement must raise \`${expected}\``);
|
||||
}
|
||||
|
||||
// A second shape, so the row is not a single-cue coincidence.
|
||||
const ptText = 'O nome do usuario e truncado em 50 caracteres';
|
||||
const enText = 'The user name is truncated at 50 characters';
|
||||
assert.deepEqual(classifyShape(ptText), [], 'non-English text-shape prose also matches nothing');
|
||||
assert.ok(classifyShape(enText).includes('text'), 'the English rendering must classify as text');
|
||||
|
||||
// Multi-cue: a sentence carrying cues for two shapes yields the UNION, not one of them.
|
||||
const multi = 'The API request uploads a sorted list of items';
|
||||
const multiShapes = classifyShape(multi);
|
||||
assert.ok(multiShapes.includes('io'), 'multi-cue prose must include io');
|
||||
assert.ok(multiShapes.includes('collection'), 'multi-cue prose must include collection');
|
||||
});
|
||||
|
||||
test('#2773: translation alone does not rescue genuinely zero-cue prose', () => {
|
||||
// The issue's own repro sentence classifies to [] in ENGLISH too — it carries no shape cue
|
||||
// in any language. That is the recorded recall gap (ADR-857 §98 / ADR-550 D7b), not a
|
||||
// language failure, which is why Step 5.5 must point at the `shapes` override rather than
|
||||
// promise that translation restores classification.
|
||||
const enZeroCue = 'The command exits with code 1 and prints to stderr on invalid input';
|
||||
assert.deepEqual(
|
||||
classifyShape(enZeroCue),
|
||||
[],
|
||||
'a genuinely zero-cue requirement stays unclassified even in English — the doc must not over-promise',
|
||||
);
|
||||
|
||||
// The authored `shapes` override is the deterministic escape hatch for exactly this case.
|
||||
assert.deepEqual(
|
||||
applicableCategories(['stateful']),
|
||||
['idempotency', 'concurrency'],
|
||||
'an authored `shapes` override raises categories with no dependence on prose cues',
|
||||
);
|
||||
});
|
||||
|
||||
test('#2773 property: a cue word classifies regardless of surrounding text', () => {
|
||||
// A translated sentence carries its cue word amid arbitrary other words. The Step 5.5
|
||||
// advice is only sound if classification is robust to that surrounding context rather than
|
||||
// anchored to a fixed sentence shape.
|
||||
const cueForShape = {
|
||||
'numeric-range': 'threshold',
|
||||
collection: 'items',
|
||||
text: 'unicode',
|
||||
stateful: 'persist',
|
||||
io: 'endpoints',
|
||||
};
|
||||
const filler = fc.string({ minLength: 0, maxLength: 24 }).filter((s) => !/[A-Za-z]/.test(s));
|
||||
|
||||
fc.assert(
|
||||
fc.property(
|
||||
fc.constantFrom(...Object.keys(cueForShape)),
|
||||
filler,
|
||||
filler,
|
||||
(shape, before, after) => {
|
||||
const sentence = `${before} ${cueForShape[shape]} ${after}`;
|
||||
assert.ok(
|
||||
classifyShape(sentence).includes(shape),
|
||||
`cue "${cueForShape[shape]}" must classify as ${shape} inside ${JSON.stringify(sentence)}`,
|
||||
);
|
||||
},
|
||||
),
|
||||
{ numRuns: 100, seed: 2773 },
|
||||
);
|
||||
});
|
||||
|
||||
test('#2773: the edge-probe reference documents the English-cue assumption', () => {
|
||||
const ref = fs.readFileSync(EDGE_PROBE_REF_PATH, 'utf8');
|
||||
assert.match(
|
||||
ref,
|
||||
/English/,
|
||||
'the edge-probe reference `## Inputs` contract must state that the heuristic classifier is English-cue based',
|
||||
);
|
||||
assert.match(
|
||||
ref,
|
||||
/response_language/,
|
||||
'the edge-probe reference must point non-English projects at the translated-input requirement',
|
||||
);
|
||||
});
|
||||
|
||||
test('#2773: no Step 5.5 exit path leaks the $REQS_JSON temp file', () => {
|
||||
// The temp file holds the SPEC's requirement text. Every guard between its creation and
|
||||
// the unconditional cleanup must `rm -f` it before `exit 1`, or a failed spec run strands
|
||||
// requirement content in TMPDIR. The engine-failure guard always did; the empty/placeholder
|
||||
// guard directly above it did not, so the two siblings disagreed about their own invariant.
|
||||
const block = extractStep55Block(readSpecPhase());
|
||||
assert.ok(block.length > 0, 'Step 5.5 block must be extractable from spec-phase.md');
|
||||
|
||||
const lines = block.split('\n');
|
||||
const createIdx = lines.findIndex((l) => /REQS_JSON=\$\(mktemp/.test(l));
|
||||
assert.ok(createIdx !== -1, 'Step 5.5 must create $REQS_JSON via mktemp');
|
||||
|
||||
// The region ends at the first UNCONDITIONAL cleanup (a bare `rm -f "$REQS_JSON"` at column
|
||||
// zero); past that the file is already gone and later exits cannot leak it.
|
||||
const afterCreate = lines.slice(createIdx + 1);
|
||||
const endOffset = afterCreate.findIndex((l) => /^rm -f "\$REQS_JSON"/.test(l));
|
||||
assert.ok(endOffset !== -1, 'Step 5.5 must unconditionally rm -f "$REQS_JSON" after the engine run');
|
||||
const region = afterCreate.slice(0, endOffset);
|
||||
|
||||
// Walk the region tracking whether the current guard branch has cleaned up. `then`/`else`
|
||||
// opens a fresh branch; a cleanup inside it arms the branch; an `exit` must find it armed.
|
||||
let cleanedInBranch = false;
|
||||
const leaks = [];
|
||||
for (const line of region) {
|
||||
if (/\bthen\b|^\s*else\b|^\s*elif\b/.test(line)) cleanedInBranch = false;
|
||||
if (/rm -f "\$REQS_JSON"/.test(line)) cleanedInBranch = true;
|
||||
if (/^\s*exit\s+\d+/.test(line) && !cleanedInBranch) leaks.push(line.trim());
|
||||
}
|
||||
|
||||
assert.deepEqual(
|
||||
leaks,
|
||||
[],
|
||||
`every exit between the mktemp and the unconditional cleanup must rm -f "$REQS_JSON" first; leaking exits: ${JSON.stringify(leaks)}`,
|
||||
);
|
||||
});
|
||||
|
||||
@@ -36,7 +36,7 @@
|
||||
"agents/gsd-verifier.toml": "#2834: Codex agent TOML now carries model-routing fields on first install.",
|
||||
"gsd-code-fixer.md": "#2647: the three worktree-path sites (setup_worktree bash, concrete-steps prose, critical_rules) replaced the hardcoded /tmp/sv- mktemp path with a repo-relative .claude/worktrees/rf-<phase>-<pid>-<epoch> path, and added a defense-in-depth padded_phase validation at the sink (the agent prompt is a literal bash contract any caller can spawn; the orchestrator validates upstream but the sink now self-defends against path-traversal/branch-name injection). On Windows/Git Bash the /tmp path landed outside the project tree (outside the session permission allowlist, prompting on every read) and mktemp's MAX_PATH substitute was un-removable; .claude/worktrees/ is the same dir the harness-managed executor worktrees use (gitignored via .claude/, inside the permission scope). Growth is the path-resolution bash (main_repo via `git worktree list --porcelain | awk`) + the $$-PID/epoch uniqueness replacing mktemp's XXXXXX + the padded_phase guard + the #2647 rationale comments at each site. Supersedes the prior #2825 attribution, whose gated-bash + guardrail growth is already in next.",
|
||||
"spec-phase.md": {
|
||||
"reason": "#2733: five transitions in gsd-core/workflows/spec-phase.md were re-pointed so control reaches the mandatory Step 5.5 edge-completeness and Step 5.6 prohibition-completeness probes, which no path could reach before. Four upstream gate-passed jumps went from 'Jump to Step 6' to 'Jump to Step 5.5', and Step 5.5's own terminal soft gate at :305 went from 'proceed to Step 6' to 'proceed to Step 5.6' so the common all-edges-resolved path stops skipping the prohibition probe. The +10 bytes is exactly those five targets growing by 2 bytes each ('Step 6' -> 'Step 5.5' / 'Step 5.6'); it is the literal fix, not incidental prose growth, and cannot be avoided without leaving a probe unreachable. Verified: 31987 -> 31997 bytes, DEFAULT tier, cap 40960. #3132: realigned retired covered/backstop-as-status vocab to resolved+verification. #3102: Step 5.5 now RENDERS the edge-probe coverage report into the model's context (a raw printf of $COVERAGE after the well-formedness guard) and binds those rows in the resolution loop and --auto as a deterministic FLOOR, plus the corrected block comment; load-bearing workflow instruction that makes ADR-550 D7b (deterministic propose + LLM resolve) real at runtime and honors ADR-857 sec98's recall gap (floor, not ceiling). 32238 -> 34056 bytes (+1818), DEFAULT tier, cap 40960."
|
||||
"reason": "#2733: five transitions in gsd-core/workflows/spec-phase.md were re-pointed so control reaches the mandatory Step 5.5 edge-completeness and Step 5.6 prohibition-completeness probes, which no path could reach before. Four upstream gate-passed jumps went from 'Jump to Step 6' to 'Jump to Step 5.5', and Step 5.5's own terminal soft gate at :305 went from 'proceed to Step 6' to 'proceed to Step 5.6' so the common all-edges-resolved path stops skipping the prohibition probe. The +10 bytes is exactly those five targets growing by 2 bytes each ('Step 6' -> 'Step 5.5' / 'Step 5.6'); it is the literal fix, not incidental prose growth, and cannot be avoided without leaving a probe unreachable. Verified: 31987 -> 31997 bytes, DEFAULT tier, cap 40960. #3132: realigned retired covered/backstop-as-status vocab to resolved+verification. #3102: Step 5.5 now RENDERS the edge-probe coverage report into the model's context (a raw printf of $COVERAGE after the well-formedness guard) and binds those rows in the resolution loop and --auto as a deterministic FLOOR, plus the corrected block comment; load-bearing workflow instruction that makes ADR-550 D7b (deterministic propose + LLM resolve) real at runtime and honors ADR-857 sec98's recall gap (floor, not ceiling). 32238 -> 34056 bytes (+1818), DEFAULT tier, cap 40960. #2773: Step 5.5 now tells a `response_language` project that the edge-probe `$REQS_JSON` payload is engine input rather than user-facing output, so each requirement's `text` carries a faithful English translation while the SPEC keeps its original language and requirement ids stay unchanged. The shape cues in src/edge-probe.cts are English word-boundary regexes, so prose in another language matched nothing, classified to zero shapes, and put every row in the `unclassified` sentinel (#1110) — the taxonomy contributed nothing. Growth is that instruction plus the pointer to the authored `shapes` override for prose that classifies to zero even in English (measured: the issue's own repro sentence returns [] in English too, so translation is necessary but not sufficient), and a one-line fix an isolated security review surfaced in the same block — the empty/placeholder guard exited without `rm -f \"$REQS_JSON\"` while its engine-failure sibling below it did, stranding the SPEC requirement text in TMPDIR on every failed run. The instruction sits BEFORE the heredoc deliberately: the `$APPLICABLE = 0` warning fires only when every requirement is unclassified, so a partly-classified non-English spec would otherwise slip through with no signal. Load-bearing runtime instruction; the compiled engine is deliberately untouched (the `lang`-hint / per-language cue-set fix is out of scope per the #2773 triage). 34020 -> 35730 bytes (+1710), DEFAULT tier, cap 40960."
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user