golem15/msd-core
4
0
Fork 0
Code Issues 1 Pull Requests Actions 2 Packages Projects Releases Wiki Activity

feat: add gates ensuring discuss-phase decisions are translated to plans and verified (closes #2492) (#2611)

Browse Source 
* feat(#2492): add gates ensuring discuss-phase decisions are translated and verified

Two gates close the loop between CONTEXT.md `<decisions>` and downstream
work, fixing #2492:

- Plan-phase **translation gate** (BLOCKING). After requirements
  coverage, refuses to mark a phase planned when a trackable decision
  is not cited (by id `D-NN` or by 6+-word phrase) in any plan's
  `must_haves`, `truths`, or body. Failure message names each missed
  decision with id, category, text, and remediation paths.

- Verify-phase **validation gate** (NON-BLOCKING). Searches plans,
  SUMMARY.md, files modified, and recent commit subjects for each
  trackable decision. Misses are written to VERIFICATION.md as a
  warning section but do not change verification status. Asymmetry is
  deliberate — fuzzy-match miss should not fail an otherwise green
  phase.

Shared helper `parseDecisions()` lives in `sdk/src/query/decisions.ts`
so #2493 can consume the same parser.

Decisions opt out of both gates via `### Claude's Discretion` heading
or `[informational]` / `[folded]` / `[deferred]` tags.

Both gates skip silently when `workflow.context_coverage_gate=false`
(default `true`).

Closes #2492

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(#2492): make plan-phase decision gate actually block (review F1, F8, F9, F10, F15)

- F1: replace `${context_path}` with `${CONTEXT_PATH}` in the plan-phase
  gate snippet so the BLOCKING gate receives a non-empty path. The
  variable was defined in Step 4 (`CONTEXT_PATH=$(_gsd_field "$INIT" ...)`)
  and the gate snippet referenced the lowercase form, leaving the gate to
  run with an empty path argument and silently skip.
- F15: wrap the SDK call with `jq -e '.data.passed == true' || exit 1` so
  failure halts the workflow instead of being printed and ignored. The
  verify-phase counterpart deliberately keeps no exit-1 (non-blocking by
  design) and now carries an inline note documenting the asymmetry.
- F10: tag the JSON example fence as `json` and the options-list fence as
  `text` (MD040).
- F8/F9: anchor the heading-presence test regexes to `^## 13[a-z]?\\.` so
  prose substrings like "Requirements Coverage Gate" mentioned in body
  text cannot satisfy the assertion. Added two new regression tests
  (variable-name match, exit-1 guard) so a future revert is caught.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(#2492): tighten decision-coverage gates against false positives and config drift (review F3,F4,F5,F6,F7,F16,F18,F19)

- F3: forward `workstream` arg through both gate handlers so workstream-scoped
  `workflow.context_coverage_gate=false` actually skips. Added negative test
  that creates a workstream config disabling the gate while the root config
  has it enabled and asserts the workstream call is skipped.
- F4: restrict the plan-phase haystack to designated sections — front-matter
  `must_haves` / `truths` / `objective` plus body sections under headings
  matching `must_haves|truths|tasks|objective`. HTML comments and fenced
  code blocks are stripped before extraction so a commented-out citation or
  a literal example never counts as coverage. Verify-phase keeps the broader
  artifact-wide haystack by design (non-blocking).
- F5: reject decisions with fewer than 6 normalized words from soft-matching
  (previously only rejected when the resulting phrase was under 12 chars
  AFTER slicing — too lenient). Short decisions now require an explicit
  `D-NN` citation, with regression tests for the boundary.
- F6: walk every `*-SUMMARY.md` independently and use `matchAll` with the
  `/g` flag so multiple `files_modified:` blocks across multiple summaries
  are all aggregated. Previously only the first block in the concatenated
  string was parsed, silently dropping later plans' files.
- F7: validate every `files_modified` path stays inside `projectDir` after
  resolution (rejects absolute paths, `../` traversal). Cap each file read
  at 256 KB. Skipped paths emit a stderr warning naming the entry.
- F16: validate `workflow.context_coverage_gate` is boolean in
  `loadGateConfig`; warn loudly on numeric or other-shaped values and
  default to ON. Mirrors the schema-vs-loadConfig validation gap from
  #2609.
- F18: bump verify-phase `git log -n` cap from 50 to 200 so longer-running
  phases are not undercounted. Documented as a precision-vs-recall tradeoff
  appropriate for a non-blocking gate.
- F19: tighten `QueryResult` / `QueryHandler` to be parameterized
  (`<T = unknown>`). Drops the `as unknown as Record<string, unknown>`
  casts in the gate handlers and surfaces shape mismatches at compile time
  for callers that pass a typed `data` value.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(#2492): harden decisions parser and verify-phase glob (review F11,F12,F13,F14,F17,F20)

- F11: strip fenced code blocks from CONTEXT.md before searching for
  `<decisions>` so an example block inside ``` ``` is not mis-parsed.
- F12: accept tab-indented continuation lines (previously required a leading
  space) so decisions split with `\t` continue cleanly.
- F13: parse EVERY `<decisions>` block in the file via `matchAll`, not just
  the first. CONTEXT.md may legitimately carry more than one block.
- F14: `decisions.parse` handler now resolves a relative path against
  `projectDir` — symmetric with the gate handlers — and still accepts
  absolute paths.
- F17: replace `ls "${PHASE_DIR}"/*-CONTEXT.md | head -1` in verify-phase.md
  with a glob loop (ShellCheck SC2012 fix). Also avoids spawning an extra
  subprocess and survives filenames with whitespace.
- F20: extend the unicode quote-stripping in the discretion-heading match
  to cover U+2018/2019/201A/201B and the U+201C-F double-quote variants
  plus backtick, so any rendering of "Claude's Discretion" collapses to
  the same key.

Each fix has a regression test in `decisions.test.ts`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
2026-04-23 00:26:53 -04:00
committed by GitHub
parent 1a3d953767
commit f30da8326a
16 changed files with 1900 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

519
sdk/src/query/check-decision-coverage.test.ts Normal file
View File

@@ -0,0 +1,519 @@
/**
* Decision-coverage gate tests for issue #2492.
*
* Two gates, two semantics:
*
* - `check.decision-coverage-plan` — translation gate, BLOCKING.
* Each trackable CONTEXT.md decision must appear (by id or text) in at
* least one PLAN.md `must_haves` / `truths` / body.
*
* - `check.decision-coverage-verify` — validation gate, NON-BLOCKING.
* Each trackable decision should appear in shipped artifacts (PLANs,
* SUMMARY.md, files_modified, recent commit messages). Missing items
* are reported as warnings only.
*/
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
import { mkdtemp, writeFile, mkdir, rm } from 'node:fs/promises';
import { join } from 'node:path';
import { tmpdir } from 'node:os';
import {
checkDecisionCoveragePlan,
checkDecisionCoverageVerify,
} from './check-decision-coverage.js';
let tmp: string;
let phaseDir: string;
let contextPath: string;
async function setupPhase(decisionsBlock: string, plans: Record<string, string>, summary?: string) {
await mkdir(phaseDir, { recursive: true });
await writeFile(contextPath, `# Phase 17 Context\n\n${decisionsBlock}\n`, 'utf-8');
for (const [name, content] of Object.entries(plans)) {
await writeFile(join(phaseDir, name), content, 'utf-8');
}
if (summary !== undefined) {
await writeFile(join(phaseDir, '17-SUMMARY.md'), summary, 'utf-8');
}
}
function planFile(mustHavesYaml: string, body = ''): string {
return `---
phase: 17
plan: 1
type: implementation
wave: 1
depends_on: []
files_modified: []
autonomous: true
must_haves:
${mustHavesYaml}
---
${body}
`;
}
beforeEach(async () => {
tmp = await mkdtemp(join(tmpdir(), 'gsd-deccov-'));
phaseDir = join(tmp, '.planning', 'phases', '17-foo');
contextPath = join(phaseDir, '17-CONTEXT.md');
});
afterEach(async () => {
await rm(tmp, { recursive: true, force: true });
});
describe('checkDecisionCoveragePlan — translation gate (#2492)', () => {
it('passes when every trackable decision is cited by id in a plan', async () => {
await setupPhase(
`<decisions>
### Cat
- **D-01:** Use bit offsets
- **D-02:** Display TArray element type
</decisions>`,
{
'17-01-PLAN.md': planFile(
` truths:
- "D-01: bit offsets are exposed via API"
artifacts: []
key_links: []`,
// D-02 cited under a designated `## tasks` heading (review F4).
'## tasks\n- Implements D-02: TArray display logic.\n',
),
},
);
const result = await checkDecisionCoveragePlan([phaseDir, contextPath], tmp);
expect(result.data.passed).toBe(true);
expect(result.data.uncovered).toEqual([]);
expect(result.data.total).toBe(2);
expect(result.data.covered).toBe(2);
});
it('fails when a decision is not covered by any plan and names it', async () => {
await setupPhase(
`<decisions>
### Cat
- **D-01:** Use bit offsets, not byte offsets
- **D-99:** A decision nobody bothered to plan
</decisions>`,
{
'17-01-PLAN.md': planFile(
` truths:
- "D-01: bit offsets are exposed"
artifacts: []
key_links: []`,
),
},
);
const result = await checkDecisionCoveragePlan([phaseDir, contextPath], tmp);
expect(result.data.passed).toBe(false);
expect(result.data.uncovered.map((u: { id: string }) => u.id)).toEqual(['D-99']);
expect(result.data.message).toMatch(/D-99/);
});
it('honors `truths` AND `must_haves` body bullets', async () => {
await setupPhase(
`<decisions>
### Cat
- **D-01:** First decision
- **D-02:** Second decision
</decisions>`,
{
'17-01-PLAN.md': planFile(
` truths:
- "D-01 honored"
artifacts: []
key_links: []`,
'## must_haves\n- D-02: also honored in body\n',
),
},
);
const result = await checkDecisionCoveragePlan([phaseDir, contextPath], tmp);
expect(result.data.passed).toBe(true);
});
it('skips when context_coverage_gate is disabled in config', async () => {
await setupPhase(
`<decisions>
### Cat
- **D-01:** Anything
- **D-02:** Anything else
</decisions>`,
{ '17-01-PLAN.md': planFile(` truths: []\n artifacts: []\n key_links: []`) },
);
await mkdir(join(tmp, '.planning'), { recursive: true });
await writeFile(
join(tmp, '.planning', 'config.json'),
JSON.stringify({ workflow: { context_coverage_gate: false } }),
'utf-8',
);
const result = await checkDecisionCoveragePlan([phaseDir, contextPath], tmp);
expect(result.data.skipped).toBe(true);
expect(result.data.passed).toBe(true);
});
it('skips cleanly when CONTEXT.md is missing', async () => {
await mkdir(phaseDir, { recursive: true });
const result = await checkDecisionCoveragePlan([phaseDir, contextPath], tmp);
expect(result.data.skipped).toBe(true);
expect(result.data.reason).toMatch(/CONTEXT/);
});
it('skips cleanly when <decisions> block is missing', async () => {
await mkdir(phaseDir, { recursive: true });
await writeFile(contextPath, '# Phase 17\n\nNo decisions block here.\n', 'utf-8');
const result = await checkDecisionCoveragePlan([phaseDir, contextPath], tmp);
expect(result.data.skipped).toBe(true);
});
it('does not flag non-trackable decisions (Discretion / informational / folded)', async () => {
await setupPhase(
`<decisions>
### Cat
- **D-01:** trackable
- **D-02 [informational]:** opt-out
- **D-03 [folded]:** opt-out
### Claude's Discretion
- **D-99:** never tracked
</decisions>`,
{
'17-01-PLAN.md': planFile(
` truths:
- "D-01"
artifacts: []
key_links: []`,
),
},
);
const result = await checkDecisionCoveragePlan([phaseDir, contextPath], tmp);
expect(result.data.passed).toBe(true);
expect(result.data.total).toBe(1); // only D-01 is trackable
});
});
describe('checkDecisionCoverageVerify — validation gate (#2492)', () => {
it('reports honored decisions when ID appears in shipped artifacts', async () => {
await setupPhase(
`<decisions>
### Cat
- **D-05:** Validate input
</decisions>`,
{ '17-01-PLAN.md': planFile(` truths: ["D-05"]\n artifacts: []\n key_links: []`) },
'## Summary\nImplemented D-05.\nfiles_modified: []\n',
);
const result = await checkDecisionCoverageVerify([phaseDir, contextPath], tmp);
expect(result.data.honored).toBe(1);
expect(result.data.not_honored).toEqual([]);
expect(result.data.blocking).toBe(false);
});
it('reports decisions not honored when ID appears nowhere', async () => {
await setupPhase(
`<decisions>
### Cat
- **D-50:** Add metrics endpoint
</decisions>`,
{ '17-01-PLAN.md': planFile(` truths: []\n artifacts: []\n key_links: []`) },
'## Summary\nDid other things.\n',
);
const result = await checkDecisionCoverageVerify([phaseDir, contextPath], tmp);
expect(result.data.honored).toBe(0);
expect(result.data.not_honored.map((u: { id: string }) => u.id)).toEqual(['D-50']);
expect(result.data.blocking).toBe(false); // non-blocking by spec
expect(result.data.message).toMatch(/D-50/);
});
it('skips when context_coverage_gate is disabled', async () => {
await setupPhase(
`<decisions>
### Cat
- **D-50:** anything
</decisions>`,
{ '17-01-PLAN.md': planFile(` truths: []\n artifacts: []\n key_links: []`) },
);
await mkdir(join(tmp, '.planning'), { recursive: true });
await writeFile(
join(tmp, '.planning', 'config.json'),
JSON.stringify({ workflow: { context_coverage_gate: false } }),
'utf-8',
);
const result = await checkDecisionCoverageVerify([phaseDir, contextPath], tmp);
expect(result.data.skipped).toBe(true);
expect(result.data.blocking).toBe(false);
});
it('skips cleanly when CONTEXT.md is missing', async () => {
await mkdir(phaseDir, { recursive: true });
const result = await checkDecisionCoverageVerify([phaseDir, contextPath], tmp);
expect(result.data.skipped).toBe(true);
});
});
// ─── Adversarial-review regression tests ──────────────────────────────────
describe('translation gate haystack restriction (review F4)', () => {
it('does NOT count a D-NN citation buried in an HTML comment', async () => {
await setupPhase(
`<decisions>
### Cat
- **D-77:** A trackable decision worth six or more words long
</decisions>`,
{
'17-01-PLAN.md': planFile(
` truths: []\n artifacts: []\n key_links: []`,
'<!-- D-77 was here -->\nNothing else mentions the decision.',
),
},
);
const result = await checkDecisionCoveragePlan([phaseDir, contextPath], tmp);
expect(result.data.passed).toBe(false);
expect(result.data.uncovered.map((u: { id: string }) => u.id)).toContain('D-77');
});
it('does NOT count a D-NN citation buried in a fenced code example', async () => {
await setupPhase(
`<decisions>
### Cat
- **D-78:** A trackable decision worth six or more words long
</decisions>`,
{
'17-01-PLAN.md': planFile(
` truths: []\n artifacts: []\n key_links: []`,
'## Design notes\n\n```text\nExample: D-78 should appear here\n```\n',
),
},
);
const result = await checkDecisionCoveragePlan([phaseDir, contextPath], tmp);
expect(result.data.passed).toBe(false);
expect(result.data.uncovered.map((u: { id: string }) => u.id)).toContain('D-78');
});
it('counts a citation in front-matter `must_haves`', async () => {
await setupPhase(
`<decisions>
### Cat
- **D-79:** Trackable decision text long enough to soft-match.
</decisions>`,
{
'17-01-PLAN.md': `---
phase: 17
plan: 1
must_haves:
- "D-79 must be honored"
truths: []
artifacts: []
key_links: []
---
`,
},
);
const result = await checkDecisionCoveragePlan([phaseDir, contextPath], tmp);
expect(result.data.passed).toBe(true);
});
it('counts a citation in front-matter `truths`', async () => {
await setupPhase(
`<decisions>
### Cat
- **D-80:** Trackable decision text long enough to soft-match.
</decisions>`,
{
'17-01-PLAN.md': planFile(` truths: ["D-80 honored"]\n artifacts: []\n key_links: []`),
},
);
const result = await checkDecisionCoveragePlan([phaseDir, contextPath], tmp);
expect(result.data.passed).toBe(true);
});
});
describe('soft-phrase length gating (review F5)', () => {
it('flags a sub-6-word decision when only the body paraphrases — id citation is required', async () => {
await setupPhase(
// 4 words → cannot soft-match; user must cite the id.
`<decisions>
### Cat
- **D-81:** Use bit offsets always
</decisions>`,
{
'17-01-PLAN.md': planFile(
` truths: ["something else"]\n artifacts: []\n key_links: []`,
// No D-81 citation, paraphrase only.
'## tasks\n- Use bit offsets in storage layer\n',
),
},
);
const result = await checkDecisionCoveragePlan([phaseDir, contextPath], tmp);
expect(result.data.passed).toBe(false);
expect(result.data.uncovered.map((u: { id: string }) => u.id)).toEqual(['D-81']);
});
it('still passes a sub-6-word decision when the id is cited', async () => {
await setupPhase(
`<decisions>
### Cat
- **D-82:** Disable cache
</decisions>`,
{
'17-01-PLAN.md': planFile(` truths: ["D-82"]\n artifacts: []\n key_links: []`),
},
);
const result = await checkDecisionCoveragePlan([phaseDir, contextPath], tmp);
expect(result.data.passed).toBe(true);
});
});
describe('verify-phase summary parsing (review F6, F7)', () => {
it('reads files_modified from EVERY summary, not just the first', async () => {
await mkdir(phaseDir, { recursive: true });
await writeFile(
contextPath,
`# Phase 17 Context
<decisions>
### Cat
- **D-83:** A long-enough trackable decision text for soft matching honored elsewhere.
</decisions>
`,
'utf-8',
);
await writeFile(
join(phaseDir, '17-01-PLAN.md'),
planFile(` truths: []\n artifacts: []\n key_links: []`),
'utf-8',
);
// Summary 01 — no files_modified mentioning D-83.
await writeFile(
join(phaseDir, '17-01-SUMMARY.md'),
'files_modified:\n - "src/unrelated.ts"\n',
'utf-8',
);
// Summary 02 — files_modified entry whose content mentions D-83.
await writeFile(
join(phaseDir, '17-02-SUMMARY.md'),
'files_modified:\n - "src/keeper.ts"\n',
'utf-8',
);
await mkdir(join(tmp, 'src'), { recursive: true });
await writeFile(join(tmp, 'src', 'unrelated.ts'), '// nothing relevant\n', 'utf-8');
await writeFile(join(tmp, 'src', 'keeper.ts'), '// honors D-83 in code\n', 'utf-8');
const result = await checkDecisionCoverageVerify([phaseDir, contextPath], tmp);
// If only the first SUMMARY were parsed, D-83 would be missing.
expect(result.data.honored).toBe(1);
expect(result.data.not_honored).toEqual([]);
});
it('rejects absolute files_modified paths outside projectDir (path traversal guard)', async () => {
await mkdir(phaseDir, { recursive: true });
await writeFile(
contextPath,
`# Phase 17
<decisions>
### Cat
- **D-84:** A trackable decision text spanning enough words to soft-match.
</decisions>
`,
'utf-8',
);
await writeFile(
join(phaseDir, '17-01-PLAN.md'),
planFile(` truths: []\n artifacts: []\n key_links: []`),
'utf-8',
);
// Summary points at /etc/passwd and a parent-traversal path. Both must be skipped.
await writeFile(
join(phaseDir, '17-01-SUMMARY.md'),
'files_modified:\n - "/etc/passwd"\n - "../../../etc/hostname"\n',
'utf-8',
);
const result = await checkDecisionCoverageVerify([phaseDir, contextPath], tmp);
// Should not honor D-84 from those files (and should not throw).
expect(result.data.honored).toBe(0);
expect(result.data.not_honored.map((u: { id: string }) => u.id)).toEqual(['D-84']);
});
});
describe('workstream-aware config (review F3)', () => {
it('honors workstream-scoped context_coverage_gate=false', async () => {
await setupPhase(
`<decisions>
### Cat
- **D-85:** A trackable decision long enough to potentially soft match.
</decisions>`,
{ '17-01-PLAN.md': planFile(` truths: []\n artifacts: []\n key_links: []`) },
);
// Root config does NOT disable the gate.
await mkdir(join(tmp, '.planning'), { recursive: true });
await writeFile(
join(tmp, '.planning', 'config.json'),
JSON.stringify({ workflow: { context_coverage_gate: true } }),
'utf-8',
);
// Workstream config DOES disable it.
await mkdir(join(tmp, '.planning', 'workstreams', 'feat-x'), { recursive: true });
await writeFile(
join(tmp, '.planning', 'workstreams', 'feat-x', 'config.json'),
JSON.stringify({ workflow: { context_coverage_gate: false } }),
'utf-8',
);
// Without workstream → enabled → would fail
const rootResult = await checkDecisionCoveragePlan([phaseDir, contextPath], tmp);
expect(rootResult.data.skipped).toBe(false);
expect(rootResult.data.passed).toBe(false);
// With workstream → workstream config disables → skipped
const wsResult = await checkDecisionCoveragePlan(
[phaseDir, contextPath],
tmp,
'feat-x',
);
expect(wsResult.data.skipped).toBe(true);
expect(wsResult.data.passed).toBe(true);
// Same for verify
const wsVerify = await checkDecisionCoverageVerify(
[phaseDir, contextPath],
tmp,
'feat-x',
);
expect(wsVerify.data.skipped).toBe(true);
});
});
describe('config-type validation (review F16)', () => {
it('warns and defaults to ON when context_coverage_gate is a number', async () => {
await setupPhase(
`<decisions>
### Cat
- **D-86:** A trackable decision text long enough to soft-match.
</decisions>`,
{ '17-01-PLAN.md': planFile(` truths: []\n artifacts: []\n key_links: []`) },
);
await mkdir(join(tmp, '.planning'), { recursive: true });
await writeFile(
join(tmp, '.planning', 'config.json'),
JSON.stringify({ workflow: { context_coverage_gate: 1 } }),
'utf-8',
);
const warnings: string[] = [];
const origWarn = console.warn;
console.warn = (msg: string) => warnings.push(String(msg));
try {
const result = await checkDecisionCoveragePlan([phaseDir, contextPath], tmp);
// Defaulted to ON → not skipped, runs the gate (and fails with uncovered D-86).
expect(result.data.skipped).toBe(false);
expect(result.data.passed).toBe(false);
} finally {
console.warn = origWarn;
}
expect(warnings.some((w) => /context_coverage_gate.*invalid type/.test(w))).toBe(true);
});
});