chore(#2928): productionize the CONTEXT.md predicate fact-store and gate it in CI (#2938)

* feat(#2928): port CONTEXT.md predicate fact-store into the src seam Productionizes the ADR-1671 Option-E reference example as a real module: src/context-predicates.cts (parser + selector + index builder) compiled to gsd-core/bin/lib/, plus scripts/gen-context-index.cjs following the repo's --check/--write drift-guard idiom and wired into lint:generated-sync. Parser behavior is deliberately prototype-equivalent in this commit so the next commit's regression matrix binds to the real defects rather than to a missing module. Two locked design deviations from the prototype: - duplicates carry a count, not line numbers - the committed index carries no line field at all, resolving ADR-1671 open question 4: an artifact without line numbers cannot drift on a line shift, so promoting --check to a CI gate does not make it routinely red Also reconciles the one remaining duplicate predicate ID (RULESET.WORKFLOW_MARKDOWN.FENCES was declared twice; the non-MD040 wording is removed) so the gate can land fail-closed on duplicates. Refs #1671 * test(#2928): failing-first matrix for the predicate fact-store Adds the regression matrix from the phase test plan: parser declaration forms, fence and comment regions, ID/value grammar boundaries at limit-1/limit/limit+1, CRLF fidelity, duplicate detection, the drift-guard CLI, the selector query surface, and four document-shaped fast-check properties. Seven rows are RED for behavioral reasons against the ported parser: indented-bare, star-list, plus-list and numbered-list declaration forms are dropped; a tilde fence and a four-backtick fence containing a shorter fence are not skipped; and a multi-line HTML comment is parsed as live. Eleven selector rows are RED because the query surface is not wired yet. Negative fixtures come from real repo documents that predate the grammar (CONTEXT.md, CONTRIBUTING.md's fenced env-assignment examples) per the fixture-provenance rule, and the property generators are document-shaped rather than seeded from our own serializer. Refs #1671 * fix(#2928): consume the shared fence scanner, relocate the index, wire the selector Drives the failing-first matrix green. Parser: replaces the ported naive triple-backtick toggle with the shared markdown-sectionizer fence engine. scanFencedBlocks and FencedBlockRecord gain an export keyword — the only change to that module, which has 71 upstream dependents — because it already returns line-indexed spans, which is exactly what a line-reporting parser needs. It also already documents itself as the second copy of the fence state machine pending consolidation; adding a third copy here would have been the generative-fix divergence this repo warns about. A parity suite now pins predicate fence-skipping against that scanner across eight fence shapes. HTML-comment skipping stays local because the sectionizer has no comment scanner. Declaration forms widen to indented-bare, star, plus and numbered list items. Index location: docs/CONTEXT-INDEX.json, not a module under bin/lib. The remote matrix run caught the original choice — a committed .cjs there ships ~120KB of CONTEXT.md prose into a runtime module, and two content guards fired truthfully on it (a leaked .claude install path, and four hardcoded package-name literals). Neither guard was allowlisted; the artifact moved instead, mirroring docs/INVENTORY-MANIFEST.json. Nothing at runtime needs to require it — it is a drift-detection artifact, so the selector parses CONTEXT.md live and is always current. Generator: adds a frozen REASON enum and --check --json so the gate's outcome is asserted structurally instead of by matching prose, and --context-path/--index-path so tests drive the real CLI against a temp tree with no filesystem monkeypatching. Selector: gsd_run query context-predicates with --class/--prefix/--contains, structured output carrying a matched count, own-property guards, and no project-root resolution. Registering it exposed that the query dispatch table and the usage string had drifted: a new parity test found 20 routed commands missing from the usage list, all added here rather than deferred. Refs #1671 * test(#2928): lock the newly-public scanFencedBlocks contract Exporting scanFencedBlocks made it public API for the first time, so it needs its own contract test independent of the consumer that motivated the export. Memtrace's co-change analysis flagged the gap: this suite changes together with markdown-sectionizer.cts 8 times in 90 days and was absent from the diff. Covers the documented rules: 0-based indices, -1 for an unterminated fence, the same-char/>=length/no-trailing-text closer rule, a shorter fence inside a longer one staying content, CommonMark 4.5 backtick-in-info-string, and <=3-space indent tolerance. Refs #1671 * fix(#2928): address both isolated review passes Two independent reviewers (correctness axis and security axis, neither the author) found seven findings. All are fixed here with regression tests; none deferred. BLOCKER — comment-blind fence scanning caused silent, permanent predicate loss. The HTML-comment scan and the fence scan ran as two independent passes, and the fence scanner is comment-blind, so a fence delimiter inside an HTML comment with no later close read as an unterminated fence and skipped every remaining line to EOF. Worse, the drift-guard could not catch it: it diffs against a baseline produced by the same corrupted parse. The two constructs now interleave in a single pass so each suppresses the other's boundary detection while active, covered in both directions. The parity suite still binds this scanner to markdown-sectionizer's for comment-free documents, so the two cannot diverge unnoticed. BLOCKER — the selector was not consumed anywhere, leaving the phase's acceptance criterion unmet. Now wired into the pre-work predicate-citation step in contributor-standards, which is the repo's actual brief-assembly path; no code-level brief assembler exists to wire into. MAJOR — ReDoS with an unauthenticated CI-hang exploit. The predicate-id regex nested a dot-containing character class inside a dot-prefixed repeat, so N consecutive dots had exponentially many partitions: 40 dots took 565ms and growth was exponential. CI runs this parser over a pull request's own CONTEXT.md, so any contributor could have hung a shared runner with one line. Replaced with linear per-segment validation. Doubled-dot ids are now rejected; the real document contains none. MAJOR — the duplicate-id gate had only ever been proven on synthetic fixtures. A test now re-inserts the exact line this branch removed and asserts the real generator names it. MAJOR — --check together with --write silently let write win, turning the gate into a writer; a missing path value resolved to the cwd and leaked an EISDIR stack trace. Both are now clean usage errors. MINOR — the hoisted skip-list was exported as a live mutable Set; replaced with a read-only predicate. MINOR — flag-shaped selector values were unmatchable; the inline --flag=value form now provides the escape hatch. Refs #1671 * chore(#2928): backfill changeset PR number 2938 --------- Co-authored-by: sim <sim@local>
2026-07-31 13:17:01 -04:00
parent c043f2946c
commit 05b170e448
22 changed files with 5268 additions and 61 deletions
--- a/tests/markdown-sectionizer.test.cjs
+++ b/tests/markdown-sectionizer.test.cjs
@@ -33,6 +33,7 @@ const fc = require('./helpers/fast-check-setup.cjs');
 const {
  stripFencedCode,
  extractFencedBlock,
+  scanFencedBlocks,
  tokenizeHeadings,
  collectSections,
  collectSection,
@@ -350,6 +351,160 @@ describe('extractFencedBlock', () => {
  });
 });

+// ─── scanFencedBlocks (public export) ─────────────────────────────────────────
+// #2928: `export` was newly added to this function and its `FencedBlockRecord`
+// interface, making it public API for the first time (context-predicates.cts
+// consumes it directly). Hyrum's Law: a newly-public contract needs its own
+// lock-in test, independent of the consumer that motivated exporting it.
+
+describe('scanFencedBlocks (public export)', () => {
+  test('is actually exported from the compiled module as a function', () => {
+    assert.equal(typeof scanFencedBlocks, 'function');
+  });
+
+  test('returns {char, len, infoString, openLineIdx, closeLineIdx} with 0-based line indices', () => {
+    const lines = [
+      'before',
+      '```js',
+      'const x = 1;',
+      '```',
+      'after',
+    ];
+    const blocks = scanFencedBlocks(lines);
+    assert.equal(blocks.length, 1);
+    assert.deepEqual(blocks[0], {
+      char: '`',
+      len: 3,
+      infoString: 'js',
+      openLineIdx: 1,
+      closeLineIdx: 3,
+    });
+  });
+
+  test('closeLineIdx is -1 for an unterminated fence (EOF while open)', () => {
+    const lines = [
+      'before',
+      '```',
+      'body, never closed',
+    ];
+    const blocks = scanFencedBlocks(lines);
+    assert.equal(blocks.length, 1);
+    assert.equal(blocks[0].openLineIdx, 1);
+    assert.equal(blocks[0].closeLineIdx, -1);
+  });
+
+  test('requires >=3 backticks or >=3 tildes to open a fence', () => {
+    assert.deepEqual(scanFencedBlocks(['``', 'not a fence']), []);
+    assert.deepEqual(scanFencedBlocks(['~~', 'not a fence']), []);
+    const backtickBlocks = scanFencedBlocks(['```', 'x', '```']);
+    assert.equal(backtickBlocks.length, 1);
+    assert.equal(backtickBlocks[0].char, '`');
+    const tildeBlocks = scanFencedBlocks(['~~~', 'x', '~~~']);
+    assert.equal(tildeBlocks.length, 1);
+    assert.equal(tildeBlocks[0].char, '~');
+  });
+
+  test('tolerates up to 3 spaces of indent on the opening delimiter', () => {
+    const blocks = scanFencedBlocks(['   ```', 'body', '```']);
+    assert.equal(blocks.length, 1);
+    assert.equal(blocks[0].openLineIdx, 0);
+    assert.equal(blocks[0].closeLineIdx, 2);
+  });
+
+  test('4+ spaces of indent is not recognised as a fence delimiter', () => {
+    const blocks = scanFencedBlocks(['    ```', 'still not a fence']);
+    assert.deepEqual(blocks, []);
+  });
+
+  test('a closer must be the same delimiter char with run length >= the opener, and no trailing text', () => {
+    // Same char, longer run: valid closer.
+    const longerCloser = scanFencedBlocks(['```', 'body', '`````']);
+    assert.equal(longerCloser.length, 1);
+    assert.equal(longerCloser[0].closeLineIdx, 2);
+
+    // Same char, shorter run: not a valid closer -> content, fence stays open (EOF -> -1).
+    const shorterCloser = scanFencedBlocks(['````', 'body', '```']);
+    assert.equal(shorterCloser.length, 1);
+    assert.equal(shorterCloser[0].closeLineIdx, -1);
+  });
+
+  test('mismatched delimiter char while a fence is open is CONTENT, not a boundary — a 3-backtick line inside a 4-backtick fence does not close it', () => {
+    const lines = [
+      '````outer',
+      '```coverage',
+      'nested body',
+      '```',
+      '````',
+    ];
+    const blocks = scanFencedBlocks(lines);
+    assert.equal(blocks.length, 1, 'only the outer 4-backtick fence is a real block');
+    assert.equal(blocks[0].char, '`');
+    assert.equal(blocks[0].len, 4);
+    assert.equal(blocks[0].openLineIdx, 0);
+    assert.equal(blocks[0].closeLineIdx, 4);
+  });
+
+  test('a same-char run that is too short, encountered while open, is content not a closer', () => {
+    const lines = [
+      '````',
+      '```',
+      'still inside',
+      '````',
+    ];
+    const blocks = scanFencedBlocks(lines);
+    assert.equal(blocks.length, 1);
+    assert.equal(blocks[0].openLineIdx, 0);
+    assert.equal(blocks[0].closeLineIdx, 3);
+  });
+
+  test('a same-char, sufficient-length run carrying trailing non-whitespace text, encountered while open, is content not a closer', () => {
+    const lines = [
+      '```',
+      '``` still inside (has trailing text)',
+      '```',
+      'after',
+    ];
+    const blocks = scanFencedBlocks(lines);
+    assert.equal(blocks.length, 1);
+    assert.equal(blocks[0].openLineIdx, 0);
+    assert.equal(blocks[0].closeLineIdx, 2);
+  });
+
+  test('CommonMark §4.5: a backtick fence info string must not contain a backtick — such a line is not a valid opener', () => {
+    const blocks = scanFencedBlocks(['``` has ` a backtick', 'more text']);
+    assert.deepEqual(blocks, [], 'a backtick in the info string means the line is not a valid opener at all');
+  });
+
+  test('infoString is the trimmed trailing text of the opener', () => {
+    const blocks = scanFencedBlocks(['```   js and stuff   ', 'body', '```']);
+    assert.equal(blocks.length, 1);
+    assert.equal(blocks[0].infoString, 'js and stuff');
+  });
+
+  test('multiple sequential blocks in one document are all returned, in order', () => {
+    const lines = [
+      'a',
+      '```',
+      'code1',
+      '```',
+      'b',
+      '~~~py',
+      'code2',
+      '~~~',
+      'c',
+    ];
+    const blocks = scanFencedBlocks(lines);
+    assert.equal(blocks.length, 2);
+    assert.equal(blocks[0].char, '`');
+    assert.equal(blocks[0].openLineIdx, 1);
+    assert.equal(blocks[0].closeLineIdx, 3);
+    assert.equal(blocks[1].char, '~');
+    assert.equal(blocks[1].infoString, 'py');
+    assert.equal(blocks[1].openLineIdx, 5);
+    assert.equal(blocks[1].closeLineIdx, 7);
+  });
+});
+
 // ─── tokenizeHeadings ─────────────────────────────────────────────────────────

 describe('tokenizeHeadings', () => {