* test(#2951): pin the absent-evidence provenance contract (failing first) 17 tests / 22 anchors on the deployed agent text. Measured against the parent commit: 20 anchors fail, 2 pass. The two that pass are the sibling-integrity guards on the package-name and in-repo-value rules -- green before and after is their intended signature. Refs #2951 * enhance(#2951): refuse [VERIFIED] for a compatibility claim resting on absent metadata A claim of the form "X does not support Y" drawn from MISSING metadata -- no python_requires, no engines field, no per-version classifier, no changelog entry, no matching support-matrix row -- no longer earns [VERIFIED] however authoritative the source consulted. An absence is silence about every value, so the same evidence would "prove" both the version being ruled out and the version being standardized on. The only route from an absence to [VERIFIED] is a positive falsification attempt with its failing output pasted; everything short of that is [ASSUMED], which the file already routes to "needs user confirmation before becoming a locked decision". Third member of the family beside the package-name and in-repo-value provenance rules, mirroring PR #2768's shape. A present declared constraint and an affirmatively documented incompatibility are untouched. Closes #2951 * fix(#2951): close the allow-list ambiguity and the mutation gap review found Findings from the isolated adversarial pass and the two-axis review, all fixed: MAJOR (x2, one root cause) -- the absence clause and the present-constraint carve-out gave opposite verdicts on the same evidence for the commonest real case: a classifier list enumerating :: 3.9 through :: 3.13 with no :: 3.14. A researcher could read the enumerated list as a "declared" positive constraint and re-earn [VERIFIED], which is also the evasion vector. The rule now states the decision procedure -- does the declaration bound EVERY value or only the ones it names -- and closes the positive-reframing restatement explicitly. New contract test pins all four clauses. MAJOR -- 'licenses a positive falsification attempt as the route to [VERIFIED]' asserted two independent substrings and never that the route lands on [VERIFIED]. A mutant swapping the tag for [CITED] or [ASSUMED] inverted the rule and survived all 17 tests. Now pinned as one joined sentence. MINOR -- the attributable-failure test regex-matched illustrative examples ("a missing certificate, a wrong host"), so a copy-edit would break it for no reason; relaxed to the substantive clause. The no-paraphrase guard counted only the heading, missing the drift mode in its own name; it now also pins the core proposition to one occurrence, and the test name matches what it checks. An off-by-one in the new allow-list regex bound (141 actual vs 140) is fixed. MINOR -- docs/AGENTS.md listed four of the five governed absence forms while the agent prose, docs/COMMANDS.md and the changeset listed five; three copies disagreeing on list membership is the drift this repo treats as a defect. SCOPE -- removed docs/how-to/verify-a-dependency-compatibility-claim.md and its docs/README.md index line. Both reviewers flagged them as a seventh and eighth surface beyond the six the requester capped, and CONTRIBUTING's "Agent or skill change" row requires only docs/AGENTS.md. The actionable four-case guidance is retained in docs/COMMANDS.md, which is inside the approved scope. Ack byte figures corrected for the final size: 44250 -> 46602 (+2352), 2550 bytes headroom under the LARGE cap of 49152. Refs #2951 * docs(#2951): restore the how-to the phase gate requires Reverses the removal in 6404b43d3. Both /code-review axes had flagged docs/how-to/verify-a-dependency-compatibility-claim.md and its docs/README.md index line as a seventh and eighth surface beyond the six the requester capped, and CONTRIBUTING.md's required-docs row for an "Agent or skill change" names only docs/AGENTS.md, so they were dropped. gsd-phase-gate.cjs then denied gh pr create: it refuses when the recorded enablement sequence has more than one step and the how-to quadrant is empty. The sequence here is genuinely four steps -- run plan-phase, read the [ASSUMED] claim, probe or cite or accept it unlocked, then answer discuss-phase's checkpoint -- and the last step lands on a different capability's surface, so a reference table cannot carry it. Compressing the sequence to one step to unlock howToSkipReason would be gaming the gate, which is the same Goodhart failure this whole change exists to close. A machine-enforced repo gate outranks two reviewers' scope preference and my own reading, so the page is restored and the PR body discloses the two extra surfaces instead of hiding them. Reverting is a one-file change if a maintainer prefers the tighter scope. Refs #2951 * chore(#2951): backfill the changeset PR number pr: 0 -> 3718 now that the real PR exists. The placeholder fails scripts/changeset/lint.cjs with fail_invalid_fragment, which also blocks lint-docs-required from consuming the fragment. Refs #2951 --------- Co-authored-by: sim <sim@local>
This commit is contained in:
Tom Boucher
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"version": 1,
|
||||
"paths": {
|
||||
"gsd-phase-researcher.md": "#3409: guarded `cat \"$phase_dir\"/*-CONTEXT.md` against nullglob wiping the pattern to zero operands when no CONTEXT.md exists \u2014 a bare `cat` with no operands blocks reading stdin (hangs the agent) instead of the `2>/dev/null` guard ever firing, since a stalled read is not a failing exit. Now checks `${_CTX[0]}` is a real path before invoking cat. Growth is the array-guard idiom itself (+43 bytes).",
|
||||
"gsd-phase-researcher.md": "#3409: guarded `cat \"$phase_dir\"/*-CONTEXT.md` against nullglob wiping the pattern to zero operands when no CONTEXT.md exists \u2014 a bare `cat` with no operands blocks reading stdin (hangs the agent) instead of the `2>/dev/null` guard ever firing, since a stalled read is not a failing exit. Now checks `${_CTX[0]}` is a real path before invoking cat. Growth is the array-guard idiom itself (+43 bytes). \u2014 #2951 append (merged into this fragment because two ack sources may never name the same path; the #3409 entry above is at the base and therefore spent, and a second fragment naming this path is a hard failure in scripts/lint-emitted-drift-ack.cjs): +2352 bytes, 44250 -> 46602 (LARGE tier, cap 49152, 2550 bytes headroom). Adds the absent-evidence provenance rule to the claim-provenance section \u2014 the third member of the family alongside the package-name and in-repo-value rules. A compatibility claim resting on MISSING metadata (no python_requires, no engines field, no per-version classifier, no changelog entry, no matching support-matrix row) no longer earns [VERIFIED] however authoritative the source consulted; the only route from an absence to [VERIFIED] is a positive falsification attempt with its failing output pasted, and everything short of that is [ASSUMED], which the file already routes to \"needs user confirmation before becoming a locked decision\". Growth is inline prose in the agent body, deliberately NOT relocated into an eagerly @-imported reference, which ADR-1610 Decision 4 names as gaming the size proxy.",
|
||||
"gsd-verifier.md": "#3409: same nullglob-hang fix as gsd-phase-researcher.md, applied to `cat \"$PHASE_DIR\"/*-VERIFICATION.md` in Step 0 \u2014 an absent VERIFICATION.md previously left a zero-operand `cat` blocking on stdin instead of falling through to first-verification mode. Growth is the array-guard idiom (+49 bytes). \u2014 #3206 append (merged into this fragment because two ack sources may never name the same path): +52 bytes, 49098 -> 49150 (2 under the LARGE cap). The growth is the literal fix for the term 5b used undefined: the compressed explicit-evidence definition inlined at 5b (+34 net on the rewritten line \u2014 the trailing honest-verifier cite there is dropped as superseded by the inline definition; honest-verifier.md stays cited at 5c) plus gsd-core/ path-prefix repairs on the two 404ing bare references/ cites at 5c (honest-verifier.md) and the MVP-mode section (verify-mvp-mode.md) (+9 each). Lazy extraction remains untakeable in this change: the large extractable blocks are content-pinned by tests that read the agent file directly (tests/verifier-behavior-unverified.test.cjs, tests/verification-overrides.test.cjs), so extraction is its own coordinated change.",
|
||||
"complete-milestone.md": "#3409: guarded `cat .planning/phases/*-*/*-SUMMARY.md` \u2014 with `shopt -s nullglob` active in this block's preamble (#2962), zero matching phase summaries collapses the glob to nothing and a bare `cat` blocks reading stdin rather than producing empty output, wedging the milestone-completion review. Growth is the array-existence-check idiom (+73 bytes, two glob segments makes this longer than the single-glob sites). \u2014 #2142 append (merged into this fragment because two ack sources may never name the same path): +1605 bytes, 40498 -> 42103. The `archive_milestone` step now documents the opt-in `--archive-quick` quick-task archival flag (default OFF, deliberately NOT symmetrical with phase archival's default-ON posture), folds the AskUserQuestion decision for it into the SAME `milestone.complete` invocation (avoiding a redundant second call), and states the known bucket-all provenance limit.",
|
||||
"discuss-phase-assumptions.md": "#3409: replaced the unreachable `AUTO_MODE=$(gsd_run query check auto-mode --pick active 2>/dev/null || echo \"false\")` \u2014 `||` never fires because the query exits 0 with empty stdout when the field is absent, not a failure, so AUTO_MODE silently ended up empty rather than \"false\" \u2014 with a two-line capture-then-default (`AUTO_MODE=\"${AUTO_MODE:-false}\"`) that actually reaches the fallback. Growth is the extra default-assignment line (+19 bytes).",
|
||||
|
||||
@@ -627,3 +627,239 @@ describe('gsd-phase-researcher in-repo value provenance rule (#1699)', () => {
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Absent-evidence provenance rule (#2951)
|
||||
//
|
||||
// The two sibling rules above classify WHERE you looked. Neither classifies
|
||||
// whether what you saw supports the claim you drew from it. Consulting PyPI and
|
||||
// finding no `python_requires` and no per-minor classifier is a tool-confirmed
|
||||
// observation from an authoritative source — it earns `[VERIFIED: PyPI]` under a
|
||||
// plain reading of the base taxonomy. The negative conclusion ("ldap3 does not
|
||||
// support 3.14") then rides into a locked CONTEXT.md decision on a tag that was
|
||||
// honestly applied to the lookup. The claim was true of 3.14 and equally true of
|
||||
// 3.12 and 3.13 — the versions the plan was standardizing ON — so the same
|
||||
// evidence "proved" both, and a wrong interpreter downgrade shipped.
|
||||
//
|
||||
// An absence can be verified; the tag did not distinguish a verified absence from
|
||||
// a verified constraint. These assert the governed prose contract on the deployed
|
||||
// agent definition.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('gsd-phase-researcher absent-evidence provenance rule (#2951)', () => {
|
||||
const agentPath = path.join(ROOT, 'agents', 'gsd-phase-researcher.md');
|
||||
const read = () => fs.readFileSync(agentPath, 'utf-8');
|
||||
|
||||
test('names the absent-metadata forms the rule governs', () => {
|
||||
const content = read();
|
||||
for (const term of ['`python_requires`', '`engines` field', 'per-version classifier', 'changelog entry', 'support matrix']) {
|
||||
assert.ok(
|
||||
content.includes(term),
|
||||
`agent must name "${term}" as a governed form of absent metadata`
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test('states that absence is silence about every value, not a constraint on one', () => {
|
||||
const content = read();
|
||||
assert.ok(
|
||||
content.includes('Absence is silence about **every** value, not a constraint on one'),
|
||||
'the core proposition must be stated, not paraphrased'
|
||||
);
|
||||
assert.match(
|
||||
content,
|
||||
/says nothing about the version you want \*and\* nothing about the version you are standardizing on/,
|
||||
'the agent must spell out that the same absence "proves" both sides, which is why the ldap3 claim was worthless'
|
||||
);
|
||||
});
|
||||
|
||||
test('refuses the tag however authoritative the consulted source was', () => {
|
||||
assert.ok(
|
||||
read().includes('however authoritative the source you consulted'),
|
||||
'source authority is what made the original claim pass review — it must be explicitly insufficient'
|
||||
);
|
||||
});
|
||||
|
||||
test("keys on the evidence, not the claim's wording", () => {
|
||||
const content = read();
|
||||
assert.ok(
|
||||
content.includes('keys on the **evidence, not the wording**'),
|
||||
'a rule keyed on surface polarity is evaded by rephrasing'
|
||||
);
|
||||
assert.ok(
|
||||
content.includes('supports only up to 3.13'),
|
||||
'the agent must name the positive rephrasing as resting on the identical absence'
|
||||
);
|
||||
});
|
||||
|
||||
test('rejects absence as evidence of support, not only of non-support', () => {
|
||||
assert.match(
|
||||
read(),
|
||||
/an absence is equally not evidence that the target \*is\* supported/,
|
||||
'the mirror-image error must be closed, or the rule licenses "no upper bound declared, so any version works"'
|
||||
);
|
||||
});
|
||||
|
||||
test('leaves a present declared constraint earning [VERIFIED]', () => {
|
||||
const content = read();
|
||||
assert.ok(
|
||||
content.includes('A **present** constraint is the opposite case and is untouched'),
|
||||
'the rule targets MISSING fields, never merely unfavorable ones'
|
||||
);
|
||||
assert.ok(
|
||||
content.includes('`requires-python = ">=3.9,<3.12"` is a declared exclusion'),
|
||||
'a concrete present-constraint example must show what still earns the tag'
|
||||
);
|
||||
});
|
||||
|
||||
test('leaves an affirmatively documented incompatibility on [CITED]', () => {
|
||||
assert.match(
|
||||
read(),
|
||||
/documentation stating the incompatibility affirmatively \(`\[CITED: …\]`\)/,
|
||||
'an affirmative statement about the world is not an absence and must keep its existing tag'
|
||||
);
|
||||
});
|
||||
|
||||
test('licenses a positive falsification attempt as the route to [VERIFIED]', () => {
|
||||
const content = read();
|
||||
// Pinned as ONE joined sentence, not as independent substrings: the rule's whole
|
||||
// payoff is the TARGET of the route. A mutant swapping `[VERIFIED]` for `[CITED]`
|
||||
// or `[ASSUMED]` inverts the rule while leaving every separate phrase intact.
|
||||
assert.match(
|
||||
content,
|
||||
/The only route from an absence to `\[VERIFIED\]` is a \*\*positive falsification attempt\*\*/,
|
||||
'the route and the tag it reaches must be asserted together, or inverting the tag survives'
|
||||
);
|
||||
assert.ok(
|
||||
content.includes('run it against the real target'),
|
||||
'the probe must exercise the real target, not a proxy'
|
||||
);
|
||||
});
|
||||
|
||||
test('makes the pasted failing output the artifact, not the claim of having run it', () => {
|
||||
const content = read();
|
||||
assert.ok(
|
||||
content.includes('**paste the failing output**'),
|
||||
'the output is the falsifiable artifact — mirrors the in-repo rule making the quote, not the citation, the artifact'
|
||||
);
|
||||
assert.ok(
|
||||
content.includes('asserting that you ran it does not earn the tag'),
|
||||
'an unpasted probe assertion is the box-ticking mode this rule exists to close'
|
||||
);
|
||||
});
|
||||
|
||||
test('rejects a failure not attributable to the incompatibility', () => {
|
||||
assert.match(
|
||||
read(),
|
||||
// The parenthetical examples are illustrative; a copy-edit that swaps them must
|
||||
// not break this. Only the substantive clause is pinned.
|
||||
/a failure attributable to something else[^.]{0,80}is not a falsification/,
|
||||
'any-failure-will-do turns the probe requirement into a formality'
|
||||
);
|
||||
});
|
||||
|
||||
test('refutes rather than downgrades a claim whose probe succeeds', () => {
|
||||
assert.match(
|
||||
read(),
|
||||
/A probe that \*succeeds\* refutes the claim: drop it rather than downgrade it/,
|
||||
'a disproved claim must leave RESEARCH.md entirely, not survive as [ASSUMED]'
|
||||
);
|
||||
});
|
||||
|
||||
test('separates a failed lookup from a declared absence', () => {
|
||||
assert.match(
|
||||
read(),
|
||||
/When the lookup itself failed, report \*no observation\*, never a declared absence/,
|
||||
'an unobserved field must not be laundered into a declared-absent field'
|
||||
);
|
||||
});
|
||||
|
||||
test('keeps [ASSUMED] available when a probe cannot be run', () => {
|
||||
assert.match(
|
||||
read(),
|
||||
/costs a confirmation checkpoint, not a blocked plan/,
|
||||
'a rule that demanded a feasible probe would become a gate that is routinely skipped'
|
||||
);
|
||||
});
|
||||
|
||||
test('sits inside the claim-provenance block ahead of the [ASSUMED] routing sentence', () => {
|
||||
const content = read();
|
||||
const taxonomy = content.indexOf('**Claim provenance:**');
|
||||
const rule = content.indexOf('**Absent-evidence provenance rule:**');
|
||||
const routing = content.indexOf('Claims tagged `[ASSUMED]` signal to the planner and discuss-phase');
|
||||
assert.ok(taxonomy >= 0, 'the base claim-provenance taxonomy must still be present');
|
||||
assert.ok(rule >= 0, 'the absent-evidence rule must be present');
|
||||
assert.ok(routing >= 0, 'the [ASSUMED] routing sentence must still be present');
|
||||
assert.ok(
|
||||
taxonomy < rule && rule < routing,
|
||||
'the rule must follow the taxonomy it constrains and precede the routing sentence it terminates in — '
|
||||
+ 'otherwise "routes into the existing [ASSUMED] path" is not true of the deployed ordering'
|
||||
);
|
||||
});
|
||||
|
||||
test('distinguishes a bounding declaration from an allow-list that stops short', () => {
|
||||
const content = read();
|
||||
// The ldap3 class of case: classifiers present for some versions, absent for the
|
||||
// target. Without this, the absence clause and the present-constraint carve-out
|
||||
// give opposite verdicts on the same evidence and the rule cannot be applied.
|
||||
assert.ok(
|
||||
content.includes('whether the declaration bounds **every** value or only the ones it names'),
|
||||
'the rule must give a decision procedure for present-list-vs-absent-entry, not two conflicting readings'
|
||||
);
|
||||
assert.match(
|
||||
content,
|
||||
/an enumerated allow-list that stops short of your target[\s\S]{0,400}is still a governed absence/,
|
||||
'an allow-list missing the target version must stay a governed absence'
|
||||
);
|
||||
assert.ok(
|
||||
content.includes('unless the project states the list is exhaustive'),
|
||||
'the one condition that turns an allow-list into a real constraint must be named'
|
||||
);
|
||||
assert.match(
|
||||
content,
|
||||
/Reframing that silence as a positive finding[\s\S]{0,120}earns the same tag/,
|
||||
'the positive-reframing evasion must be closed explicitly, not left to inference'
|
||||
);
|
||||
});
|
||||
|
||||
test('does not disturb the package name provenance rule', () => {
|
||||
const content = read();
|
||||
assert.ok(
|
||||
content.includes('**Package name provenance rule:**'),
|
||||
'the first sibling rule must survive unchanged'
|
||||
);
|
||||
assert.ok(
|
||||
content.includes('a slopsquatted package also passes `npm view`'),
|
||||
'package-legitimacy reasoning must remain intact'
|
||||
);
|
||||
});
|
||||
|
||||
test('does not disturb the in-repo value provenance rule', () => {
|
||||
const content = read();
|
||||
assert.ok(
|
||||
content.includes('**In-repo value provenance rule:**'),
|
||||
'the second sibling rule must survive unchanged'
|
||||
);
|
||||
assert.match(
|
||||
content,
|
||||
/opened the source-of-truth file with `Read` \*\*this session\*\*/,
|
||||
'the same-session Read requirement must remain intact'
|
||||
);
|
||||
});
|
||||
|
||||
test('defines the rule and its core proposition once each (META.RULE.brief-no-paraphrase)', () => {
|
||||
const content = read();
|
||||
// Counting the heading alone would miss the drift mode this guard is named for: the
|
||||
// same substance restated under a different heading. Pin the load-bearing sentence too.
|
||||
assert.equal(
|
||||
content.split('Absent-evidence provenance rule').length - 1,
|
||||
1,
|
||||
'the rule must be headed at exactly one site; a second copy is the prose-drift mode'
|
||||
);
|
||||
assert.equal(
|
||||
content.split('Absence is silence about **every** value').length - 1,
|
||||
1,
|
||||
'the core proposition must appear once; a restatement elsewhere is the drift this guards'
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user