* test(#4740): pin the per-step role-family partition Failing-first coverage for the Loop Host Contract role partition. At this commit crossCheckRoleFamilies does not exist, so the rows throw "crossCheckRoleFamilies is not a function" -- the RED proof they bind to behavior rather than restating it. ADR-894 section 3 assigns roles per step but parenthesises the assignment as "(illustrative roles)", and nothing enforced it. The only thing standing in the way was a single deepEqual in this same file, which is editable prose. Rows cover: each step's own family accepted; a strict subset accepted; a foreign role rejected at every step; an unknown role rejected; an unknown step failing CLOSED; capitalization not silently matched; every offending role reported rather than only the first; and purity, because buildContract puts the same array into the generated contract. Two rows exist because an earlier cut of this suite was vacuous. The purity fixture is deliberately UNSORTED -- an alphabetically-sorted fixture cannot fail an in-place sort(), and the mutant was being killed by three unrelated rows instead. A parity row asserts ROLE_FAMILY and ROLE_TO_AGENT cover the exact same role-name domain, both directions: they are parallel constants over one domain, so divergence is the generative-fix class CLAUDE.md names. Every negative row asserts the offending ROLE NAME and the STEP NAME appear in the message. A count-only assertion survives a mutant that reports the wrong role, which the 80% Stryker gate would surface only after a full CI round-trip. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(#4740): reject a cross-family agent-role declaration Orchestration and execution are distinct functions of the loop and must not drift into one another. That partition was real but unenforced: ADR-894 section 3 calls its own role assignment "illustrative", and the generator accepted anything. Adding orchestrator to execute-phase.md's agent-roles line compiled, --check passed once regenerated, and capability-validator.cjs then began accepting into:"orchestrator" at every execute point. ROLE_FAMILY maps every role to one of orchestration, planning or execution. EXPECTED_FAMILY_BY_STEP gives each of the five steps exactly one family. crossCheckRoleFamilies rejects a cross-family role, a role outside the vocabulary, and an unknown step. It reports every offender, not the first. It fails CLOSED on an unknown step, deliberately diverging from assertPointsCoverage's "unknown step -- caught elsewhere". For points that is true: the canonical-set and duplicate checks catch it. For roles there is no second net, so failing open would leave an unknown step as the one input that bypasses the gate. crossCheckRoles' orchestrator exemption is untouched. ROLE_TO_AGENT maps roles to agent FILES and the orchestrator is the host, owning none -- admissibility and agent-file presence are separate concerns with separate checks. Additive to section 3's existing rule that contribution.into must be a member of the step's agentRoles, which is unchanged. That governs what a CAPABILITY may target; this governs what a WORKFLOW may declare. No capability is affected, and all five workflows already declare single-family sets, so the gate is green on the commit that introduces it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(#4740): make the ADR-894 role assignment normative Section 3 parenthesises its per-step role assignment as "(illustrative roles)". That word was accurate about the list's PURPOSE -- it illustrated the shape of a generated contract entry -- and wrong about its STATUS, because the assignment was load-bearing from the moment the generator consumed it. Read literally it makes the partition an example rather than a rule. Appended as a dated in-place section per docs/contributor-standards.md, which records that an accepted ADR is never rewritten and names this the default pattern. Section 3's original body is untouched. The amendment states the three disjoint families, the one family each step admits, that a step may declare a strict subset but never outside it, and why this is a clarification rather than a new decision: the contract is generated from the workflow markers "so it cannot drift into a lie", and all five workflows have always declared single-family sets. What was absent was any statement that it is required, and any check that it holds. It also pins the distinction that is easy to re-merge: contribution.into being a member of agentRoles governs what a CAPABILITY may target and is unchanged; the family rule governs what a WORKFLOW may declare. The CONTEXT.md glossary entry for the Loop Host Contract records the same, beside the agent-reference drift guard it already documented. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(#4740): add changeset fragment pr:0 placeholder is backfilled with the real number once the PR exists. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(#4740): backfill changeset pr number Replaces the pr:0 placeholder with 4742 now that the PR exists. Verified with GITHUB_BASE_REF=next, the way CI runs them: changeset lint and lint:docs both go from invalid_pr(0) to ok. Without that env both report success without evaluating the branch at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#4740): stop injecting the orchestrator procedure into executors claude-orchestration declared a contribution at execute:wave:pre with into:"executor". loop-hook-dispatch.md defines a contribution as "inject fragment.inline verbatim into the context for the role named in into", so its 267 lines were injected into EXECUTOR prompts whenever the capability was enabled. Those lines are orchestration end to end -- construct a wave manifest, resolve the dispatch backend, invoke the Workflow tool to spawn executors, bridge per-agent results into the merge chain. An executor can act on none of it. Retargeting to into:"orchestrator" would not have been a fix. ROLE_TO_AGENT carries no orchestrator entry by design: the orchestrator IS the host, and the host's procedure lives in execute-phase.md. A step's agentRoles enumerates agents a capability may inject context INTO, so adding orchestrator there would model the host as an injectable agent -- the same category error pointed the other way, and it would need an exception carved into the partition the same issue just made normative. So the defect is the mechanism, not the label. A contribution injects into an agent's context; "replace step 3's inline dispatch loop" is a change to what the HOST does. The contribution channel was serving as a host-behaviour directive because it was the only channel available at an execute point. The entry is removed. plan:post into:"planner" is correct and untouched. The procedure is preserved verbatim at docs/workflow-backend-dispatch.md inside the capability -- it is the only copy in the repo -- and is no longer injected anywhere. Consequence, not softened: the Workflow backend now has no loop wiring. Detection, emission and config remain and the design is intact, but nothing dispatches it. Under the separation ADR-1143 itself asserts it never had a legitimate channel; ADR-1143's own audit already records the end-to-end path has never been exercised. Wiring it properly needs a host-level mechanism that does not exist today. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#4740): invert the stale execute:wave:pre registry assertions Removing the contribution left four surfaces asserting or describing the old state. Caught by an isolated review before a verification run was spent, which is the point of reviewing first: the first of these was a guaranteed CI red. execute-wave-post-gate-pipeline-e2e asserted against the REAL generated registry that byLoopPoint['execute:wave:pre'] held exactly one contribution with capId claude-orchestration. It now holds zero. Inverted to assert exactly 0 -- not a vague >= 0 -- and the #2285 comment above it now explains the current state rather than the one it was written for. CONTEXT.md's Claude Orchestration entry claimed two contributions at wired points. It is now one, and the entry's execute:wave:post label was already wrong before this change: the manifest said execute:wave:pre. Rewritten to one plan:post contribution, why the execute-point one was removed, and where the procedure now lives. One assertion in claude-orchestration.test.cjs could not fail. It tested for the prose "(into the executor)" while the doc says "(`into: executor`)", so no plausible wording matched it and the paired plan:post assertion was carrying the row. Replaced with a check on the structural claim, and proved RED by restoring the two-contribution wording before reverting. The moved procedure keeps section headings that speak as a live contribution -- "When this contribution is active", "Why execute:wave:pre". Preserving the body verbatim was deliberate, so the headings stay and an editor's note under the header explains why they read that way. A sweep of all 17 files referencing byLoopPoint found no further siblings: the remaining hits are a synthetic capability fixture and an empty-points test that already expected no active hooks, both correct before and after. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: sim <sim@local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
@@ -13,12 +13,14 @@ const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const os = require('node:os');
|
||||
const path = require('node:path');
|
||||
const fc = require('fast-check');
|
||||
|
||||
const { cleanup } = require('./helpers.cjs');
|
||||
|
||||
const {
|
||||
parseLoopHostBlock,
|
||||
crossCheckRoles,
|
||||
crossCheckRoleFamilies,
|
||||
assertPointsCoverage,
|
||||
buildContract,
|
||||
serializeContract,
|
||||
@@ -27,6 +29,7 @@ const {
|
||||
CANONICAL_POINTS,
|
||||
EXPECTED_POINTS_BY_STEP,
|
||||
ROLE_TO_AGENT,
|
||||
ROLE_FAMILY,
|
||||
} = require('../scripts/gen-loop-host-contract.cjs');
|
||||
|
||||
const { LOOP_HOST_CONTRACT } = require('../gsd-core/bin/lib/loop-host-contract.cjs');
|
||||
@@ -202,6 +205,158 @@ describe('crossCheckRoles', () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ─── 2b. crossCheckRoleFamilies (#4740) ──────────────────────────────────────
|
||||
|
||||
describe('crossCheckRoleFamilies (#4740)', () => {
|
||||
test('accepts the execution family at execute', () => {
|
||||
const errors = crossCheckRoleFamilies('execute', ['executor', 'verifier'], 'execute-phase.md');
|
||||
assert.deepEqual(errors, []);
|
||||
});
|
||||
|
||||
test('accepts orchestrator at discuss', () => {
|
||||
const errors = crossCheckRoleFamilies('discuss', ['orchestrator'], 'discuss-phase.md');
|
||||
assert.deepEqual(errors, []);
|
||||
});
|
||||
|
||||
test('accepts the planning family at plan', () => {
|
||||
const errors = crossCheckRoleFamilies('plan', ['researcher', 'planner', 'checker'], 'plan-phase.md');
|
||||
assert.deepEqual(errors, []);
|
||||
});
|
||||
|
||||
test('accepts orchestrator at verify and ship', () => {
|
||||
assert.deepEqual(crossCheckRoleFamilies('verify', ['orchestrator'], 'verify-work.md'), []);
|
||||
assert.deepEqual(crossCheckRoleFamilies('ship', ['orchestrator'], 'ship.md'), []);
|
||||
});
|
||||
|
||||
test('a strict subset of the family is legal', () => {
|
||||
const errors = crossCheckRoleFamilies('execute', ['executor'], 'execute-phase.md');
|
||||
assert.deepEqual(errors, []);
|
||||
});
|
||||
|
||||
test('rejects orchestrator added to execute', () => {
|
||||
const errors = crossCheckRoleFamilies('execute', ['executor', 'verifier', 'orchestrator'], 'execute-phase.md');
|
||||
assert.strictEqual(errors.length, 1, 'expected exactly 1 error');
|
||||
assert.match(errors[0], /orchestrator/);
|
||||
assert.match(errors[0], /execute/);
|
||||
});
|
||||
|
||||
test('rejects an execution role at an orchestration step', () => {
|
||||
const errors = crossCheckRoleFamilies('discuss', ['orchestrator', 'executor'], 'discuss-phase.md');
|
||||
assert.strictEqual(errors.length, 1, 'expected exactly 1 error');
|
||||
assert.match(errors[0], /executor/);
|
||||
});
|
||||
|
||||
test('rejects an execution role at plan', () => {
|
||||
const errors = crossCheckRoleFamilies('plan', ['planner', 'executor'], 'plan-phase.md');
|
||||
assert.strictEqual(errors.length, 1, 'expected exactly 1 error');
|
||||
assert.match(errors[0], /executor/);
|
||||
});
|
||||
|
||||
test('rejects a planning role at execute', () => {
|
||||
const errors = crossCheckRoleFamilies('execute', ['executor', 'planner'], 'execute-phase.md');
|
||||
assert.strictEqual(errors.length, 1, 'expected exactly 1 error');
|
||||
assert.match(errors[0], /planner/);
|
||||
});
|
||||
|
||||
test('rejects a role with no family', () => {
|
||||
const errors = crossCheckRoleFamilies('execute', ['executer'], 'execute-phase.md');
|
||||
assert.ok(errors.length >= 1, 'expected at least 1 error');
|
||||
assert.ok(errors.some((e) => e.includes('executer')));
|
||||
});
|
||||
|
||||
test('fails closed on an unknown step', () => {
|
||||
const errors = crossCheckRoleFamilies('audit', ['orchestrator'], 'audit.md');
|
||||
assert.strictEqual(errors.length, 1, 'expected exactly 1 error for unknown step');
|
||||
assert.match(errors[0], /audit/);
|
||||
});
|
||||
|
||||
test('rejects a role whose capitalization does not match', () => {
|
||||
const errors = crossCheckRoleFamilies('execute', ['Orchestrator'], 'execute-phase.md');
|
||||
assert.ok(errors.length >= 1, 'capitalized role must not silently match');
|
||||
assert.ok(errors.some((e) => e.includes('Orchestrator')));
|
||||
});
|
||||
|
||||
test('a duplicated in-family role is not an error', () => {
|
||||
const errors = crossCheckRoleFamilies('execute', ['executor', 'executor'], 'execute-phase.md');
|
||||
assert.deepEqual(errors, []);
|
||||
});
|
||||
|
||||
test('reports every offending role, not just the first', () => {
|
||||
const errors = crossCheckRoleFamilies('execute', ['executor', 'orchestrator', 'planner'], 'execute-phase.md');
|
||||
assert.strictEqual(errors.length, 2, 'expected one error per offending role');
|
||||
const combined = errors.join('\n');
|
||||
assert.ok(combined.includes('orchestrator'));
|
||||
assert.ok(combined.includes('planner'));
|
||||
});
|
||||
|
||||
test('is pure and does not mutate its arguments', () => {
|
||||
// Deliberately NOT alphabetically sorted — an in-place agentRoles.sort()
|
||||
// must be caught by THIS test, not merely coincide with already-sorted
|
||||
// input (#4740 review finding).
|
||||
const agentRoles = ['verifier', 'executor', 'orchestrator'];
|
||||
const snapshot = agentRoles.slice();
|
||||
const first = crossCheckRoleFamilies('execute', agentRoles, 'execute-phase.md');
|
||||
assert.deepEqual(agentRoles, snapshot, 'input array must not be mutated');
|
||||
const second = crossCheckRoleFamilies('execute', agentRoles, 'execute-phase.md');
|
||||
assert.deepEqual(first, second, 'repeated calls must produce identical results');
|
||||
assert.deepEqual(agentRoles, snapshot, 'input array must still not be mutated after second call');
|
||||
});
|
||||
|
||||
test('ROLE_FAMILY and ROLE_TO_AGENT cover the exact same role-name domain (#4740)', () => {
|
||||
// Generative-fix-divergence guard: ROLE_FAMILY and ROLE_TO_AGENT are parallel
|
||||
// constants over the same role-name domain (every non-orchestrator role, plus
|
||||
// orchestrator which is family-only). Exact set equality both directions —
|
||||
// not a subset check — so adding a role to only one map is caught here.
|
||||
const familyKeys = new Set(Object.keys(ROLE_FAMILY));
|
||||
const expectedKeys = new Set([...Object.keys(ROLE_TO_AGENT), 'orchestrator']);
|
||||
assert.deepEqual(
|
||||
[...familyKeys].sort(),
|
||||
[...expectedKeys].sort(),
|
||||
'ROLE_FAMILY keys must equal ROLE_TO_AGENT keys plus "orchestrator", exactly',
|
||||
);
|
||||
});
|
||||
|
||||
test('property: in-family subsets pass, any foreign role fails', () => {
|
||||
const ROLE_FAMILY_LOCAL = {
|
||||
orchestrator: 'orchestration',
|
||||
researcher: 'planning', planner: 'planning', checker: 'planning',
|
||||
executor: 'execution', verifier: 'execution',
|
||||
};
|
||||
const familyRoles = {
|
||||
orchestration: ['orchestrator'],
|
||||
planning: ['researcher', 'planner', 'checker'],
|
||||
execution: ['executor', 'verifier'],
|
||||
};
|
||||
const stepsByFamily = {
|
||||
discuss: 'orchestration', plan: 'planning', execute: 'execution',
|
||||
verify: 'orchestration', ship: 'orchestration',
|
||||
};
|
||||
const steps = Object.keys(stepsByFamily);
|
||||
|
||||
fc.assert(
|
||||
fc.property(
|
||||
fc.constantFrom(...steps),
|
||||
fc.subarray(Object.keys(ROLE_FAMILY_LOCAL), { minLength: 1 }),
|
||||
(step, roles) => {
|
||||
const family = stepsByFamily[step];
|
||||
const inFamilySubset = familyRoles[family];
|
||||
const errors = crossCheckRoleFamilies(step, inFamilySubset, 'x.md');
|
||||
assert.deepEqual(errors, []);
|
||||
|
||||
const hasForeign = roles.some((r) => ROLE_FAMILY_LOCAL[r] !== family);
|
||||
const errors2 = crossCheckRoleFamilies(step, roles, 'x.md');
|
||||
if (hasForeign) {
|
||||
assert.ok(errors2.length >= 1);
|
||||
} else {
|
||||
assert.deepEqual(errors2, []);
|
||||
}
|
||||
},
|
||||
),
|
||||
{ seed: 4740, numRuns: 200 },
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
// ─── 3. assertPointsCoverage ─────────────────────────────────────────────────
|
||||
|
||||
describe('assertPointsCoverage', () => {
|
||||
@@ -333,6 +488,19 @@ describe('buildContract from real workflows', () => {
|
||||
assert.deepEqual(ship.coreArtifacts.produces, []);
|
||||
assert.deepEqual(ship.coreArtifacts.consumes, ['UAT.md']);
|
||||
});
|
||||
|
||||
test('the shipped workflows declare no cross-family role (#4740)', () => {
|
||||
// Pins something real about THIS change, distinct from the per-step field
|
||||
// assertions above: running crossCheckRoleFamilies directly against every
|
||||
// entry of the REAL contract returns zero errors for every step, not merely
|
||||
// that buildContract() didn't throw (which every other test in this describe
|
||||
// block already exercises incidentally).
|
||||
const contract = buildContract();
|
||||
for (const entry of contract) {
|
||||
const errors = crossCheckRoleFamilies(entry.step, entry.agentRoles, entry.step + '.md');
|
||||
assert.deepEqual(errors, [], 'step "' + entry.step + '" must declare no cross-family role');
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// ─── 5. cross-check rejects nonexistent agent-role ───────────────────────────
|
||||
|
||||
Reference in New Issue
Block a user