fix: recover silently-excluded test dirs + test-architecture audit hardening (#1195)
* fix: recurse test discovery so subdir test suites actually run
scripts/run-tests.cjs discovered tests with a flat readdirSync(testDir),
silently excluding tests/observability/ (4 files), tests/dispatch/ (1) and
tests/installer-migrations/ (1) — 94 passing tests — from `npm test` and all
CI lanes. Walk the tree recursively (relative subpaths preserved), classify
suites by basename, and add a fail-on-zero-executed guard for suite/default
runs (escape hatch GSD_ALLOW_EMPTY_SUITE=1) while preserving the empty
--files/--files-from path the CI inert lane relies on.
Unit suite 735 -> 741 files; surfaces ADR-227's observability/dispatch seam.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: retire 5 verified-worthless tests
Adversarial verification confirmed these 5 prove nothing — their coverage is
provided more strictly elsewhere:
- enh-2790 'has a name: field' spot-checks (command-contract enforces /^gsd[:-]/)
- command-routing-hub duplicate construct + duplicate ERROR_KINDS assertions
- no-cjs-sdk-handsync-tooling (guarded files that never existed on main; bug-190
covers the real retired SDK artifacts)
- runtime-artifact-layout cline edge case (subsumed by the explicit-global test
and bug-782-cline-skills-emission)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: add ADR-218 release version-validation coverage
ADR-218 (reject leading-zero versions like 1.01.0; npm duplicate pre-check) had
zero tests — the logic lived only in release.yml bash. Add a test that extracts
the actual rejection regexes from the workflow and exercises them against a
boundary table (leading-zero/malformed rejected, valid accepted) plus structural
wiring assertions. Goes red if the regex is reverted to [0-9]+.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: redesign weak tests into behavioral, deterministic assertions
Per the ADR test audit, rewrite 27 weak test files (test-only, no source
changes) so each can go red for the defect it guards:
- kill pass-always assert.ok(true) placeholders (research-cli, worktree-baseref,
bug-260 security guard, eslint-rules x24, clusters '|| true')
- replace source-text grep with behavioral calls (install Kilo, sh-hook-paths,
plan-review-convergence) and add a repo-layout governance test
- de-flake real-clock/Math.random coupling (phase last_updated, bug-3707 mtime,
context-utilization property, feat-3594)
- fix independence/shared-state violations (bug-492 singleton, issue-844 tmpRoot,
core reapStaleTempFiles, active-workstream TTY, feat-488 GSD_HOME)
- strengthen property/shape-only tests (research-provider/store classification +
collision) and unconditional plugin.json schema validation (issue-766)
Verified: all 28 files run together 1220 pass / 0 fail / 1 skip.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: add no-tautological-assert lint rule, error in test suite
New custom ESLint rule (eslint-rules/no-tautological-assert.cjs) bans asserts
that can never fail: assert(true)/assert.ok(<always-truthy literal>),
'cond || true' inside an assert, and equality asserts comparing two identical
literals. Wired as error on tests/**; full sweep confirmed zero existing
violations so the suite stays green. Prevents the placeholder-assert regressions
the audit redesigns just removed. RuleTester coverage added (6 valid, 8 invalid).
Note: no-only-tests was already enforced via eslint-plugin-no-only-tests, so no
duplicate rule was added.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: gate new allow-test-rule exemptions to require an issue ref
ADR-456 requires any allow-test-rule exemption added after the ADR to carry a
tracking issue number, but nothing enforced it. New ratchet gate
(scripts/lint-allow-test-rule-refs.cjs, wired into lint:ci) fails when a NEW
allow-test-rule comment lacks a #NNN/URL reference; the 323 existing untracked
exemptions are grandfathered in an allowlist that ratchets down as they gain
refs. Red-green verified (novel untracked offender fails; compliant passes).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: add ADR test-audit evidence report (#1192)
Full risk-first qa-test-architect audit of the ADR portfolio (37 ADRs + 4
platform lenses, adversarial verification of retire verdicts) that drove the
P0 discovery fix, ADR-218 coverage, 5 retires, 27 redesigns, and the two new
lint gates. Filed as point-in-time evidence under docs/issueevidence/, named
for tracking issue #1192.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: replace pre-existing raw NUL byte with escape in feat-3594 fixture
feat-3594's null-byte parser fixture contained a literal NUL byte (pre-existing
on next at b10e5681 — confirmed: base blob has 1 NUL, this fix has 0), which
made git treat the file as binary and would break grep/editors. Switch to the
\x00 escape; the runtime string value (a real NUL in the parser input) is
unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: address adversarial-review findings
Codex adversarial pass over the branch:
- capability-registry drift test no longer mutates the committed generated
capability-registry.cjs in place (concurrency hazard) — uses in-memory
checkPipeline comparison instead.
- allow-test-rule ratchet now detects exemptions in ALL comment forms (block
/* */ too, matching no-source-grep) so a block comment can't bypass it;
one newly-surfaced pre-existing offender grandfathered (323->324).
- install.test Kilo case asserts on what install(false,'kilo') actually writes
rather than manually calling configureKiloPermissions (masked the call site).
- issue-766 drops the undeclared transitive ajv dep for explicit structural
assertions from the schema fixture.
- adr-218 test notes the hotfix leading-zero gap is tracked in #1186.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: address code-review findings (subdir discovery, rule + test gaps)
xhigh code review surfaced 15 confirmed issues, all fixed:
- run-tests.cjs --files now resolves subdir tests by bare basename + handles
Windows backslash paths (ambiguous basenames error clearly).
- affected-tests-lib.cjs listTestFiles made recursive — the targeted CI lane was
silently dropping changed subdir tests (same false-green class the audit fixed).
- no-tautological-assert now catches 'true || cond' and empty []/{} equality.
- verify-test-quality: restore provenance-classification coverage, tighten the
writeFile circular-detection check, guard the module-level file read.
- sh-hook-paths: cover the global-install .sh delegation branch (#2045 guard).
- active-workstream null-guard runs deterministically (no longer skipped on TTY).
- adr-218 structural guards tightened (major/minor leading-zero; needs: membership).
- repo-layout AGENTS.md guard no longer false-alarms on equivalent refactors.
- cross-ai ordering guard fails red when the step is missing.
- issue-766 parses required fields from the schema fixture (auto-enforced).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: stub USERPROFILE alongside HOME in feat-488 (Windows parity)
The feat-488 redesign stubbed process.env.HOME but not USERPROFILE; os.homedir()
resolves from USERPROFILE on Windows, so the home stub was not hermetic there —
caught by windows-test-parity-guard (stubsHomeNoUserProfile). Save/set/restore
USERPROFILE symmetrically with HOME (delete-if-originally-undefined).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: reconcile allow-test-rule allowlist after rebase onto next
Rebasing onto current next pulled in merged PR #1170, which added
inventory-headings-countfree.test.cjs (a baseline allow-test-rule exemption) and
deleted inventory-counts.test.cjs. Grandfather the former and prune the latter so
the ratchet matches the merged tree. No new debt from this PR.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -6,6 +6,11 @@
|
||||
* Asserts structural and semantic correctness of:
|
||||
* .claude-plugin/plugin.json — plugin manifest
|
||||
* hooks/hooks.json — plugin hook wiring
|
||||
*
|
||||
* Section C1 validates plugin.json against the snapshotted schema fixture
|
||||
* (tests/fixtures/plugin-manifest-schema.json) using explicit structural
|
||||
* assertions instead of an Ajv dependency, so this gate runs unconditionally
|
||||
* without requiring ajv in devDependencies.
|
||||
*/
|
||||
|
||||
const { test, describe } = require('node:test');
|
||||
@@ -217,8 +222,138 @@ describe('B: hooks/hooks.json', () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ─── Section C: Optional CLI integration test ─────────────────────────────────
|
||||
describe('C: claude plugin validate (CLI integration)', () => {
|
||||
// ─── Section C: Unconditional JSON schema gate + opportunistic CLI integration ──
|
||||
//
|
||||
// The `claude plugin validate --strict` binary is absent on CI, so Section C was
|
||||
// previously SKIPPED there — the only full-schema gate never ran. This section
|
||||
// replaces the skip-on-absent pattern with two tiers:
|
||||
//
|
||||
// C1 (UNCONDITIONAL) — Validate plugin.json against a snapshotted JSON schema
|
||||
// fixture that captures the fields `--strict` requires. Runs on every
|
||||
// platform, every CI job, every local run. A bug that removes `version`
|
||||
// or changes `name` to an invalid form goes red immediately.
|
||||
//
|
||||
// C2 (OPPORTUNISTIC) — When the `claude` binary IS on PATH, also run
|
||||
// `claude plugin validate . --strict` as an end-to-end smoke test.
|
||||
// This tier provides defence-in-depth for schema changes Claude Code
|
||||
// may introduce that the fixture hasn't yet captured.
|
||||
//
|
||||
describe('C: plugin.json schema validation', () => {
|
||||
|
||||
const SCHEMA_FIXTURE_PATH = path.join(__dirname, 'fixtures', 'plugin-manifest-schema.json');
|
||||
|
||||
// ── C1: Unconditional structural gate ────────────────────────────────────────
|
||||
//
|
||||
// Validates plugin.json against the required fields from the snapshotted
|
||||
// schema fixture (tests/fixtures/plugin-manifest-schema.json) using explicit
|
||||
// structural assertions. This avoids a runtime dependency on `ajv` (which is
|
||||
// only a transitive dep) while providing identical coverage for the fields that
|
||||
// `claude plugin validate --strict` requires.
|
||||
//
|
||||
// Required fields and constraints are derived directly from SCHEMA_FIXTURE_PATH.
|
||||
// If the fixture changes (new required field, new pattern), update this test too.
|
||||
|
||||
test('C1: plugin.json satisfies the snapshotted Claude Code plugin schema (unconditional)', () => {
|
||||
assert.ok(
|
||||
fs.existsSync(SCHEMA_FIXTURE_PATH),
|
||||
`Schema fixture must exist: ${SCHEMA_FIXTURE_PATH}`
|
||||
);
|
||||
assert.ok(
|
||||
fs.existsSync(PLUGIN_JSON_PATH),
|
||||
`.claude-plugin/plugin.json must exist: ${PLUGIN_JSON_PATH}`
|
||||
);
|
||||
|
||||
const manifest = JSON.parse(fs.readFileSync(PLUGIN_JSON_PATH, 'utf-8'));
|
||||
const schema = JSON.parse(fs.readFileSync(SCHEMA_FIXTURE_PATH, 'utf-8'));
|
||||
const errors = [];
|
||||
|
||||
const schemaRequired = Array.isArray(schema.required) ? schema.required : [];
|
||||
const schemaProps = (schema.properties && typeof schema.properties === 'object') ? schema.properties : {};
|
||||
|
||||
// Helper: assert a required field exists with the expected type.
|
||||
function requireField(key, type) {
|
||||
if (!(key in manifest)) {
|
||||
errors.push(`"${key}" is required but missing`);
|
||||
} else if (typeof manifest[key] !== type) {
|
||||
errors.push(`"${key}" must be a ${type}, got ${typeof manifest[key]}`);
|
||||
}
|
||||
}
|
||||
|
||||
// Derive required fields and their types directly from the schema fixture.
|
||||
// Each required field whose "properties" entry has a primitive "type" is
|
||||
// checked via requireField; "object"-typed fields are handled below.
|
||||
for (const key of schemaRequired) {
|
||||
const propDef = schemaProps[key];
|
||||
const fieldType = propDef && propDef.type;
|
||||
if (fieldType === 'object') {
|
||||
// Object fields are validated with deeper checks below.
|
||||
continue;
|
||||
}
|
||||
requireField(key, fieldType || 'string');
|
||||
}
|
||||
|
||||
// Validate "object"-typed required fields from the schema.
|
||||
// For each such field, check existence, type, and any nested "required" sub-fields.
|
||||
for (const key of schemaRequired) {
|
||||
const propDef = schemaProps[key];
|
||||
if (!propDef || propDef.type !== 'object') continue;
|
||||
|
||||
if (!(key in manifest)) {
|
||||
errors.push(`"${key}" is required but missing`);
|
||||
} else if (typeof manifest[key] !== 'object' || manifest[key] === null) {
|
||||
errors.push(`"${key}" must be an object`);
|
||||
} else {
|
||||
// Validate nested required sub-fields declared in the schema.
|
||||
const nestedRequired = Array.isArray(propDef.required) ? propDef.required : [];
|
||||
const nestedProps = (propDef.properties && typeof propDef.properties === 'object') ? propDef.properties : {};
|
||||
for (const subKey of nestedRequired) {
|
||||
const subDef = nestedProps[subKey];
|
||||
const subType = subDef && subDef.type;
|
||||
if (!(subKey in manifest[key])) {
|
||||
errors.push(`"${key}.${subKey}" is required but missing`);
|
||||
} else if (subType && typeof manifest[key][subKey] !== subType) {
|
||||
errors.push(`"${key}.${subKey}" must be a ${subType}, got ${typeof manifest[key][subKey]}`);
|
||||
}
|
||||
// minLength check for nested string sub-fields
|
||||
if (subType === 'string' && subDef.minLength !== undefined) {
|
||||
if (typeof manifest[key][subKey] === 'string' && manifest[key][subKey].length < subDef.minLength) {
|
||||
errors.push(`"${key}.${subKey}" must have minLength ${subDef.minLength}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Derive pattern and minLength constraints from the schema fixture properties.
|
||||
for (const key of schemaRequired) {
|
||||
const propDef = schemaProps[key];
|
||||
if (!propDef || propDef.type === 'object') continue;
|
||||
const value = manifest[key];
|
||||
|
||||
if (propDef.pattern && typeof value === 'string') {
|
||||
const re = new RegExp(propDef.pattern);
|
||||
if (!re.test(value)) {
|
||||
errors.push(`"${key}" must match ${propDef.pattern}, got "${value}"`);
|
||||
}
|
||||
}
|
||||
|
||||
if (propDef.minLength !== undefined && typeof value === 'string') {
|
||||
if (value.length < propDef.minLength) {
|
||||
errors.push(`"${key}" must have minLength ${propDef.minLength}, got length ${value.length}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (errors.length > 0) {
|
||||
assert.fail(
|
||||
`plugin.json fails structural validation against ${path.relative(ROOT, SCHEMA_FIXTURE_PATH)}:\n` +
|
||||
errors.map(e => ` - ${e}`).join('\n') +
|
||||
`\n\nFull manifest:\n${JSON.stringify(manifest, null, 2)}`
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
// ── C2: Opportunistic CLI integration (skipped when claude not on PATH) ──────
|
||||
|
||||
const claudeAvailable = (() => {
|
||||
try {
|
||||
@@ -230,8 +365,8 @@ describe('C: claude plugin validate (CLI integration)', () => {
|
||||
})();
|
||||
|
||||
test(
|
||||
'claude plugin validate . --strict exits 0 (skip if claude not on PATH)',
|
||||
{ skip: !claudeAvailable ? 'claude binary not available on PATH' : false },
|
||||
'C2: claude plugin validate . --strict exits 0 (opportunistic — skip when claude not on PATH)',
|
||||
{ skip: !claudeAvailable ? 'claude binary not on PATH' : false },
|
||||
() => {
|
||||
const result = spawnSync('claude', ['plugin', 'validate', '.', '--strict'], {
|
||||
cwd: ROOT,
|
||||
|
||||
Reference in New Issue
Block a user