golem15/msd-core
4
0
Fork 0
Code Issues 1 Pull Requests Actions 2 Packages Projects Releases Wiki Activity

fix: recover silently-excluded test dirs + test-architecture audit hardening (#1195)

Browse Source 
* fix: recurse test discovery so subdir test suites actually run

scripts/run-tests.cjs discovered tests with a flat readdirSync(testDir),
silently excluding tests/observability/ (4 files), tests/dispatch/ (1) and
tests/installer-migrations/ (1) — 94 passing tests — from `npm test` and all
CI lanes. Walk the tree recursively (relative subpaths preserved), classify
suites by basename, and add a fail-on-zero-executed guard for suite/default
runs (escape hatch GSD_ALLOW_EMPTY_SUITE=1) while preserving the empty
--files/--files-from path the CI inert lane relies on.

Unit suite 735 -> 741 files; surfaces ADR-227's observability/dispatch seam.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test: retire 5 verified-worthless tests

Adversarial verification confirmed these 5 prove nothing — their coverage is
provided more strictly elsewhere:
- enh-2790 'has a name: field' spot-checks (command-contract enforces /^gsd[:-]/)
- command-routing-hub duplicate construct + duplicate ERROR_KINDS assertions
- no-cjs-sdk-handsync-tooling (guarded files that never existed on main; bug-190
  covers the real retired SDK artifacts)
- runtime-artifact-layout cline edge case (subsumed by the explicit-global test
  and bug-782-cline-skills-emission)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test: add ADR-218 release version-validation coverage

ADR-218 (reject leading-zero versions like 1.01.0; npm duplicate pre-check) had
zero tests — the logic lived only in release.yml bash. Add a test that extracts
the actual rejection regexes from the workflow and exercises them against a
boundary table (leading-zero/malformed rejected, valid accepted) plus structural
wiring assertions. Goes red if the regex is reverted to [0-9]+.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test: redesign weak tests into behavioral, deterministic assertions

Per the ADR test audit, rewrite 27 weak test files (test-only, no source
changes) so each can go red for the defect it guards:
- kill pass-always assert.ok(true) placeholders (research-cli, worktree-baseref,
  bug-260 security guard, eslint-rules x24, clusters '|| true')
- replace source-text grep with behavioral calls (install Kilo, sh-hook-paths,
  plan-review-convergence) and add a repo-layout governance test
- de-flake real-clock/Math.random coupling (phase last_updated, bug-3707 mtime,
  context-utilization property, feat-3594)
- fix independence/shared-state violations (bug-492 singleton, issue-844 tmpRoot,
  core reapStaleTempFiles, active-workstream TTY, feat-488 GSD_HOME)
- strengthen property/shape-only tests (research-provider/store classification +
  collision) and unconditional plugin.json schema validation (issue-766)

Verified: all 28 files run together 1220 pass / 0 fail / 1 skip.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: add no-tautological-assert lint rule, error in test suite

New custom ESLint rule (eslint-rules/no-tautological-assert.cjs) bans asserts
that can never fail: assert(true)/assert.ok(<always-truthy literal>),
'cond || true' inside an assert, and equality asserts comparing two identical
literals. Wired as error on tests/**; full sweep confirmed zero existing
violations so the suite stays green. Prevents the placeholder-assert regressions
the audit redesigns just removed. RuleTester coverage added (6 valid, 8 invalid).

Note: no-only-tests was already enforced via eslint-plugin-no-only-tests, so no
duplicate rule was added.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: gate new allow-test-rule exemptions to require an issue ref

ADR-456 requires any allow-test-rule exemption added after the ADR to carry a
tracking issue number, but nothing enforced it. New ratchet gate
(scripts/lint-allow-test-rule-refs.cjs, wired into lint:ci) fails when a NEW
allow-test-rule comment lacks a #NNN/URL reference; the 323 existing untracked
exemptions are grandfathered in an allowlist that ratchets down as they gain
refs. Red-green verified (novel untracked offender fails; compliant passes).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: add ADR test-audit evidence report (#1192)

Full risk-first qa-test-architect audit of the ADR portfolio (37 ADRs + 4
platform lenses, adversarial verification of retire verdicts) that drove the
P0 discovery fix, ADR-218 coverage, 5 retires, 27 redesigns, and the two new
lint gates. Filed as point-in-time evidence under docs/issueevidence/, named
for tracking issue #1192.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test: replace pre-existing raw NUL byte with escape in feat-3594 fixture

feat-3594's null-byte parser fixture contained a literal NUL byte (pre-existing
on next at b10e5681 — confirmed: base blob has 1 NUL, this fix has 0), which
made git treat the file as binary and would break grep/editors. Switch to the
\x00 escape; the runtime string value (a real NUL in the parser input) is
unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test: address adversarial-review findings

Codex adversarial pass over the branch:
- capability-registry drift test no longer mutates the committed generated
  capability-registry.cjs in place (concurrency hazard) — uses in-memory
  checkPipeline comparison instead.
- allow-test-rule ratchet now detects exemptions in ALL comment forms (block
  /* */ too, matching no-source-grep) so a block comment can't bypass it;
  one newly-surfaced pre-existing offender grandfathered (323->324).
- install.test Kilo case asserts on what install(false,'kilo') actually writes
  rather than manually calling configureKiloPermissions (masked the call site).
- issue-766 drops the undeclared transitive ajv dep for explicit structural
  assertions from the schema fixture.
- adr-218 test notes the hotfix leading-zero gap is tracked in #1186.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: address code-review findings (subdir discovery, rule + test gaps)

xhigh code review surfaced 15 confirmed issues, all fixed:
- run-tests.cjs --files now resolves subdir tests by bare basename + handles
  Windows backslash paths (ambiguous basenames error clearly).
- affected-tests-lib.cjs listTestFiles made recursive — the targeted CI lane was
  silently dropping changed subdir tests (same false-green class the audit fixed).
- no-tautological-assert now catches 'true || cond' and empty []/{}  equality.
- verify-test-quality: restore provenance-classification coverage, tighten the
  writeFile circular-detection check, guard the module-level file read.
- sh-hook-paths: cover the global-install .sh delegation branch (#2045 guard).
- active-workstream null-guard runs deterministically (no longer skipped on TTY).
- adr-218 structural guards tightened (major/minor leading-zero; needs: membership).
- repo-layout AGENTS.md guard no longer false-alarms on equivalent refactors.
- cross-ai ordering guard fails red when the step is missing.
- issue-766 parses required fields from the schema fixture (auto-enforced).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test: stub USERPROFILE alongside HOME in feat-488 (Windows parity)

The feat-488 redesign stubbed process.env.HOME but not USERPROFILE; os.homedir()
resolves from USERPROFILE on Windows, so the home stub was not hermetic there —
caught by windows-test-parity-guard (stubsHomeNoUserProfile). Save/set/restore
USERPROFILE symmetrically with HOME (delete-if-originally-undefined).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: reconcile allow-test-rule allowlist after rebase onto next

Rebasing onto current next pulled in merged PR #1170, which added
inventory-headings-countfree.test.cjs (a baseline allow-test-rule exemption) and
deleted inventory-counts.test.cjs. Grandfather the former and prune the latter so
the ratchet matches the merged tree. No new debt from this PR.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
2026-06-13 23:35:08 -04:00
committed by GitHub
parent 3ebd13c45d
commit 5fa4dcd78c
44 changed files with 3958 additions and 757 deletions
Download Patch File Download Diff File Expand all files Collapse all files

441
tests/verify-test-quality.test.cjs
View File

@@ -1,209 +1,312 @@
// allow-test-rule: source-text-is-the-product
// Tests write synthetic fixture files and apply regex detectors to them.
// The fixture text IS the product being tested (testing linter/detector logic,
// not GSD command JSON output). Migrated from pending-migration-to-typed-ir per #455.
// Structural guard: reads gsd-core/workflows/verify-phase.md and asserts that
// the audit_test_quality step contains the skip-pattern marker, circular-detection
// marker, provenance-classification contract, and assertion-strength table markers.
// Goes red if that workflow guidance is removed or the step is renamed/deleted.
/**
* Tests for the audit_test_quality step in verify-phase.md
*
* Validates that the verifier's test quality audit detects:
* - Disabled tests (it.skip) covering requirements
* - Circular tests (system generating its own expected values)
* - Weak assertions on requirement-linked tests
*/
'use strict';
const { describe, test, beforeEach, afterEach } = require('node:test');
const { describe, test, before } = require('node:test');
const assert = require('node:assert/strict');
const fs = require('fs');
const path = require('path');
const { createTempProject, cleanup } = require('./helpers.cjs');
describe('audit_test_quality step', () => {
let tmpDir;
const WORKFLOW_PATH = path.join(
__dirname,
'..',
'gsd-core',
'workflows',
'verify-phase.md'
);
beforeEach(() => {
tmpDir = createTempProject();
// Locate the audit_test_quality step boundaries so sub-assertions are scoped
// to that step only, not the full file.
const STEP_OPEN = '<step name="audit_test_quality">';
const STEP_CLOSE = '</step>';
function extractAuditStep(src) {
const start = src.indexOf(STEP_OPEN);
if (start === -1) return null;
const end = src.indexOf(STEP_CLOSE, start + STEP_OPEN.length);
if (end === -1) return null;
return src.slice(start, end + STEP_CLOSE.length);
}
// workflowSrc and auditStepSrc are populated in the before() hook so that a
// missing or renamed verify-phase.md produces a descriptive test FAILURE rather
// than a module-load crash that prevents any test from registering.
let workflowSrc = null;
let auditStepSrc = null;
before(() => {
assert.ok(
fs.existsSync(WORKFLOW_PATH),
`verify-phase.md not found at expected path: ${WORKFLOW_PATH} — ` +
'the file may have been renamed or moved'
);
workflowSrc = fs.readFileSync(WORKFLOW_PATH, 'utf8');
auditStepSrc = extractAuditStep(workflowSrc);
});
describe('verify-phase.md audit_test_quality structural guard', () => {
test('verify-phase.md exists at gsd-core/workflows/verify-phase.md', () => {
assert.ok(
fs.existsSync(WORKFLOW_PATH),
`missing workflow file: ${WORKFLOW_PATH}`
);
});
afterEach(() => {
cleanup(tmpDir);
test('audit_test_quality step is present in verify-phase.md', () => {
assert.ok(
auditStepSrc !== null,
`<step name="audit_test_quality"> not found in ${WORKFLOW_PATH} — the step ` +
'may have been renamed or removed'
);
});
describe('disabled test detection', () => {
test('detects it.skip in test files', () => {
const testDir = path.join(tmpDir, 'tests');
fs.mkdirSync(testDir, { recursive: true });
fs.writeFileSync(path.join(testDir, 'parity.test.js'), [
'describe("parity", () => {',
' it.skip("matches PHP output", async () => {',
' expect(result).toBeCloseTo(155.96, 2);',
' });',
'});',
].join('\n'));
const content = fs.readFileSync(path.join(testDir, 'parity.test.js'), 'utf8');
const skipPatterns = /it\.skip|describe\.skip|test\.skip|xit\(|xdescribe\(|xtest\(/g;
const matches = content.match(skipPatterns);
assert.ok(matches, 'Should detect skip patterns');
assert.strictEqual(matches.length, 1);
describe('skip-pattern marker', () => {
test('audit_test_quality step contains the disabled-test grep pattern', () => {
// The step must instruct the verifier to search for skip patterns such as
// it\.skip / describe\.skip / test\.skip (regex-escaped, as used in the bash grep).
// Removing this guidance would mean skipped requirement tests are no longer flagged.
assert.ok(
auditStepSrc !== null,
'Cannot check skip-pattern marker: audit_test_quality step not found'
);
// The markdown shows a bash grep -E pattern, so dots are backslash-escaped:
// 'it\\.skip' in JS is the string it\.skip (backslash + dot).
const hasSkipPattern =
auditStepSrc.includes('it\\.skip') &&
auditStepSrc.includes('describe\\.skip') &&
auditStepSrc.includes('test\\.skip');
assert.ok(
hasSkipPattern,
'audit_test_quality step must reference it\\.skip, describe\\.skip, and test\\.skip ' +
'as the disabled-test grep pattern — one or more are missing'
);
});
test('detects multiple skip patterns across frameworks', () => {
const testDir = path.join(tmpDir, 'tests');
fs.mkdirSync(testDir, { recursive: true });
fs.writeFileSync(path.join(testDir, 'multi.test.js'), [
'describe.skip("suite", () => {});',
'xit("old jasmine", () => {});',
'test.skip("jest skip", () => {});',
'it.todo("not implemented");',
].join('\n'));
const content = fs.readFileSync(path.join(testDir, 'multi.test.js'), 'utf8');
const skipPatterns = /it\.skip|describe\.skip|test\.skip|xit\(|xdescribe\(|xtest\(|it\.todo|test\.todo/g;
const matches = content.match(skipPatterns);
assert.ok(matches, 'Should detect all skip variants');
assert.strictEqual(matches.length, 4);
});
test('does not flag active tests as skipped', () => {
const testDir = path.join(tmpDir, 'tests');
fs.mkdirSync(testDir, { recursive: true });
fs.writeFileSync(path.join(testDir, 'active.test.js'), [
'describe("active suite", () => {',
' it("does the thing", () => {',
' expect(result).toBe(true);',
' });',
' test("also works", () => {',
' expect(other).toBe(42);',
' });',
'});',
].join('\n'));
const content = fs.readFileSync(path.join(testDir, 'active.test.js'), 'utf8');
const skipPatterns = /it\.skip|describe\.skip|test\.skip|xit\(|xdescribe\(|xtest\(|it\.todo|test\.todo/g;
const matches = content.match(skipPatterns);
assert.strictEqual(matches, null, 'Active tests should not match skip patterns');
test('audit_test_quality step references todo variants alongside skip variants', () => {
// it\.todo / test\.todo are also considered disabled patterns by the step.
assert.ok(
auditStepSrc !== null,
'Cannot check todo marker: audit_test_quality step not found'
);
const hasTodo =
auditStepSrc.includes('it\\.todo') || auditStepSrc.includes('test\\.todo');
assert.ok(
hasTodo,
'audit_test_quality step must reference it\\.todo or test\\.todo as a disabled pattern'
);
});
});
describe('circular test detection', () => {
test('detects script that imports system-under-test and writes fixtures', () => {
const testDir = path.join(tmpDir, 'tests');
fs.mkdirSync(testDir, { recursive: true });
fs.writeFileSync(path.join(testDir, 'captureBaseline.js'), [
'import { CalculationService } from "../server/services/calculationService.js";',
'import { writeFileSync } from "fs";',
'',
'const result = await CalculationService.execute(input);',
'fixture.expectedOutput = result.value;',
'writeFileSync("fixtures/data.json", JSON.stringify(fixture));',
].join('\n'));
const content = fs.readFileSync(path.join(testDir, 'captureBaseline.js'), 'utf8');
const importsSystem = /import.*(?:Service|Engine|Calculator|Controller)/.test(content);
const writesFiles = /writeFileSync|writeFile|fs\.write/.test(content);
assert.ok(importsSystem, 'Should detect system-under-test import');
assert.ok(writesFiles, 'Should detect file writing');
assert.ok(importsSystem && writesFiles, 'Script that imports SUT and writes fixtures is CIRCULAR');
describe('circular-detection marker', () => {
test('audit_test_quality step contains writeFileSync in the circular file-write grep pattern', () => {
// The step must tell the verifier to grep for writeFileSync in the circular
// detection pattern. Removing writeFileSync from the pattern would miss the
// most common Node.js synchronous file-write idiom.
assert.ok(
auditStepSrc !== null,
'Cannot check circular-detection marker: audit_test_quality step not found'
);
assert.ok(
auditStepSrc.includes('writeFileSync'),
'audit_test_quality step must include writeFileSync in the circular-detection grep pattern'
);
});
test('does not flag test helpers that only read fixtures', () => {
const testDir = path.join(tmpDir, 'tests');
fs.mkdirSync(testDir, { recursive: true });
test('audit_test_quality step contains standalone writeFile (not just as part of writeFileSync) in the circular file-write grep pattern', () => {
// The pattern must also catch the async fs.writeFile variant, not just the
// synchronous writeFileSync. A plain includes('writeFile') check is satisfied
// by the 'writeFileSync' substring and would pass even if the standalone
// 'writeFile' alternative were removed. Use a word-boundary / non-Sync regex
// to detect the standalone form specifically.
assert.ok(
auditStepSrc !== null,
'Cannot check writeFile marker: audit_test_quality step not found'
);
// Match 'writeFile' that is NOT followed by 'Sync' — i.e. the standalone form.
const standaloneWriteFile = /writeFile(?!Sync)/.test(auditStepSrc);
assert.ok(
standaloneWriteFile,
'audit_test_quality step must reference standalone writeFile (not just writeFileSync) ' +
'in the circular-detection grep pattern — narrowing the pattern to writeFileSync only ' +
'would be caught by this test'
);
});
fs.writeFileSync(path.join(testDir, 'loadFixtures.js'), [
'import { readFileSync } from "fs";',
'export function loadFixture(name) {',
' return JSON.parse(readFileSync(`fixtures/${name}.json`, "utf8"));',
'}',
].join('\n'));
test('audit_test_quality step contains fs\\.write in the circular file-write grep pattern', () => {
// The fs\.write pattern (dot backslash-escaped) covers lower-level write calls.
assert.ok(
auditStepSrc !== null,
'Cannot check fs\\.write marker: audit_test_quality step not found'
);
assert.ok(
auditStepSrc.includes('fs\\.write'),
'audit_test_quality step must include fs\\.write in the circular-detection grep pattern'
);
});
const content = fs.readFileSync(path.join(testDir, 'loadFixtures.js'), 'utf8');
const importsSystem = /import.*(?:Service|Engine|Calculator|Controller)/.test(content);
const writesFiles = /writeFileSync|writeFile|fs\.write/.test(content);
assert.ok(!importsSystem, 'Should not flag read-only helper as importing SUT');
assert.ok(!writesFiles, 'Should not flag read-only helper as writing files');
test('audit_test_quality step defines CIRCULAR as a blocker verdict', () => {
// The step must explicitly name CIRCULAR as an outcome and mark it as a blocker.
assert.ok(
auditStepSrc !== null,
'Cannot check CIRCULAR verdict: audit_test_quality step not found'
);
assert.ok(
auditStepSrc.includes('CIRCULAR'),
'audit_test_quality step must define CIRCULAR as a verdict for circular tests'
);
});
});
describe('assertion strength classification', () => {
test('classifies existence-only assertions as INSUFFICIENT for value requirements', () => {
const assertions = [
'expect(result).toBeDefined()',
'expect(result).not.toBeNull()',
'assert.ok(result)',
];
describe('provenance-classification contract', () => {
// Finding #5: the redesign dropped all coverage of the provenance-classification
// contract. These tests assert that the audit_test_quality step still defines the
// provenance keywords and classification tiers so that removing them goes RED.
const existencePattern = /toBeDefined|not\.toBeNull|assert\.ok\(/;
const valuePattern = /toEqual|toBeCloseTo|strictEqual|deepStrictEqual/;
for (const assertion of assertions) {
assert.ok(existencePattern.test(assertion), `"${assertion}" should match existence pattern`);
assert.ok(!valuePattern.test(assertion), `"${assertion}" should NOT match value pattern`);
}
test('audit_test_quality step defines the VALID provenance classification', () => {
assert.ok(
auditStepSrc !== null,
'Cannot check provenance classifications: audit_test_quality step not found'
);
assert.ok(
auditStepSrc.includes('VALID'),
'audit_test_quality step must define VALID as a provenance classification'
);
});
test('classifies value assertions as sufficient', () => {
const assertions = [
'expect(result).toBeCloseTo(155.96, 2)',
'expect(result).toEqual({ amount: 100 })',
'assert.strictEqual(result, 42)',
];
test('audit_test_quality step defines the UNKNOWN provenance classification', () => {
assert.ok(
auditStepSrc !== null,
'Cannot check provenance classifications: audit_test_quality step not found'
);
assert.ok(
auditStepSrc.includes('UNKNOWN'),
'audit_test_quality step must define UNKNOWN as a provenance classification'
);
});
const valuePattern = /toEqual|toBeCloseTo|strictEqual|deepStrictEqual/;
test('audit_test_quality step maps UNKNOWN to SUSPECT treatment', () => {
// The contract requires "UNKNOWN: No provenance information — treat as SUSPECT"
// so consumers know UNKNOWN is handled the same as SUSPECT.
assert.ok(
auditStepSrc !== null,
'Cannot check SUSPECT treatment: audit_test_quality step not found'
);
assert.ok(
auditStepSrc.includes('SUSPECT'),
'audit_test_quality step must mention SUSPECT (UNKNOWN must map to treat as SUSPECT)'
);
});
for (const assertion of assertions) {
assert.ok(valuePattern.test(assertion), `"${assertion}" should match value pattern`);
}
test('audit_test_quality step names "legacy" as a VALID provenance keyword', () => {
// VALID is defined as "Expected value from external/legacy system output,
// manual capture, or independent oracle". The word "legacy" is load-bearing:
// it clarifies that values captured from a superseded system are authoritative.
assert.ok(
auditStepSrc !== null,
'Cannot check "legacy" keyword: audit_test_quality step not found'
);
assert.ok(
auditStepSrc.includes('legacy'),
'audit_test_quality step must name "legacy" as a VALID provenance source ' +
'(e.g. "external/legacy system output")'
);
});
test('audit_test_quality step names "manual" as a VALID provenance keyword', () => {
// "manual capture" is the second example of a VALID provenance source and
// distinguishes human-curated expected values from machine-generated ones.
assert.ok(
auditStepSrc !== null,
'Cannot check "manual" keyword: audit_test_quality step not found'
);
assert.ok(
auditStepSrc.includes('manual'),
'audit_test_quality step must name "manual" as a VALID provenance source ' +
'(e.g. "manual capture")'
);
});
test('audit_test_quality step names "computed" as a SUSPECT provenance indicator', () => {
// The circular indicator comments list "computed from engine" as an example
// of a SUSPECT expected-value comment. Removing it would mean verifiers no
// longer know to flag tests whose fixtures declare computed provenance.
assert.ok(
auditStepSrc !== null,
'Cannot check "computed" indicator: audit_test_quality step not found'
);
assert.ok(
auditStepSrc.includes('computed'),
'audit_test_quality step must name "computed" as a SUSPECT provenance indicator ' +
'(e.g. "computed from engine" comment example)'
);
});
test('audit_test_quality step names "baseline" as a SUSPECT provenance indicator', () => {
// "captured from baseline" is the other canonical SUSPECT comment example.
assert.ok(
auditStepSrc !== null,
'Cannot check "baseline" indicator: audit_test_quality step not found'
);
assert.ok(
auditStepSrc.includes('baseline'),
'audit_test_quality step must name "baseline" as a SUSPECT provenance indicator ' +
'(e.g. "captured from baseline" comment example)'
);
});
});
describe('provenance classification', () => {
test('fixture with legacy system comment classified as VALID', () => {
const fixture = {
legacyId: 10341,
comment: 'Real PHP fixture - output from legacy system',
dbDependent: true,
expectedOutput: { value: 155.96 },
};
const hasLegacySource = /legacy|php|real|manual|captured from/i.test(fixture.comment || '');
assert.ok(hasLegacySource, 'Comment referencing legacy system = VALID provenance');
describe('assertion-strength table markers', () => {
test('audit_test_quality step contains the assertion-strength section header', () => {
// The "5. Assertion strength" section heading anchors the classification table.
assert.ok(
auditStepSrc !== null,
'Cannot check assertion-strength header: audit_test_quality step not found'
);
assert.ok(
auditStepSrc.includes('Assertion strength'),
'audit_test_quality step must contain the "Assertion strength" section header'
);
});
test('fixture with synthetic/baseline comment classified as SUSPECT', () => {
const fixture = {
legacyId: null,
comment: 'Synthetic offline fixture - computed from known algorithm',
dbDependent: false,
expectedOutput: { value: 1240.68 },
};
const hasSyntheticSource = /synthetic|computed|baseline|generated|captured from engine/i.test(fixture.comment || '');
const hasLegacySource = /legacy|php|real output|manual capture/i.test(fixture.comment || '');
assert.ok(hasSyntheticSource, 'Comment indicating synthetic source detected');
assert.ok(!hasLegacySource, 'Should NOT be classified as legacy source');
test('audit_test_quality step lists existence-only examples in the assertion table', () => {
// The table must include toBeDefined as an example of an existence-level assertion.
assert.ok(
auditStepSrc !== null,
'Cannot check assertion table: audit_test_quality step not found'
);
assert.ok(
auditStepSrc.includes('toBeDefined'),
'audit_test_quality step must include toBeDefined as an existence-level assertion example'
);
});
test('fixture with no comment classified as UNKNOWN', () => {
const fixture = {
expectedOutput: { value: 42 },
};
test('audit_test_quality step lists value-level examples in the assertion table', () => {
// The table must include toBeCloseTo as an example of a value-level assertion.
assert.ok(
auditStepSrc !== null,
'Cannot check value assertion example: audit_test_quality step not found'
);
assert.ok(
auditStepSrc.includes('toBeCloseTo'),
'audit_test_quality step must include toBeCloseTo as a value-level assertion example'
);
});
const hasAnyProvenance = (fixture.comment || '').length > 0;
assert.ok(!hasAnyProvenance, 'No comment = UNKNOWN provenance');
test('audit_test_quality step defines INSUFFICIENT verdict for weak assertions', () => {
// The step must explicitly name INSUFFICIENT as the verdict when assertion strength
// is below what the requirement demands.
assert.ok(
auditStepSrc !== null,
'Cannot check INSUFFICIENT verdict: audit_test_quality step not found'
);
assert.ok(
auditStepSrc.includes('INSUFFICIENT'),
'audit_test_quality step must define INSUFFICIENT as a verdict for weak assertions'
);
});
});
});