feat(#656): Research module — content-addressed cache + provider seam + registry-API legitimacy (#664)
* feat(#656): add Research Store module (content-addressed cache, TTL staleness) Content-addressed research cache behind a clock seam: researchKey (sha256, deterministic), putResearch/getResearch ({hit,stale}, never throws), ttlForSource (curated HIGH 30d / MED 7d / web LOW 1d), two-tier resolveStorePath (curated -> ~/.gsd/research-cache, web/synthesis -> project .planning/research/.cache). 28 behavioral + property tests; boundary coverage at ttl-1/ttl/ttl+1. Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(#656): add Research Provider module (waterfall + confidence + plan) Single source of truth for the Balanced provider waterfall (docs Context7->Ref->Jina, web Exa+Tavily, fallback Perplexity/Brave, Firecrawl scrape-only). classifyConfidence stamps HIGH|MEDIUM|LOW by provider (never throws). providerAvailability maps config flags to usable providers. planResearch checks the Research Store (injected seam) and returns cache-hits + a per-question fetch plan, falling through the waterfall to the always-available websearch terminal. 22 behavioral + property tests. Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(#656): add Package Legitimacy module (registry-API verdicts, slopcheck optional) Replaces the pip-install-or-degrade slopcheck prose gate with code: classifyPackage (pure, never throws) computes OK|SUS|SLOP from tunable thresholds (minAgeDays 30, minWeeklyDownloads 1000, requireRepo). checkPackages queries injectable npm/PyPI/crates registry adapters (real https with 5s timeout, degraded-not-thrown on failure); slopcheck is one optional adapter that can only escalate severity, never degrade to [ASSUMED]. 34 behavioral + property tests; boundary coverage on age and downloads (limit-1/limit/limit+1). Known follow-up: real npm adapter must add api.npmjs.org last-week downloads fetch (currently null -> unknown-downloads). Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(#656): detect Tavily/Ref/Perplexity/Jina provider keys; complete npm downloads adapter config: add tavily_search/ref_search/perplexity/jina availability flags (env var or ~/.gsd/<x>_api_key), mirroring brave_search/exa_search/firecrawl, so the Research Provider waterfall can gate them. package-legitimacy: real npm adapter now fetches api.npmjs.org last-week downloads (bounded, degraded-not-thrown) so weeklyDownloads is populated. +12 config tests; 34 legitimacy tests unchanged. Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(#656): expose Research seam via gsd-tools query (research-plan, research-store, package-legitimacy) Routes the L2-hybrid surface so agents reach it as CLI: 'query research-store get/put' (cache, HOME-sandboxable), 'query research-plan --input' (cache-hits + fetch plan from planResearch), 'query package-legitimacy check --ecosystem' (async registry verdicts). Commands skip .planning root resolution and appear in top-level usage. 5 behavioral runGsdTools tests; command-contract unchanged (335). Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(#656): document Research module (CONTEXT predicates, ADR-0656, architecture, changeset) Adds GSD-RESEARCH.* + DEFECT.RESEARCH-PROVIDER-PROSE-DRIFT predicates to CONTEXT.md, ADR-0656 recording the L2-hybrid seam decision, a docs/ARCHITECTURE.md Research Module subsection, and an Added changeset fragment (pr:0, backfill on PR). Notes the #657 deferrals (agent collapse + install.js MCP mapping). Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(#656): sync inventory for research modules Regenerate INVENTORY-MANIFEST.json and bump docs/INVENTORY.md CLI Modules count 82->85 with rows for research-store/research-provider/package-legitimacy (DEFECT.INVENTORY-DRIFT). Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(#656): eslint-ignore generated research .cjs artifacts (ADR-457) research-store/research-provider/package-legitimacy .cjs are tsc-generated from src/*.cts, so they belong in the ESLint ignore block (lint the .cts source, not the emitted .cjs). Fixes tests/551-eslint-bin-lib-coverage. Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(#656): backfill changeset pr number to #664 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(#656): satisfy eslint lint-tests gate Fix 20 eslint errors in the new research files: use helpers.cleanup() instead of raw fs.rmSync() in tests (local/no-raw-rmsync-in-tests, Windows-EBUSY retry budget); drop redundant '| string' union members and unnecessary type assertions; deterministic object normalization in researchKey (no-base-to-string). Logic unchanged; 6180 tests still green. Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#656): harden package legitimacy per review (W1/W2/I3/I4) W1: httpsGet now reads statusCode; npm/PyPI/crates map 404 -> exists:false -> SLOP (registry-existence is the #1 slopsquatting defense; previously only npm caught it). Transport made injectable (_setHttpGet) for hermetic 404 tests. W2: suspicious-postinstall is now terminal SLOP independent of the optional slopcheck adapter, and the regex drops the bare https?:// arm (over-fired on esbuild/sharp/node-gyp) for shell-exec/download-exec signatures only. I3: checkPackages now threads version to registry.lookup and adapters verify that specific version exists. I4: moreServerVerdict -> moreSevereVerdict. +11 regression tests (all RED-first); 45 total green. Addresses review by @davesienkowski on #664. Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#656): research-store tier coherence + freshness + version TTL (W4/I1/I2/I4) I1: tier now derives from source (curated -> user ~/.gsd, else -> project .planning), not kind, so put-tier and get-tier can't diverge; kind is a key component only. W4: getResearch searches both tiers and returns the freshest (non-stale preferred), never letting a stale curated entry shadow a fresh web one; blank version caps TTL at 1 day (no 30d on version-blind keys). I2: atomic platformWriteSync instead of raw fs.writeFileSync on the shared global path. I4: dropped the dead ttlForSource arm. CLI get now searches both tiers. +5 RED-first regression tests; 38 green. Addresses review by @davesienkowski on #664. Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#656): expose classifyConfidence as a CLI route, killing dead code (W3) Adds 'gsd-tools query classify-confidence --provider X [--verified]' so research agents get the confidence tier FROM CODE (provider waterfall + verification lever) instead of asserting it in prose. classifyConfidence previously had no runtime caller. HIGH means 'trusted provider'; --verified raises web results to MEDIUM (verification semantics documented in ADR-0656). +4 behavioral tests. Addresses review by @davesienkowski on #664 (W3). Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#656): close Codex adversarial-review findings (path-traversal, version-age, malformed-cache) HIGH: research key must be 64-hex sha256 (isValidResearchKey) + resolved-path containment check in put/get + CLI validation -> blocks '../../x' arbitrary-file-write. HIGH: package legitimacy now derives publishedAt from the REQUESTED version (npm time[version], PyPI releases[version] upload_time, crates versions[].created_at) so a new malicious version of an old package can't inherit old age and evade 'too-new'. MEDIUM: getResearch validates entry shape (finite fetched_at + positive ttl + required fields) -> malformed cache entry is a miss, not fresh-forever. +regression tests (RED-first); 111 green. Codex adversarial review (required pre-PR gate). Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#656): close code-review correctness findings (1) package-legitimacy CLI now rejects unknown --flags instead of silently consuming the following package as a flag value; only --ecosystem takes a value. (2) crates recent_downloads (90-day) normalized to a weekly figure before the minWeeklyDownloads threshold (was ~13x too lenient). (3) research-plan --input validates parsed JSON is an object with an Array questions before destructuring -> clean usage error instead of an uncaught TypeError on null/bad input. (4) research-store put rejects a flag value that is itself a --flag (no more storing '--source' as content). (5) planResearch skips questions whose text is not a non-empty string instead of emitting question:undefined. +13 RED-first regression tests; 143 green. Code-review gate. Issue #656. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(#657): extract researcher documentation_lookup to shared @-reference 6 researcher agents carried a near-duplicate <documentation_lookup> block; consolidate into gsd-core/references/research-documentation-lookup.md (@-included). Unifies the ctx7 CLI fallback to the safer 'command -v ctx7' guard (drops silent 'npx --yes ctx7@latest' execution in 5 agents). Behavior-preserving dedup; inventory 63->64 references. Phase A of the agent collapse. Issue #657. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(#657): extract researcher philosophy + verification-protocol to shared @-references philosophy and the pitfalls+pre-submission-checklist common-core were near-duplicated in project/phase researchers; consolidate into gsd-core/references/research-{philosophy,verification-protocol}.md (@-included). phase-researcher keeps its 3 extra checklist items inline. Pre-submission domains checklist made agent-agnostic so project-researcher doesn't lose features/architecture coverage. Write-contract intentionally left inline (bug-214 tests assert it verbatim). Inventory 64->66 refs. Behavior-preserving. Phase A. Issue #657. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(#657): wire gsd-phase-researcher to the Research seam (Phase B / S1) The phase researcher now CALLS the code seam instead of carrying inline mechanics: provider waterfall -> 'gsd-tools query research-plan' (+ research-store put to cache digests); confidence-tier prose -> 'gsd-tools query classify-confidence'; slopcheck pip-install protocol -> 'gsd-tools query package-legitimacy check'. This makes the Research module a real runtime consumer (validates the seam end-to-end, addresses reviewer S1) and removes the duplicated waterfall/confidence/slopcheck prose. RESEARCH.md output contract, commit step, structured returns, and Phase-A @-includes unchanged. package-legitimacy-gate.test.cjs rewritten prose-grep -> behavioral (asserts the seam invocation). Issue #657. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(#657): wire gsd-project-researcher to the seam + add tavily/ref/jina MCP tools (Phase C.1) project-researcher now calls gsd-tools query research-plan / classify-confidence (+ research-store put) instead of the inline provider waterfall + confidence-tier prose (mirrors the phase-researcher rewire; no package-legitimacy — phase-only). Output contract (STACK/FEATURES/ARCHITECTURE/PITFALLS/SUMMARY.md + sections, no-commit, structured returns, Phase-A @-includes) unchanged. Adds mcp__tavily/ref/jina__* to the project/phase/ui researcher tools frontmatter (Balanced provider set) so install.js MCP mapping (C.2) has a consumer. Issue #657. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(#657): cover tavily/ref/jina MCP install handling + frontmatter parity guard (Phase C.2) Investigation: exa/firecrawl have no explicit per-runtime tool-mapping — every mcp__<server>__* except context7 rides the generic passthrough (Copilot lowercases; OpenCode/Cursor/Windsurf/Augment keep as-is; Gemini auto-discovers). tavily/ref/jina are handled identically, no install path broken. Added 12 copilot-install passthrough tests + a mcp-tool-inheritance parity guard (tavily co-declared with exa, jina with firecrawl, ref present across the 3 web researchers) so the MCP set can't drift. No io.github registry ids invented (none sourceable in-repo); documented as a follow-up. 488 tests green. Issue #657. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(#657): profiles as source of truth for researcher agents + drift-guard (Phase C.3) scripts/research-profiles.cjs declares each of the 7 researcher agents' identity + contract (name, description, color, tools, required @-includes, required gsd-tools seam calls, output-contract markers). scripts/gen-research-agents.cjs --check validates every committed agent against its profile; --write regenerates ONLY the frontmatter from profiles (body untouched) and is a verified no-op against the current agents (zero diff = fidelity). tests/research-agent-profiles.test.cjs is the DEFECT.GENERATIVE-FIX drift guard. Design note: profiles govern the generatable/contract surface rather than destructively regenerating the disparate operational prose bodies (those were deduped via @-includes in Phase A). scripts/ is not inventoried (no inventory change). Issue #657. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#657): complete agent provider-dispatch + parity guard; align legitimacy field; validate profiles Adversarial-review findings: (HIGH) the seam-wired agents' Step-C dispatch only mapped 6 providers, so a planResearch result of jina/ref/perplexity/brave (reachable via the waterfall fallbacks) had no handling -> agent stall; completed both agents' dispatch to all 9 PROVIDER_WATERFALL ids + a catch-all, and added a parity test asserting agent dispatch stays in sync with research-provider PROVIDER_WATERFALL (DEFECT.GENERATIVE-FIX). (MEDIUM) phase-researcher package-legitimacy JSON example used 'package' but the module returns 'name' -> aligned. (LOW) gen-research-agents checkAgent now returns a clear failure for a malformed profile instead of throwing. +parity/validation tests (RED-first). Issue #657. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#656): make classifyConfidence verification-evidence-driven (W3) Confidence conflated provider authority with claim verification — context7/ref stamped HIGH purely by provider identity, and the only verification lever was a self-set --verified flag. Split into two axes: provider authority (static) + verification evidence (code-computed). HIGH now requires ground-truth corroboration (legitimacyVerdict OK), independent of provider; authority alone caps at MEDIUM; SLOP caps at LOW; the self-reported --verified is demoted to a MEDIUM-only web lever. HIGH = corroborated-against-authoritative-source, not a correctness guarantee. Adds --legitimacy-verdict to the classify-confidence CLI; updates CONTEXT.md predicate + ADR-0656 (tier set unchanged, ADR-consistent). Addresses davesienkowski's W3 review on #664. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#656): bind classify-confidence verdict to code, closing CLI self-grading Adversarial review found the new --legitimacy-verdict flag was caller-supplied, so an agent could self-assert OK->HIGH without any real legitimacy check — reintroducing the exact self-grading hole W3 closes. Remove the free flag; the CLI now computes the verdict via checkPackages only when --package/--ecosystem is given (code-computed, not agent-asserted). Update the stale CLI test (context7 alone -> MEDIUM) and extend the property test to vary legitimacyVerdict. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
@@ -176,6 +176,14 @@ function buildNewProjectConfig(userChoices: Record<string, unknown>): Record<str
|
||||
const hasFirecrawl = !!(process.env['FIRECRAWL_API_KEY'] || fs.existsSync(firecrawlKeyFile));
|
||||
const exaKeyFile = path.join(homedir, '.gsd', 'exa_api_key');
|
||||
const hasExaSearch = !!(process.env['EXA_API_KEY'] || fs.existsSync(exaKeyFile));
|
||||
const tavilyKeyFile = path.join(homedir, '.gsd', 'tavily_api_key');
|
||||
const hasTavilySearch = !!(process.env['TAVILY_API_KEY'] || fs.existsSync(tavilyKeyFile));
|
||||
const refKeyFile = path.join(homedir, '.gsd', 'ref_api_key');
|
||||
const hasRefSearch = !!(process.env['REF_API_KEY'] || fs.existsSync(refKeyFile));
|
||||
const perplexityKeyFile = path.join(homedir, '.gsd', 'perplexity_api_key');
|
||||
const hasPerplexity = !!(process.env['PERPLEXITY_API_KEY'] || fs.existsSync(perplexityKeyFile));
|
||||
const jinaKeyFile = path.join(homedir, '.gsd', 'jina_api_key');
|
||||
const hasJina = !!(process.env['JINA_API_KEY'] || fs.existsSync(jinaKeyFile));
|
||||
|
||||
// Load user-level defaults from ~/.gsd/defaults.json if available
|
||||
const globalDefaultsPath = path.join(homedir, '.gsd', 'defaults.json');
|
||||
@@ -205,6 +213,10 @@ function buildNewProjectConfig(userChoices: Record<string, unknown>): Record<str
|
||||
brave_search: hasBraveSearch,
|
||||
firecrawl: hasFirecrawl,
|
||||
exa_search: hasExaSearch,
|
||||
tavily_search: hasTavilySearch,
|
||||
ref_search: hasRefSearch,
|
||||
perplexity: hasPerplexity,
|
||||
jina: hasJina,
|
||||
git: {
|
||||
branching_strategy: CONFIG_DEFAULTS.branching_strategy,
|
||||
create_tag: true,
|
||||
|
||||
471
src/package-legitimacy.cts
Normal file
471
src/package-legitimacy.cts
Normal file
@@ -0,0 +1,471 @@
|
||||
/**
|
||||
* Package Legitimacy Module
|
||||
*
|
||||
* Replaces the bolt-on prose slopcheck gate (which pip-installed `slopcheck`
|
||||
* and degraded ALL packages to [ASSUMED] when pip failed) with registry-API
|
||||
* verdicts computed in code.
|
||||
*
|
||||
* Public interface:
|
||||
* DEFAULT_THRESHOLDS — baseline thresholds
|
||||
* classifyPackage — pure function: signals → { verdict, reasons }
|
||||
* checkPackages — async: resolves registry signals and classifies
|
||||
* _setHttpGet — test seam: override the HTTP transport (pass null to restore)
|
||||
*
|
||||
* All network IO is injected via a `registry` client option so that tests
|
||||
* never touch the real network (same seam pattern as clock injection).
|
||||
*
|
||||
* ADR-457 build-at-publish: authored as TypeScript .cts → emits .cjs via tsc.
|
||||
*/
|
||||
|
||||
import * as https from 'node:https';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Types
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
type Verdict = 'OK' | 'SUS' | 'SLOP';
|
||||
type Ecosystem = string;
|
||||
|
||||
interface Thresholds {
|
||||
minAgeDays: number;
|
||||
minWeeklyDownloads: number;
|
||||
requireRepo: boolean;
|
||||
}
|
||||
|
||||
interface PackageSignals {
|
||||
exists: boolean | null | undefined;
|
||||
publishedAt: string | null | undefined;
|
||||
weeklyDownloads: number | null | undefined;
|
||||
repoUrl: string | null | undefined;
|
||||
deprecated: boolean | null | undefined;
|
||||
postinstall: string | null | undefined;
|
||||
ecosystem?: string | null | undefined;
|
||||
}
|
||||
|
||||
interface ClassifyResult {
|
||||
verdict: Verdict;
|
||||
reasons: string[];
|
||||
}
|
||||
|
||||
interface CheckResult {
|
||||
name: string;
|
||||
verdict: Verdict;
|
||||
signals: PackageSignals;
|
||||
reasons: string[];
|
||||
}
|
||||
|
||||
interface RegistryClient {
|
||||
lookup(ecosystem: Ecosystem, name: string, version?: string): Promise<PackageSignals>;
|
||||
}
|
||||
|
||||
interface SlopcheckAdapter {
|
||||
check(ecosystem: Ecosystem, name: string): Promise<Verdict | null>;
|
||||
}
|
||||
|
||||
interface ClassifyOptions {
|
||||
thresholds?: Thresholds;
|
||||
clock?: { now(): number };
|
||||
}
|
||||
|
||||
interface CheckPackagesInput {
|
||||
ecosystem: Ecosystem;
|
||||
packages: string[];
|
||||
version?: string;
|
||||
}
|
||||
|
||||
interface CheckPackagesOptions {
|
||||
registry?: RegistryClient;
|
||||
clock?: { now(): number };
|
||||
thresholds?: Thresholds;
|
||||
slopcheck?: SlopcheckAdapter | null;
|
||||
}
|
||||
|
||||
/** Shape returned by the injectable HTTP transport */
|
||||
interface HttpResponse {
|
||||
statusCode: number;
|
||||
body: string;
|
||||
}
|
||||
|
||||
type HttpGetFn = (url: string, timeoutMs: number) => Promise<HttpResponse>;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Constants
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const DEFAULT_THRESHOLDS: Thresholds = {
|
||||
minAgeDays: 30,
|
||||
minWeeklyDownloads: 1000,
|
||||
requireRepo: true,
|
||||
};
|
||||
|
||||
// Matches common dangerous postinstall execution patterns.
|
||||
// Deliberately EXCLUDES bare https?:// (over-fires on legit packages like
|
||||
// esbuild/sharp/node-gyp that reference download URLs without executing them).
|
||||
// Shell-execution / download-and-exec signatures only:
|
||||
const SUSPICIOUS_POSTINSTALL_RE =
|
||||
/(curl |wget |\|\s*(ba)?sh|bash -c|sh -c|node -e|eval|base64 -d|\/etc\/|\.\.\/|~\/|nc |>\s*\/)/i;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Severity ordering for verdict merging (SLOP > SUS > OK)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const SEVERITY: Record<Verdict, number> = { OK: 0, SUS: 1, SLOP: 2 };
|
||||
|
||||
function moreSevereVerdict(a: Verdict, b: Verdict): Verdict {
|
||||
return SEVERITY[a] >= SEVERITY[b] ? a : b;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// classifyPackage — pure, no IO
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function classifyPackage(
|
||||
signals: Partial<PackageSignals>,
|
||||
{ thresholds = DEFAULT_THRESHOLDS, clock = Date }: ClassifyOptions = {}
|
||||
): ClassifyResult {
|
||||
const reasons: string[] = [];
|
||||
|
||||
// Terminal: package does not exist
|
||||
if (signals.exists === false) {
|
||||
return { verdict: 'SLOP', reasons: ['does-not-exist'] };
|
||||
}
|
||||
|
||||
// Age check
|
||||
if (signals.publishedAt == null) {
|
||||
reasons.push('unknown-age');
|
||||
} else {
|
||||
const parsed = Date.parse(String(signals.publishedAt));
|
||||
if (!Number.isFinite(parsed)) {
|
||||
// Unparseable date — treat as unknown
|
||||
reasons.push('unknown-age');
|
||||
} else {
|
||||
const ageDays = Math.floor((clock.now() - parsed) / 86_400_000);
|
||||
if (ageDays < thresholds.minAgeDays) {
|
||||
reasons.push('too-new');
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Downloads check
|
||||
const downloads = signals.weeklyDownloads;
|
||||
if (downloads == null) {
|
||||
reasons.push('unknown-downloads');
|
||||
} else if (typeof downloads !== 'number' || !Number.isFinite(downloads)) {
|
||||
// Odd type / NaN — treat as unknown
|
||||
reasons.push('unknown-downloads');
|
||||
} else if (downloads < thresholds.minWeeklyDownloads) {
|
||||
reasons.push('low-downloads');
|
||||
}
|
||||
|
||||
// Repository check
|
||||
if (thresholds.requireRepo && !signals.repoUrl) {
|
||||
reasons.push('no-repository');
|
||||
}
|
||||
|
||||
// Deprecated check
|
||||
if (signals.deprecated === true) {
|
||||
reasons.push('deprecated');
|
||||
}
|
||||
|
||||
// Suspicious postinstall (npm only — but apply whenever postinstall is present)
|
||||
if (signals.postinstall != null && typeof signals.postinstall === 'string') {
|
||||
if (SUSPICIOUS_POSTINSTALL_RE.test(signals.postinstall)) {
|
||||
reasons.push('suspicious-postinstall');
|
||||
}
|
||||
}
|
||||
|
||||
// Terminal: suspicious postinstall is a slopsquatting execution risk
|
||||
if (reasons.includes('suspicious-postinstall')) {
|
||||
return { verdict: 'SLOP', reasons };
|
||||
}
|
||||
|
||||
const verdict: Verdict = reasons.length > 0 ? 'SUS' : 'OK';
|
||||
return { verdict, reasons };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Injectable HTTP transport (test seam — W1)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** The real HTTPS transport — resolves { statusCode, body } */
|
||||
function realHttpsGet(url: string, timeoutMs: number): Promise<HttpResponse> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const req = https.get(
|
||||
url,
|
||||
{ headers: { 'User-Agent': 'gsd-core-package-legitimacy/1.0' } },
|
||||
(res) => {
|
||||
const chunks: Buffer[] = [];
|
||||
res.on('data', (c: Buffer) => chunks.push(c));
|
||||
res.on('end', () =>
|
||||
resolve({
|
||||
statusCode: res.statusCode ?? 0,
|
||||
body: Buffer.concat(chunks).toString('utf8'),
|
||||
})
|
||||
);
|
||||
res.on('error', reject);
|
||||
}
|
||||
);
|
||||
req.setTimeout(timeoutMs, () => {
|
||||
req.destroy(new Error(`timeout after ${timeoutMs}ms`));
|
||||
});
|
||||
req.on('error', reject);
|
||||
});
|
||||
}
|
||||
|
||||
/** Module-level transport pointer — overrideable via _setHttpGet for tests */
|
||||
let httpsGet: HttpGetFn = realHttpsGet;
|
||||
|
||||
/**
|
||||
* Test seam: replace the HTTP transport. Pass null to restore the real transport.
|
||||
* Tests call this before exercising a real-adapter code path; always restore in finally.
|
||||
*/
|
||||
function _setHttpGet(fn: HttpGetFn | null): void {
|
||||
httpsGet = fn ?? realHttpsGet;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Real registry adapters (not exercised by tests — tests inject fakes)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function degradedSignals(): PackageSignals {
|
||||
return {
|
||||
exists: null,
|
||||
publishedAt: null,
|
||||
weeklyDownloads: null,
|
||||
repoUrl: null,
|
||||
deprecated: false,
|
||||
postinstall: null,
|
||||
};
|
||||
}
|
||||
|
||||
async function lookupNpm(name: string, version?: string): Promise<PackageSignals> {
|
||||
try {
|
||||
const resp = await httpsGet(`https://registry.npmjs.org/${encodeURIComponent(name)}`, 5000);
|
||||
if (resp.statusCode === 404) return { ...degradedSignals(), exists: false };
|
||||
if (resp.statusCode < 200 || resp.statusCode >= 300) return degradedSignals();
|
||||
|
||||
const data = JSON.parse(resp.body) as Record<string, unknown>;
|
||||
if (data.error) return { ...degradedSignals(), exists: false };
|
||||
|
||||
const time = (data.time as Record<string, string> | undefined) ?? {};
|
||||
const allVersions = (data.versions as Record<string, unknown> | undefined) ?? {};
|
||||
|
||||
// I3: when a specific version is requested, verify it exists
|
||||
if (version !== undefined) {
|
||||
if (!(version in allVersions)) {
|
||||
return { ...degradedSignals(), exists: false };
|
||||
}
|
||||
}
|
||||
|
||||
const latestVersion = (data['dist-tags'] as Record<string, string> | undefined)?.latest ?? '';
|
||||
const resolvedVersion = version !== undefined ? version : latestVersion;
|
||||
const versionMeta = allVersions[resolvedVersion] ?? {};
|
||||
|
||||
const scripts =
|
||||
((versionMeta as Record<string, unknown>).scripts as Record<string, string> | undefined) ??
|
||||
{};
|
||||
const postinstall = scripts.postinstall ?? null;
|
||||
|
||||
const repoField = (versionMeta as Record<string, unknown>).repository;
|
||||
let repoUrl: string | null = null;
|
||||
if (typeof repoField === 'string') repoUrl = repoField;
|
||||
else if (repoField && typeof (repoField as Record<string, unknown>).url === 'string') {
|
||||
repoUrl = (repoField as Record<string, string>).url;
|
||||
}
|
||||
|
||||
const deprecated =
|
||||
typeof (versionMeta as Record<string, unknown>).deprecated === 'string' ? true : false;
|
||||
|
||||
// Fetch weekly download count from the npm downloads API
|
||||
let weeklyDownloads: number | null = null;
|
||||
try {
|
||||
const dlResp = await httpsGet(
|
||||
`https://api.npmjs.org/downloads/point/last-week/${encodeURIComponent(name)}`,
|
||||
5000
|
||||
);
|
||||
if (dlResp.statusCode >= 200 && dlResp.statusCode < 300) {
|
||||
const dlData = JSON.parse(dlResp.body) as Record<string, unknown>;
|
||||
if (typeof dlData.downloads === 'number') {
|
||||
weeklyDownloads = dlData.downloads;
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Degraded: leave weeklyDownloads as null, never throw
|
||||
}
|
||||
|
||||
return {
|
||||
exists: true,
|
||||
publishedAt: time[resolvedVersion] ?? time.created ?? null,
|
||||
weeklyDownloads,
|
||||
repoUrl,
|
||||
deprecated,
|
||||
postinstall,
|
||||
ecosystem: 'npm',
|
||||
};
|
||||
} catch {
|
||||
return degradedSignals();
|
||||
}
|
||||
}
|
||||
|
||||
async function lookupPypi(name: string, version?: string): Promise<PackageSignals> {
|
||||
try {
|
||||
const resp = await httpsGet(`https://pypi.org/pypi/${encodeURIComponent(name)}/json`, 5000);
|
||||
if (resp.statusCode === 404) return { ...degradedSignals(), exists: false };
|
||||
if (resp.statusCode < 200 || resp.statusCode >= 300) return degradedSignals();
|
||||
|
||||
const data = JSON.parse(resp.body) as Record<string, unknown>;
|
||||
const info = (data.info as Record<string, unknown>) ?? {};
|
||||
|
||||
// I3: when a specific version is requested, verify it exists in releases
|
||||
const releases = (data.releases as Record<string, unknown> | undefined) ?? {};
|
||||
if (version !== undefined) {
|
||||
if (!(version in releases)) {
|
||||
return { ...degradedSignals(), exists: false };
|
||||
}
|
||||
}
|
||||
|
||||
// Finding 2: when version is provided, derive publishedAt from the
|
||||
// version-specific release record rather than the package-level urls[] array
|
||||
// (which reflects the latest release, not the requested version).
|
||||
let uploadTime: string | null = null;
|
||||
if (version !== undefined) {
|
||||
const versionFiles = (releases[version] as Array<Record<string, unknown>> | undefined) ?? [];
|
||||
uploadTime =
|
||||
versionFiles.length > 0
|
||||
? (versionFiles[0].upload_time_iso_8601 as string | undefined) ?? null
|
||||
: null;
|
||||
} else {
|
||||
const urls = (data.urls as Array<Record<string, unknown>>) ?? [];
|
||||
uploadTime =
|
||||
urls.length > 0 ? (urls[0].upload_time_iso_8601 as string | undefined) ?? null : null;
|
||||
}
|
||||
|
||||
const projectUrls = info.project_urls as Record<string, string> | undefined;
|
||||
const repoUrl =
|
||||
projectUrls?.['Source'] ??
|
||||
projectUrls?.['Homepage'] ??
|
||||
(info.home_page as string | undefined) ??
|
||||
null;
|
||||
|
||||
return {
|
||||
exists: true,
|
||||
publishedAt: uploadTime,
|
||||
weeklyDownloads: null, // PyPI weekly downloads require a separate API
|
||||
repoUrl: repoUrl || null,
|
||||
deprecated: false, // PyPI doesn't have a first-class deprecated field
|
||||
postinstall: null, // Not applicable for PyPI
|
||||
ecosystem: 'pypi',
|
||||
};
|
||||
} catch {
|
||||
return degradedSignals();
|
||||
}
|
||||
}
|
||||
|
||||
async function lookupCrates(name: string, version?: string): Promise<PackageSignals> {
|
||||
try {
|
||||
const resp = await httpsGet(
|
||||
`https://crates.io/api/v1/crates/${encodeURIComponent(name)}`,
|
||||
5000
|
||||
);
|
||||
if (resp.statusCode === 404) return { ...degradedSignals(), exists: false };
|
||||
if (resp.statusCode < 200 || resp.statusCode >= 300) return degradedSignals();
|
||||
|
||||
const data = JSON.parse(resp.body) as Record<string, unknown>;
|
||||
const krate = (data.crate as Record<string, unknown>) ?? {};
|
||||
|
||||
// I3: when a specific version is requested, verify it exists in versions list
|
||||
const versions = (data.versions as Array<Record<string, unknown>> | undefined) ?? [];
|
||||
if (version !== undefined) {
|
||||
const found = versions.some(
|
||||
(v) => (v.num as string | undefined) === version
|
||||
);
|
||||
if (!found) {
|
||||
return { ...degradedSignals(), exists: false };
|
||||
}
|
||||
}
|
||||
|
||||
const repoUrl = (krate.repository as string | undefined) ?? null;
|
||||
// Finding 2: when version is provided, use the version-specific created_at
|
||||
// rather than the package-level crate.created_at (first-ever publish date).
|
||||
let created: string | null;
|
||||
if (version !== undefined) {
|
||||
const versionObj = versions.find((v) => (v.num as string | undefined) === version);
|
||||
created = (versionObj?.created_at as string | undefined) ?? null;
|
||||
} else {
|
||||
created = (krate.created_at as string | undefined) ?? null;
|
||||
}
|
||||
// recent_downloads is a 90-day count; normalize to a weekly figure for comparison
|
||||
// against minWeeklyDownloads (which is a weekly threshold).
|
||||
const rawDownloads = krate.recent_downloads;
|
||||
const downloads = (rawDownloads != null && typeof Number(rawDownloads) === 'number' && !isNaN(Number(rawDownloads)))
|
||||
? Math.round(Number(rawDownloads) * 7 / 90)
|
||||
: null;
|
||||
|
||||
return {
|
||||
exists: true,
|
||||
publishedAt: created,
|
||||
weeklyDownloads: downloads,
|
||||
repoUrl,
|
||||
deprecated: false,
|
||||
postinstall: null,
|
||||
ecosystem: 'crates',
|
||||
};
|
||||
} catch {
|
||||
return degradedSignals();
|
||||
}
|
||||
}
|
||||
|
||||
const realRegistry: RegistryClient = {
|
||||
async lookup(ecosystem: Ecosystem, name: string, version?: string): Promise<PackageSignals> {
|
||||
switch (ecosystem) {
|
||||
case 'npm':
|
||||
return lookupNpm(name, version);
|
||||
case 'pypi':
|
||||
return lookupPypi(name, version);
|
||||
case 'crates':
|
||||
return lookupCrates(name, version);
|
||||
default:
|
||||
return degradedSignals();
|
||||
}
|
||||
},
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// checkPackages — orchestrates lookup + classify + slopcheck merge
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
async function checkPackages(
|
||||
{ ecosystem, packages, version }: CheckPackagesInput,
|
||||
{
|
||||
registry = realRegistry,
|
||||
clock = Date,
|
||||
thresholds = DEFAULT_THRESHOLDS,
|
||||
slopcheck = null,
|
||||
}: CheckPackagesOptions = {}
|
||||
): Promise<CheckResult[]> {
|
||||
const results: CheckResult[] = [];
|
||||
|
||||
for (const name of packages) {
|
||||
const signals = await registry.lookup(ecosystem, name, version);
|
||||
const { verdict: registryVerdict, reasons } = classifyPackage(signals, { thresholds, clock });
|
||||
|
||||
let finalVerdict: Verdict = registryVerdict;
|
||||
|
||||
if (slopcheck != null) {
|
||||
const slopVerdict = await slopcheck.check(ecosystem, name);
|
||||
if (slopVerdict != null) {
|
||||
finalVerdict = moreSevereVerdict(finalVerdict, slopVerdict);
|
||||
}
|
||||
}
|
||||
|
||||
results.push({ name, verdict: finalVerdict, signals, reasons });
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Module export (CommonJS interop — export = only, no other export keywords)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export = { DEFAULT_THRESHOLDS, classifyPackage, checkPackages, _setHttpGet };
|
||||
281
src/research-provider.cts
Normal file
281
src/research-provider.cts
Normal file
@@ -0,0 +1,281 @@
|
||||
/**
|
||||
* Research Provider Module
|
||||
*
|
||||
* Encodes the Balanced-set provider decision: PROVIDER_WATERFALL constant,
|
||||
* classifyConfidence, providerAvailability, and planResearch (with injectable
|
||||
* store for testability).
|
||||
*
|
||||
* ADR-457 build-at-publish: authored as TypeScript .cts → emits .cjs via tsc.
|
||||
*/
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Types
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
type ProviderKind = 'docs' | 'web' | 'scrape';
|
||||
type ConfidenceLevel = 'HIGH' | 'MEDIUM' | 'LOW';
|
||||
|
||||
interface ProviderWaterfall {
|
||||
docs: string[];
|
||||
web: string[];
|
||||
scrape: string[];
|
||||
}
|
||||
|
||||
interface ClassifyConfidenceInput {
|
||||
provider?: unknown;
|
||||
verifiedAgainstOfficial?: unknown;
|
||||
legitimacyVerdict?: unknown;
|
||||
}
|
||||
|
||||
interface ProviderAvailabilityConfig {
|
||||
exa_search?: unknown;
|
||||
tavily_search?: unknown;
|
||||
brave_search?: unknown;
|
||||
firecrawl?: unknown;
|
||||
ref_search?: unknown;
|
||||
perplexity?: unknown;
|
||||
jina?: unknown;
|
||||
[key: string]: unknown;
|
||||
}
|
||||
|
||||
interface Question {
|
||||
text: string;
|
||||
kind: string;
|
||||
library?: string;
|
||||
version?: string;
|
||||
}
|
||||
|
||||
interface StoreResult {
|
||||
hit: boolean;
|
||||
stale: boolean;
|
||||
entry: unknown;
|
||||
}
|
||||
|
||||
interface ResearchStore {
|
||||
researchKey(input: {
|
||||
ecosystem?: unknown;
|
||||
library?: unknown;
|
||||
version?: unknown;
|
||||
query?: unknown;
|
||||
kind?: unknown;
|
||||
}): string;
|
||||
getResearch(
|
||||
cwd: string,
|
||||
key: string,
|
||||
opts?: { clock?: typeof Date; homeDir?: string; kind?: string }
|
||||
): StoreResult;
|
||||
}
|
||||
|
||||
interface PlanResearchOptions {
|
||||
questions: Question[];
|
||||
ecosystem?: string;
|
||||
cwd: string;
|
||||
config?: ProviderAvailabilityConfig;
|
||||
clock?: typeof Date;
|
||||
homeDir?: string;
|
||||
store?: ResearchStore;
|
||||
}
|
||||
|
||||
interface CacheInfo {
|
||||
hit: boolean;
|
||||
stale: boolean;
|
||||
}
|
||||
|
||||
interface FetchInfo {
|
||||
provider: string;
|
||||
query: string;
|
||||
}
|
||||
|
||||
interface ResearchItem {
|
||||
question: string;
|
||||
key: string;
|
||||
cache?: CacheInfo;
|
||||
fetch?: FetchInfo;
|
||||
}
|
||||
|
||||
interface PlanResearchResult {
|
||||
items: ResearchItem[];
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Cycle 1 / Cycle 6: PROVIDER_WATERFALL (Balanced-set decision)
|
||||
// firecrawl appears ONLY in scrape — demoted to known-URL scrape, NOT in docs/web
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const PROVIDER_WATERFALL: ProviderWaterfall = {
|
||||
docs: ['context7', 'ref', 'jina', 'websearch'],
|
||||
web: ['exa', 'tavily', 'perplexity', 'brave', 'websearch'],
|
||||
scrape: ['firecrawl', 'jina'],
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Cycle 4: classifyConfidence (evidence-driven)
|
||||
//
|
||||
// HIGH = corroborated against an authoritative source (registry ground-truth),
|
||||
// NOT a correctness guarantee.
|
||||
//
|
||||
// Two-axis model:
|
||||
// axis-1: authorityOf(provider) → 'official' | 'scrape' | 'web' | 'none'
|
||||
// axis-2: groundTruth (legitimacyVerdict normalized to OK|SUS|SLOP)
|
||||
//
|
||||
// Decision table (evaluated in order):
|
||||
// legitimacyVerdict === 'SLOP' → LOW (caps everything)
|
||||
// groundTruth && authority !== 'none' → HIGH
|
||||
// authority === 'official' || authority === 'scrape' → MEDIUM
|
||||
// groundTruth (authority 'none') → MEDIUM
|
||||
// authority === 'web' && verifiedAgainstOfficial → MEDIUM
|
||||
// else → LOW
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
type ProviderAuthority = 'official' | 'scrape' | 'web' | 'none';
|
||||
|
||||
function authorityOf(provider: unknown): ProviderAuthority {
|
||||
switch (provider) {
|
||||
case 'context7':
|
||||
case 'ref':
|
||||
return 'official';
|
||||
case 'jina':
|
||||
case 'firecrawl':
|
||||
return 'scrape';
|
||||
case 'exa':
|
||||
case 'tavily':
|
||||
case 'perplexity':
|
||||
case 'brave':
|
||||
case 'websearch':
|
||||
return 'web';
|
||||
default:
|
||||
return 'none';
|
||||
}
|
||||
}
|
||||
|
||||
function normalizeLegitimacyVerdict(raw: unknown): 'OK' | 'SUS' | 'SLOP' | null {
|
||||
if (typeof raw !== 'string') return null;
|
||||
const upper = raw.toUpperCase();
|
||||
if (upper === 'OK' || upper === 'SUS' || upper === 'SLOP') return upper;
|
||||
return null;
|
||||
}
|
||||
|
||||
function classifyConfidence(input: ClassifyConfidenceInput): ConfidenceLevel {
|
||||
try {
|
||||
const { provider, verifiedAgainstOfficial, legitimacyVerdict } = input;
|
||||
const authority = authorityOf(provider);
|
||||
const verdict = normalizeLegitimacyVerdict(legitimacyVerdict);
|
||||
const groundTruth = verdict === 'OK';
|
||||
|
||||
// SLOP caps everything — checked first
|
||||
if (verdict === 'SLOP') return 'LOW';
|
||||
|
||||
// Ground-truth corroboration + known authority → HIGH
|
||||
if (groundTruth && authority !== 'none') return 'HIGH';
|
||||
|
||||
// Official or scrape provider (authority alone) → MEDIUM
|
||||
if (authority === 'official' || authority === 'scrape') return 'MEDIUM';
|
||||
|
||||
// Ground-truth but unknown provider → MEDIUM
|
||||
if (groundTruth) return 'MEDIUM';
|
||||
|
||||
// Web provider with self-reported verification → MEDIUM
|
||||
if (authority === 'web' && verifiedAgainstOfficial === true) return 'MEDIUM';
|
||||
|
||||
return 'LOW';
|
||||
} catch {
|
||||
return 'LOW';
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Cycle 5: providerAvailability
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function providerAvailability(config?: ProviderAvailabilityConfig): Record<string, boolean> {
|
||||
const cfg = config ?? {};
|
||||
return {
|
||||
context7: true,
|
||||
jina: cfg.jina !== undefined ? Boolean(cfg.jina) : true,
|
||||
websearch: true,
|
||||
exa: Boolean(cfg.exa_search),
|
||||
tavily: Boolean(cfg.tavily_search),
|
||||
brave: Boolean(cfg.brave_search),
|
||||
firecrawl: Boolean(cfg.firecrawl),
|
||||
ref: Boolean(cfg.ref_search),
|
||||
perplexity: Boolean(cfg.perplexity),
|
||||
};
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Lazy-load default store (avoids circular require at module eval time)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
let _defaultStore: ResearchStore | undefined;
|
||||
|
||||
function getDefaultStore(): ResearchStore {
|
||||
if (!_defaultStore) {
|
||||
// eslint-disable-next-line @typescript-eslint/no-require-imports -- lazy default store; tests inject their own
|
||||
_defaultStore = require('./research-store.cjs') as ResearchStore;
|
||||
}
|
||||
return _defaultStore;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Cycle 1–3, 5, 7: planResearch
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function planResearch(options: PlanResearchOptions): PlanResearchResult {
|
||||
const {
|
||||
questions,
|
||||
ecosystem = '',
|
||||
cwd,
|
||||
config,
|
||||
clock = Date,
|
||||
homeDir,
|
||||
store = getDefaultStore(),
|
||||
} = options;
|
||||
|
||||
const availability = providerAvailability(config);
|
||||
|
||||
const items: ResearchItem[] = questions.flatMap((q) => {
|
||||
const { text, kind, library, version } = q;
|
||||
|
||||
// Skip questions without a non-empty string text — emitting an item with
|
||||
// question:undefined / fetch.query:undefined would produce corrupt output.
|
||||
if (typeof text !== 'string' || text.length === 0) {
|
||||
return [];
|
||||
}
|
||||
|
||||
const key = store.researchKey({ ecosystem, library, version, query: text, kind });
|
||||
const res = store.getResearch(cwd, key, { clock, homeDir, kind });
|
||||
|
||||
// Fresh cache hit — no fetch needed
|
||||
if (res.hit && !res.stale) {
|
||||
return { question: text, key, cache: { hit: true, stale: false } };
|
||||
}
|
||||
|
||||
// Determine which waterfall to use
|
||||
const waterfall: string[] =
|
||||
(PROVIDER_WATERFALL as unknown as Record<string, string[]>)[kind] ?? PROVIDER_WATERFALL.web;
|
||||
|
||||
// Pick first available provider
|
||||
const provider = waterfall.find((p) => availability[p] === true) ?? 'websearch';
|
||||
|
||||
const item: ResearchItem = {
|
||||
question: text,
|
||||
key,
|
||||
fetch: { provider, query: text },
|
||||
};
|
||||
|
||||
// Stale hit: include cache info
|
||||
if (res.hit) {
|
||||
item.cache = { hit: true, stale: true };
|
||||
}
|
||||
|
||||
return item;
|
||||
});
|
||||
|
||||
return { items };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Exports
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export = { PROVIDER_WATERFALL, classifyConfidence, providerAvailability, planResearch };
|
||||
252
src/research-store.cts
Normal file
252
src/research-store.cts
Normal file
@@ -0,0 +1,252 @@
|
||||
/**
|
||||
* Research Store Module
|
||||
*
|
||||
* Provides deterministic cache key generation, TTL policy, path resolution,
|
||||
* and JSON-backed put/get operations for research entries.
|
||||
*
|
||||
* ADR-457 build-at-publish: authored as TypeScript .cts → emits .cjs via tsc.
|
||||
*/
|
||||
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import crypto from 'node:crypto';
|
||||
import { platformWriteSync } from './shell-command-projection.cjs';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Constants
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const DAY_MS = 86_400_000;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Types
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
interface ResearchKeyInput {
|
||||
ecosystem?: unknown;
|
||||
library?: unknown;
|
||||
version?: unknown;
|
||||
query?: unknown;
|
||||
kind?: unknown;
|
||||
}
|
||||
|
||||
interface ResearchEntry {
|
||||
content: unknown;
|
||||
source: string;
|
||||
provider: string;
|
||||
confidence: string;
|
||||
fetched_at: string;
|
||||
ttl: number;
|
||||
kind: string;
|
||||
}
|
||||
|
||||
interface GetResult {
|
||||
hit: boolean;
|
||||
stale: boolean;
|
||||
entry: ResearchEntry | null;
|
||||
}
|
||||
|
||||
interface ClockLike {
|
||||
now(): number;
|
||||
}
|
||||
|
||||
interface PutOptions {
|
||||
clock?: ClockLike;
|
||||
homeDir?: string;
|
||||
}
|
||||
|
||||
interface GetOptions {
|
||||
clock?: ClockLike;
|
||||
homeDir?: string;
|
||||
}
|
||||
|
||||
interface PutPayload {
|
||||
content: unknown;
|
||||
source: string;
|
||||
provider: string;
|
||||
confidence: string;
|
||||
kind: string;
|
||||
version?: string;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// researchKey
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function normalize(x: unknown): string {
|
||||
if (x === null || x === undefined) return '';
|
||||
if (typeof x === 'object') return JSON.stringify(x).trim().toLowerCase();
|
||||
// After excluding null, undefined, and object, x can only be a primitive —
|
||||
// cast through number | string | boolean to avoid no-base-to-string on unknown.
|
||||
return `${x as number | string | boolean}`.trim().toLowerCase();
|
||||
}
|
||||
|
||||
function researchKey(input: ResearchKeyInput): string {
|
||||
const parts = {
|
||||
ecosystem: normalize(input.ecosystem),
|
||||
library: normalize(input.library),
|
||||
version: normalize(input.version),
|
||||
query: normalize(input.query),
|
||||
kind: normalize(input.kind),
|
||||
};
|
||||
const serialized = JSON.stringify(parts);
|
||||
return crypto.createHash('sha256').update(serialized).digest('hex');
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// ttlForSource
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function ttlForSource(source: string, confidence: string): number {
|
||||
if (source === 'curated' && confidence === 'HIGH') return 30 * DAY_MS;
|
||||
if (source === 'curated' && confidence === 'MEDIUM') return 7 * DAY_MS;
|
||||
return DAY_MS;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// tierForSource / resolveStorePath
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
const CURATED_SOURCES = new Set(['curated']);
|
||||
|
||||
function tierForSource(source: string): 'user' | 'project' {
|
||||
return CURATED_SOURCES.has(source) ? 'user' : 'project';
|
||||
}
|
||||
|
||||
function resolveStorePath(cwd: string, source: string, { homeDir = os.homedir() }: { homeDir?: string } = {}): string {
|
||||
if (tierForSource(source) === 'user') {
|
||||
return path.join(homeDir, '.gsd', 'research-cache');
|
||||
}
|
||||
return path.join(cwd, '.planning', 'research', '.cache');
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// isValidResearchKey
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Returns true iff key is a valid 64-character lowercase hexadecimal SHA-256
|
||||
* string (the exact shape produced by researchKey). Any other shape —
|
||||
* including path-traversal sequences — is rejected.
|
||||
*/
|
||||
function isValidResearchKey(key: unknown): boolean {
|
||||
return typeof key === 'string' && /^[0-9a-f]{64}$/.test(key);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// putResearch
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function putResearch(
|
||||
cwd: string,
|
||||
key: string,
|
||||
payload: PutPayload,
|
||||
{ clock = Date, homeDir = os.homedir() }: PutOptions = {}
|
||||
): ResearchEntry {
|
||||
// Defense-in-depth: reject any key that is not a 64-char sha256 hex string.
|
||||
if (!isValidResearchKey(key)) {
|
||||
throw new Error('invalid research key');
|
||||
}
|
||||
|
||||
const { content, source, provider, confidence, kind, version } = payload;
|
||||
let ttl = ttlForSource(source, confidence);
|
||||
// Cap TTL when version is blank/missing — a versionless curated entry must not
|
||||
// get the long 30-day window since we can't know if it's still current.
|
||||
if (!version) {
|
||||
ttl = Math.min(ttl, DAY_MS);
|
||||
}
|
||||
const fetched_at = new Date(clock.now()).toISOString();
|
||||
const entry: ResearchEntry = { content, source, provider, confidence, fetched_at, ttl, kind };
|
||||
const dir = resolveStorePath(cwd, source, { homeDir });
|
||||
|
||||
// Belt-and-suspenders: ensure the resolved file path stays inside the store dir.
|
||||
const resolvedDir = path.resolve(dir);
|
||||
const filePath = path.join(dir, `${key}.json`);
|
||||
const resolvedFile = path.resolve(filePath);
|
||||
if (!resolvedFile.startsWith(resolvedDir + path.sep)) {
|
||||
throw new Error('invalid research key');
|
||||
}
|
||||
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
platformWriteSync(filePath, JSON.stringify(entry));
|
||||
return entry;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// getResearch
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function getResearch(cwd: string, key: string, { clock = Date, homeDir = os.homedir() }: GetOptions = {}): GetResult {
|
||||
// Defense-in-depth: reject any key that is not a 64-char sha256 hex string.
|
||||
if (!isValidResearchKey(key)) {
|
||||
return { hit: false, stale: false, entry: null };
|
||||
}
|
||||
|
||||
try {
|
||||
// Search both physical tiers: user (curated) and project (web/etc.)
|
||||
const userDir = path.join(homeDir, '.gsd', 'research-cache');
|
||||
const projectDir = path.join(cwd, '.planning', 'research', '.cache');
|
||||
const tierDirs = [userDir, projectDir];
|
||||
|
||||
interface Candidate {
|
||||
entry: ResearchEntry;
|
||||
stale: boolean;
|
||||
age: number;
|
||||
}
|
||||
|
||||
const candidates: Candidate[] = [];
|
||||
|
||||
for (const dir of tierDirs) {
|
||||
const resolvedDir = path.resolve(dir);
|
||||
const filePath = path.join(dir, `${key}.json`);
|
||||
// Belt-and-suspenders: ensure path stays inside tier dir
|
||||
if (!path.resolve(filePath).startsWith(resolvedDir + path.sep)) continue;
|
||||
|
||||
if (!fs.existsSync(filePath)) continue;
|
||||
|
||||
let entry: ResearchEntry;
|
||||
try {
|
||||
entry = JSON.parse(fs.readFileSync(filePath, 'utf8')) as ResearchEntry;
|
||||
} catch {
|
||||
// Corrupt file in this tier — skip it
|
||||
continue;
|
||||
}
|
||||
|
||||
// Finding 3: validate entry metadata shape before accepting as a candidate.
|
||||
// An entry with missing/invalid fetched_at or ttl must be treated as a miss.
|
||||
const parsedFetchedAt = Date.parse(entry.fetched_at);
|
||||
if (!Number.isFinite(parsedFetchedAt)) continue;
|
||||
if (
|
||||
typeof entry.ttl !== 'number' ||
|
||||
!Number.isFinite(entry.ttl) ||
|
||||
entry.ttl <= 0
|
||||
) continue;
|
||||
|
||||
const age = clock.now() - parsedFetchedAt;
|
||||
const stale = age > entry.ttl;
|
||||
candidates.push({ entry, stale, age });
|
||||
}
|
||||
|
||||
if (candidates.length === 0) {
|
||||
return { hit: false, stale: false, entry: null };
|
||||
}
|
||||
|
||||
// Prefer: non-stale over stale; among same-staleness, lowest age (most recent)
|
||||
candidates.sort((a, b) => {
|
||||
if (a.stale !== b.stale) return a.stale ? 1 : -1; // non-stale first
|
||||
return a.age - b.age; // lower age (more recent) first
|
||||
});
|
||||
|
||||
const best = candidates[0];
|
||||
return { hit: true, stale: best.stale, entry: best.entry };
|
||||
} catch {
|
||||
return { hit: false, stale: false, entry: null };
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Exports
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export = { isValidResearchKey, researchKey, ttlForSource, tierForSource, resolveStorePath, putResearch, getResearch };
|
||||
Reference in New Issue
Block a user