f970c09bf3f9560eba84f42b9db8a54537359b2e
2 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Tom Boucher
|
835dd6ab44 |
test(3596): adversarial security/prompt-injection abuse suite (#3654)
* test(3596): adversarial security/prompt-injection abuse suite Adds `tests/security-prompt-injection.test.cjs` and a fixtures directory at `tests/fixtures/adversarial/security/` covering the attack classes enumerated in #3596: - Command substitution / backticks / heredoc payloads in workstream names — sentinel-file probes prove no shell is spawned, slugifier neutralises the input. - Path traversal through `--ws` and slash-bearing workstream names — rejected with structured `--json-errors` payload, no stack trace, no filesystem mutation outside the project root. - Fake `<system>` / `[SYSTEM]` / `<<SYS>>` / `[INST]` boundary tags — sanitizeForPrompt neutralises every form; structural negative property locked across all six styles in one place. - Zero-width / bidi-override codepoints — stripped per the documented codepoint set; asserted via codePoint inspection, not regex literals. - Hostile read of CONTEXT.md / PLAN.md / ROADMAP.md fixtures — `gsd-read-injection-scanner.js` surfaces the advisory; excluded paths and non-Read tools stay silent; malformed JSON does not crash the hook. - Hostile write of `.planning/` files — `gsd-prompt-guard.js` emits a `PreToolUse` advisory; non-Write/Edit tools stay silent. - Fake `ghp_*` / `sk-*` env tokens — never echoed in CLI stdout or stderr under hostile inputs; covered under `// allow-test-rule: structural-regression-guard` because the only way to assert byte-level absence is `.includes(token)` against the captured streams. - `validatePath`, `validateShellArg`, `validatePhaseNumber`, `validateFieldName` — focused negative-input contract pins. Pinned behavior gaps (documented, NOT fixed in this PR): - `<instructions>` is intentionally whitelisted by both the scanner and the sanitiser (GSD's own prompt scaffolding). Two REGRESSION GUARD tests lock that contract. - The current `scanForInjection` does NOT flag malicious markdown links (javascript:/data:/embedded-credentials URLs). PINNED with negative-proof so any future scope extension fails the assertion and forces a deliberate update to the acceptance map. - `prompt-builder.ts` does not yet wrap plan/context markdown in an "untrusted data" envelope. That seam lives on the TS side and is covered by `sdk/src/prompt-builder.test.ts`; out of scope for a CJS test file. Mentioned in the file header. Verification: - `node --test tests/security-prompt-injection.test.cjs` → 73 tests pass. - `node scripts/lint-no-source-grep.cjs` → 0 violations across 546 test files (one `allow-test-rule: structural-regression-guard` annotation on this file for the token-absence assertions). - `node scripts/run-tests.cjs` → 9730 tests pass, 0 fail. Refs #3596 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(3596): allow adversarial fixtures in scan + harden graphify status parse * fix(3596): skip adversarial security fixtures in secret scan --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Tom Boucher
|
b1317633db |
test(3594): adversarial parser fixtures + frontmatter/roadmap matrix + property-style suite (#3633)
* test(3594): adversarial parser fixtures + frontmatter/roadmap matrix + property-style suite
Lands the adversarial parser-input corpus that CONTRIBUTING.md
§"QA Matrix Requirements / Parser and project-file inputs" and
TEST-EXAMPLES.md §"Parser Adversarial Fixtures" describe.
New tests/fixtures/adversarial/ layout:
frontmatter/
duplicate-keys.md — same key twice (collapses last-wins)
crlf-mixed.md — CRLF endings throughout
unclosed-block.md — `---` open with no close
unicode-keys-and-values.md — non-ASCII + emoji + Greek
null-byte-value.md — U+0000 in a value
huge-bounded.md — 2000-item array, ~30KB
roadmap/
phase-heading-inside-fenced-code.md — #2787 fence shadowing
nested-fenced-code.md — outer + inner ``` blocks
unicode-phase-titles.md — JP / Greek / emoji titles
repeated-phase-ids.md — phase 1 declared twice
decimal-phase-mixed.md — 2 vs 2.1 vs 2.10 vs 21
markdown-headings-inside-html-comment.md — comment shadowing
Test files (all node:test, no try/finally in test bodies, no source-grep,
no raw-text matching on stdout/file content):
tests/feat-3594-parser-adversarial-frontmatter.test.cjs (12 tests)
Loads each fixture, pins parser invariants on extractFrontmatter()
return shape. Cross-corpus "does not throw on any fixture" sweep.
tests/feat-3594-parser-adversarial-roadmap.test.cjs (18 tests)
Loads each fixture into a temp project's .planning/ROADMAP.md and
drives `gsd-tools roadmap get-phase <N>` via the runCli harness
introduced by #3593. Asserts on the typed JSON payload.
tests/feat-3594-parser-property-style.test.cjs (2 tests)
Deterministic mulberry32 PRNG generates 500 malformed-ish
frontmatter inputs per test. Pins (a) extractFrontmatter is total
over the corpus (no null-deref TypeError, always returns a plain
object on success), (b) the suite completes well under 2 seconds
(quadratic-regression guard).
Known-open bugs surfaced and pinned (intentionally NOT fixed in this
PR — separate issues warranted):
- CJS roadmap parser matches `## Phase N:` headings inside fenced
code blocks (the SDK parser tracks fences per the #2787 comment in
sdk/src/query/roadmap.ts but the CJS path has not caught up).
- CJS roadmap parser matches `## Phase N:` headings inside HTML
comments.
Both are documented in-test with the "currently STILL matches it
(open: needs <fix>)" naming pattern so the day the production fix
lands, flipping the assertion from `found: true` to `found: false` is
the regression guard.
Test totals:
- 32 new feat-3594-* tests (12 frontmatter + 18 roadmap + 2 property)
- 108/108 pass when running together with the pre-existing
frontmatter.test.cjs + roadmap.test.cjs suites (76 of theirs).
Closes #3594
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(3594): use Fisher-Yates shuffle for deterministic seeded inputs
Replaces `arr.sort(() => rng() - 0.5)` with a Fisher-Yates shuffle
driven by the supplied PRNG. The sort-based shuffle is non-transitive:
V8's TimSort behavior on non-transitive comparators is engine-defined,
so the same seed produced different orderings across Node versions —
undermining the test's stated reproducibility guarantee.
Fisher-Yates is O(n), transitive (no comparator at all), and consumes
exactly n-1 RNG values in a fixed order. The mulberry32 seed now
determines the input sequence end-to-end.
Codex review on PR #3633.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|