Commit Graph

3 Commits

Author SHA1 Message Date
Colin
cd5db1f8db test(suites): seed security/slow/integration suites via measured retags
Renames (git mv) with all references updated (ci-test-scope RULES,
windows-parity allowlist, test-file-count allowlist, docs in 6 locales):

- 5 scanner tests -> *.security.test.cjs — the 'Run security tests' CI step
  ran zero files since the suite taxonomy landed; it is now honest.
- graphify-auto-update -> *.slow.test.cjs (36s, slowest file in the suite;
  e2e gsd-tools spawns) — runs on full-matrix lanes and push to next.
- installer-migration-install-integration -> *.integration.test.cjs
  (13s; an integration test by its own name).

Coverage gate measured after retags: 88.55% lines (gate 70%).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-09 23:50:41 -04:00
Tom Boucher
a28dcec981 chore(#597): replace count-based ratchet guards with AST lint + named-set allowlists (#603)
The windows-test-parity ratchet greps test source for fs.rmSync-without-
maxRetries (and six other Windows-portability anti-patterns), failing when an
integer offender COUNT exceeds a frozen baseline (rmSync: 95). A count ratchet
is a Goodhart metric: fixing one offender and adding another keeps the count
constant, so a new defect slips through green. Replace it — and every other
count ratchet in the repo — with a layered, masking-proof design.

Behavioral seam test
- tests/helpers-cleanup.test.cjs proves helpers.cleanup() carries the Windows
  EBUSY retry budget. cleanup() delegates retries to Node's fs.rmSync via
  maxRetries (it owns no loop), so the test asserts the option contract
  (recursive/force/maxRetries>0/retryDelay>0) + real-FS removal + the cwd-guard,
  rather than a loop that does not exist. The EBUSY risk is now tested ONCE at
  the helper, not approximated textually at every call site.

Write-time ESLint rule (AST-accurate, replaces the grep)
- eslint-rules/no-raw-rmsync-in-tests.cjs (error in tests/**/*.test.cjs) bans
  raw fs.rmSync, steering to cleanup(). Catches member, computed (fs['rmSync']),
  destructured and aliased forms; escape hatch is inline
  `// eslint-disable-next-line local/no-raw-rmsync-in-tests -- <reason>` only.
- Migrated 336 raw fs.rmSync teardown calls across ~116 test files to cleanup().
  ~18 genuinely load-bearing sites (mid-test SUT/fault-injection removals,
  error-swallowing or name-colliding local teardown helpers) keep the raw call
  with an inline eslint-disable + reason.

Shared anti-ratchet primitive
- scripts/lib/allowlist-ratchet.cjs:
  - assertWithinAllowlist: fails on NOVEL ids (new offender introduced) AND on
    STALE ids (a known offender was fixed but not pruned) — identity, not count,
    and a ratchet DOWN toward zero.
  - assertTightCeiling: a size/length budget whose ceiling must stay within a
    grace band of the high-water mark, so budgets may only tighten, never creep.

Ratchets converted onto the primitive
- windows-test-parity-guard.test.cjs: rmSync rule deleted (now ESLint-enforced);
  the remaining six patterns moved from integer baselines to named-set
  allowlists with ratchet-down.
- scripts/lint-test-file-count.{cjs,allowlist.json}: per-module integer counts →
  named filename sets (closes the swap-a-file-keep-the-count blind spot); a
  module dropping under cap now FAILS to force pruning its allowlist entry.
- enh-2790 skill-count `<= 63` → named skill allowlist (ratchets toward ~58).

Size budgets hardened (tighten-only)
- agent-size / workflow-size / feat-3039 help-tiered: ceilings lowered to the
  current high-water mark and an assertTightCeiling anti-creep check added per
  tier. Fixed external-contract limits (description ≤100 chars, agent ≤100 KB)
  are intentionally left as-is — they are not grandfathered creeping budgets.

No user-facing behavior change (tests + tooling only); no USER_FACING_PREFIXES
touched, so no changeset fragment is required.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-01 22:43:49 -04:00
Tom Boucher
ca3be82f71 fix(3597): clear residual Windows test failures + add ratchet lint guard
Two more diagnostic passes (clusters J: residuals in already-touched files,
K: 12 untouched files) plus a production-code path fix and a new
ratchet-style lint guard.

## Test-only fixes (14 files)

bug-1736, bug-2248, bug-2698 — replace inline 1s-budget rmSync with the
shared cleanup() helper (5s budget, 20×250ms retries). The earlier
inline maxRetries:10 / retryDelay:100 wasn't enough to absorb Windows
Defender's deferred-scan handle hold on cold runners.

bug-2256, skill-manifest — also override USERPROFILE alongside HOME in
beforeEach/runGsdTools calls. os.homedir() reads USERPROFILE on win32,
so HOME-only stubs leak the runner's real home into the SUT.

bug-2784, bug-3608, enh-2500, enh-2790, few-shot-calibration,
gsd-settings-advanced — CRLF tolerance: literal \n in regexes against
file content (frontmatter anchors, bash-fence regex, multi-line
numbered-list captures, awk-block extractors) becomes \r?\n; split('\n')
becomes split(/\r?\n/). Windows checkout with autocrlf=true puts \r
before every \n; .+ doesn't match \r in JS regex by default.

bug-2966 — three-part fix to extractStepRun (CRLF split), awk regex
(\r?\n), and conflict-marker parser (rawLine + \r$ strip).

bug-2969, config — normalize separators on test assertions where the
SUT correctly emits \ on win32 but the test compares against /.

prompt-injection-scan — normalize relPath via replace(/\\/g, '/') before
ALLOWLIST.has() lookup. ALLOWLIST keys are POSIX; path.relative returns
backslashes on win32 → falsely scans allowlisted security module → trips
the boundary-tag detector on its own legitimate detection code.

prune-orphaned-worktrees — use the existing canonicalPath +
listedWorktreePaths(repoDir).has(...) helpers instead of substring
matching the raw path. git stores long-form canonical paths
(runneradmin), but mkdtempSync returns 8.3 short-form (RUNNER~1) on
Windows runners; plain string compare misses every entry.

## Production-code fix (1 file)

get-shit-done/bin/lib/init.cjs — bug-3491 in_nested_subdir computation
canonicalizes both worktreeRoot and cwd via fs.realpathSync.native +
path.relative before declaring "nested." Windows runner cwd (8.3 short
name) vs git's --show-toplevel (long form, forward slashes) made the
raw string compare always say true even at the worktree root, breaking
the "init new-project at worktree root" subtest.

## New ratchet lint guard

tests/windows-test-parity-guard.test.cjs — scans tests/ for 7
anti-patterns that drove the Windows failure clusters. Each rule has a
baseline count snapshot from this PR; the test fails when a new
occurrence appears (count grows above baseline), ratcheting down as
existing offenders are fixed. Patterns covered:

  G1 split('\n') after readFileSync (use /\r?\n/)
  G2 ```bash\n fence regex (use ```bash\r?\n)
  G3 ^---\n frontmatter anchor (use ^---\r?\n)
  G4 hardcoded "/tmp/..." literal passed to fs.* (use os.tmpdir())
  G5 bare 'npm' to execFileSync without {shell:true} on win32
  G6 process.env.HOME stub with no USERPROFILE
  G7 fs.rmSync({recursive,force}) without maxRetries

Future Windows-parity regressions get caught at PR time rather than
five iterations into a CI loop.

Validated: holodeck (ubuntu docker) 11232/0 pass (count +8 = the 7
new ratchet tests + parent describe).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-16 12:39:02 -04:00