Files
msd-core/get-shit-done/workflows/map-codebase.md
Tom Boucher 1a694fcac3 feat: auto-remap codebase after significant phase execution (closes #2003) (#2605)
* feat: auto-remap codebase after significant phase execution (#2003)

Adds a post-phase structural drift detector that compares the committed tree
against `.planning/codebase/STRUCTURE.md` and either warns or auto-remaps
the affected subtrees when drift exceeds a configurable threshold.

## Summary
- New `bin/lib/drift.cjs` — pure detector covering four drift categories:
  new directories outside mapped paths, new barrel exports at
  `(packages|apps)/*/src/index.*`, new migration files, and new route
  modules. Prioritizes the most-specific category per file.
- New `verify codebase-drift` CLI subcommand + SDK handler, registered as
  `gsd-sdk query verify.codebase-drift`.
- New `codebase_drift_gate` step in `execute-phase` between
  `schema_drift_gate` and `verify_phase_goal`. Non-blocking by contract —
  any error logs and the phase continues.
- Two new config keys: `workflow.drift_threshold` (int, default 3) and
  `workflow.drift_action` (`warn` | `auto-remap`, default `warn`), with
  enum/integer validation in `config-set`.
- `gsd-codebase-mapper` learns an optional `--paths <p1,p2,...>` scope hint
  for incremental remapping; agent/workflow docs updated.
- `last_mapped_commit` lives in YAML frontmatter on each
  `.planning/codebase/*.md` file; `readMappedCommit`/`writeMappedCommit`
  round-trip helpers ship in `drift.cjs`.

## Tests
- 55 new tests in `tests/drift-detection.test.cjs` covering:
  classification, threshold gating at 2/3/4 elements, warn vs. auto-remap
  routing, affected-path scoping, `--paths` sanitization (traversal,
  absolute, shell metacharacter rejection), frontmatter round-trip,
  defensive paths (missing STRUCTURE.md, malformed input, non-git repos),
  CLI JSON output, and documentation parity.
- Full suite: 5044 pass / 0 fail.

## Documentation
- `docs/CONFIGURATION.md` — rows for both new keys.
- `docs/ARCHITECTURE.md` — section on the post-execute drift gate.
- `docs/AGENTS.md` — `--paths` flag on `gsd-codebase-mapper`.
- `docs/USER-GUIDE.md` — user-facing behavior note + toggle commands.
- `docs/FEATURES.md` — new 27a section with REQ-DRIFT-01..06.
- `docs/INVENTORY.md` + `docs/INVENTORY-MANIFEST.json` — drift.cjs listed.
- `get-shit-done/workflows/execute-phase.md` — `codebase_drift_gate` step.
- `get-shit-done/workflows/map-codebase.md` — `parse_paths_flag` step.
- `agents/gsd-codebase-mapper.md` — `--paths` directive under parse_focus.

## Design decisions
- **Frontmatter over sidecar JSON** for `last_mapped_commit`: keeps the
  baseline attached to the file, survives git moves, survives per-doc
  regeneration, no extra file lifecycle.
- **Substring match against STRUCTURE.md** for `isPathMapped`: the map is
  free-form markdown, not a structured manifest; any mention of a path
  prefix counts as "mapped territory". Cheap, no parser, zero false
  negatives on reasonable maps.
- **Category priority migration > route > barrel > new_dir** so a file
  matching multiple rules counts exactly once at the most specific level.
- **Empty-tree SHA fallback** (`4b825dc6…`) when `last_mapped_commit` is
  absent — semantically correct (no baseline means everything is drift)
  and deterministic across repos.
- **Four layers of non-blocking** — detector try/catch, CLI try/catch, SDK
  handler try/catch, and workflow `|| echo` shell fallback. Any single
  layer failing still returns a valid skipped result.
- **SDK handler delegates to `gsd-tools.cjs`** rather than re-porting the
  detector to TypeScript, keeping drift logic in one canonical place.

Closes #2003

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(mapper): tag --paths fenced block as text (CodeRabbit MD040)

Comment 3127255172.

* docs(config): use /gsd- dash command syntax in drift_action row (CodeRabbit)

Comment 3127255180. Matches the convention used by every other command
reference in docs/CONFIGURATION.md.

* fix(execute-phase): initialize AGENT_SKILLS_MAPPER + tag fenced blocks

Two CodeRabbit findings on the auto-remap branch of the drift gate:

- 3127255186 (must-fix): the mapper Task prompt referenced
  ${AGENT_SKILLS_MAPPER} but only AGENT_SKILLS (for gsd-executor) is
  loaded at init_context (line 72). Without this fix the literal
  placeholder string would leak into the spawned mapper's prompt.
  Add an explicit gsd-sdk query agent-skills gsd-codebase-mapper step
  right before the Task spawn.
- 3127255183: tag the warn-message and Task() fenced code blocks as
  text to satisfy markdownlint MD040.

* docs(map-codebase): wire PATH_SCOPE_HINT through every mapper prompt

CodeRabbit (review id 4158286952, comment 3127255190) flagged that the
parse_paths_flag step defined incremental-remap semantics but did not
inject a normalized variable into the spawn_agents and sequential_mapping
mapper prompts, so incremental remap could silently regress to a
whole-repo scan.

- Define SCOPED_PATHS / PATH_SCOPE_HINT in parse_paths_flag.
- Inject ${PATH_SCOPE_HINT} into all four spawn_agents Task prompts.
- Document the same scope contract for sequential_mapping mode.

* fix(drift): writeMappedCommit tolerates missing target file

CodeRabbit (review id 4158286952, drift.cjs:349-355 nitpick) noted that
readMappedCommit returns null on ENOENT but writeMappedCommit threw — an
asymmetry that breaks first-time stamping of a freshly produced doc that
the caller has not yet written.

- Catch ENOENT on the read; treat absent file as empty content.
- Add a regression test that calls writeMappedCommit on a non-existent
  path and asserts the file is created with correct frontmatter.
  Test was authored to fail before the fix (ENOENT) and passes after.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-22 21:21:44 -04:00

442 lines
15 KiB
Markdown

<purpose>
Orchestrate parallel codebase mapper agents to analyze codebase and produce structured documents in .planning/codebase/
Each agent has fresh context, explores a specific focus area, and **writes documents directly**. The orchestrator only receives confirmation + line counts, then writes a summary.
Output: .planning/codebase/ folder with 7 structured documents about the codebase state.
</purpose>
<available_agent_types>
Valid GSD subagent types (use exact names — do not fall back to 'general-purpose'):
- gsd-codebase-mapper — Maps project structure and dependencies
</available_agent_types>
<philosophy>
**Why dedicated mapper agents:**
- Fresh context per domain (no token contamination)
- Agents write documents directly (no context transfer back to orchestrator)
- Orchestrator only summarizes what was created (minimal context usage)
- Faster execution (agents run simultaneously)
**Document quality over length:**
Include enough detail to be useful as reference. Prioritize practical examples (especially code patterns) over arbitrary brevity.
**Always include file paths:**
Documents are reference material for Claude when planning/executing. Always include actual file paths formatted with backticks: `src/services/user.ts`.
</philosophy>
<process>
<step name="parse_paths_flag" priority="first">
Parse an optional `--paths <p1,p2,...>` argument. When supplied (by the
post-execute codebase-drift gate in `/gsd:execute-phase` or by a user running
`/gsd:map-codebase --paths apps/accounting,packages/ui`), the workflow
operates in **incremental-remap mode**:
- Pass `--paths <p1>,<p2>,...` through to each spawned `gsd-codebase-mapper`
agent's prompt. Agents scope their Glob/Grep/Bash exploration to the listed
repo-relative prefixes only — no whole-repo scan.
- Reject path values that contain `..`, start with `/`, or include shell
metacharacters (`;`, `` ` ``, `$`, `&`, `|`, `<`, `>`). If all provided
paths are invalid, fall back to a normal whole-repo run.
- On write, each mapper stamps `last_mapped_commit: <HEAD sha>` into the YAML
frontmatter of every document it produces (see `bin/lib/drift.cjs:writeMappedCommit`).
**Explicit contract — propagate `--paths` through a single normalized
variable.** Downstream steps (`spawn_agents`, `sequential_mapping`, and any
Task-mode prompt construction) MUST use `${PATH_SCOPE_HINT}` to ensure every
mapper receives the same deterministic scope. Without this contract
incremental-remap can silently regress to a whole-repo scan.
```bash
# Validated, comma-separated paths (empty if --paths absent or all rejected):
SCOPED_PATHS="<validated paths or empty>"
if [ -n "$SCOPED_PATHS" ]; then
PATH_SCOPE_HINT="--paths $SCOPED_PATHS"
else
PATH_SCOPE_HINT=""
fi
```
All mapper prompts built later in this workflow MUST include
`${PATH_SCOPE_HINT}` (expanded to empty when full-repo mode is in effect).
When `--paths` is absent, behave exactly as before: full-repo scan, all 7
documents refreshed.
</step>
<step name="init_context" priority="first">
Load codebase mapping context:
```bash
INIT=$(gsd-sdk query init.map-codebase)
if [[ "$INIT" == @file:* ]]; then INIT=$(cat "${INIT#@file:}"); fi
AGENT_SKILLS_MAPPER=$(gsd-sdk query agent-skills gsd-codebase-mapper 2>/dev/null)
```
Extract from init JSON: `mapper_model`, `commit_docs`, `codebase_dir`, `existing_maps`, `has_maps`, `codebase_dir_exists`, `subagent_timeout`, `date`.
</step>
<step name="check_existing">
Check if .planning/codebase/ already exists using `has_maps` from init context.
If `codebase_dir_exists` is true:
```bash
ls -la .planning/codebase/
```
**If exists:**
```
.planning/codebase/ already exists with these documents:
[List files found]
What's next?
1. Refresh - Delete existing and remap codebase
2. Update - Keep existing, only update specific documents
3. Skip - Use existing codebase map as-is
```
Wait for user response.
If "Refresh": Delete .planning/codebase/, continue to create_structure
If "Update": Ask which documents to update, continue to spawn_agents (filtered)
If "Skip": Exit workflow
**If doesn't exist:**
Continue to create_structure.
</step>
<step name="create_structure">
Create .planning/codebase/ directory:
```bash
mkdir -p .planning/codebase
```
**Expected output files:**
- STACK.md (from tech mapper)
- INTEGRATIONS.md (from tech mapper)
- ARCHITECTURE.md (from arch mapper)
- STRUCTURE.md (from arch mapper)
- CONVENTIONS.md (from quality mapper)
- TESTING.md (from quality mapper)
- CONCERNS.md (from concerns mapper)
Continue to spawn_agents.
</step>
<step name="detect_runtime_capabilities">
Before spawning agents, detect whether the current runtime supports the `Task` tool for subagent delegation.
**How to detect:** Check if you have access to a `Task` tool (may be capitalized as `Task` or lowercase as `task` depending on runtime). If you do NOT have a `Task`/`task` tool (or only have tools like `browser_subagent` which is for web browsing, NOT code analysis):
→ **Skip `spawn_agents` and `collect_confirmations`** — go directly to `sequential_mapping` instead.
**CRITICAL:** Never use `browser_subagent` or `Explore` as a substitute for `Task`. The `browser_subagent` tool is exclusively for web page interaction and will fail for codebase analysis. If `Task` is unavailable, perform the mapping sequentially in-context.
</step>
<step name="spawn_agents" condition="Task tool is available">
Spawn 4 parallel gsd-codebase-mapper agents.
Use Task tool with `subagent_type="gsd-codebase-mapper"`, `model="{mapper_model}"`, and `run_in_background=true` for parallel execution.
**CRITICAL:** Use the dedicated `gsd-codebase-mapper` agent, NOT `Explore` or `browser_subagent`. The mapper agent writes documents directly.
**Agent 1: Tech Focus**
```
Task(
subagent_type="gsd-codebase-mapper",
model="{mapper_model}",
run_in_background=true,
description="Map codebase tech stack",
prompt="Focus: tech
Today's date: {date}
Analyze this codebase for technology stack and external integrations.
Write these documents to .planning/codebase/:
- STACK.md - Languages, runtime, frameworks, dependencies, configuration
- INTEGRATIONS.md - External APIs, databases, auth providers, webhooks
IMPORTANT: Use {date} for all [YYYY-MM-DD] date placeholders in documents.
Scope: ${PATH_SCOPE_HINT:-(full repo)} — when --paths is supplied, restrict exploration to those prefixes only.
Explore thoroughly. Write documents directly using templates. Return confirmation only.
${AGENT_SKILLS_MAPPER}"
)
```
**Agent 2: Architecture Focus**
```
Task(
subagent_type="gsd-codebase-mapper",
model="{mapper_model}",
run_in_background=true,
description="Map codebase architecture",
prompt="Focus: arch
Today's date: {date}
Analyze this codebase architecture and directory structure.
Write these documents to .planning/codebase/:
- ARCHITECTURE.md - Pattern, layers, data flow, abstractions, entry points
- STRUCTURE.md - Directory layout, key locations, naming conventions
IMPORTANT: Use {date} for all [YYYY-MM-DD] date placeholders in documents.
Scope: ${PATH_SCOPE_HINT:-(full repo)} — when --paths is supplied, restrict exploration to those prefixes only.
Explore thoroughly. Write documents directly using templates. Return confirmation only.
${AGENT_SKILLS_MAPPER}"
)
```
**Agent 3: Quality Focus**
```
Task(
subagent_type="gsd-codebase-mapper",
model="{mapper_model}",
run_in_background=true,
description="Map codebase conventions",
prompt="Focus: quality
Today's date: {date}
Analyze this codebase for coding conventions and testing patterns.
Write these documents to .planning/codebase/:
- CONVENTIONS.md - Code style, naming, patterns, error handling
- TESTING.md - Framework, structure, mocking, coverage
IMPORTANT: Use {date} for all [YYYY-MM-DD] date placeholders in documents.
Scope: ${PATH_SCOPE_HINT:-(full repo)} — when --paths is supplied, restrict exploration to those prefixes only.
Explore thoroughly. Write documents directly using templates. Return confirmation only.
${AGENT_SKILLS_MAPPER}"
)
```
**Agent 4: Concerns Focus**
```
Task(
subagent_type="gsd-codebase-mapper",
model="{mapper_model}",
run_in_background=true,
description="Map codebase concerns",
prompt="Focus: concerns
Today's date: {date}
Analyze this codebase for technical debt, known issues, and areas of concern.
Write this document to .planning/codebase/:
- CONCERNS.md - Tech debt, bugs, security, performance, fragile areas
IMPORTANT: Use {date} for all [YYYY-MM-DD] date placeholders in documents.
Scope: ${PATH_SCOPE_HINT:-(full repo)} — when --paths is supplied, restrict exploration to those prefixes only.
Explore thoroughly. Write document directly using template. Return confirmation only.
${AGENT_SKILLS_MAPPER}"
)
```
Continue to collect_confirmations.
</step>
<step name="collect_confirmations">
Wait for all 4 agents to complete using TaskOutput tool.
**For each agent task_id returned by the Agent tool calls above:**
```
TaskOutput tool:
task_id: "{task_id from Agent result}"
block: true
timeout: {subagent_timeout from init context, default 300000}
```
> The timeout is configurable via `workflow.subagent_timeout` in `.planning/config.json` (milliseconds). Default: 300000 (5 minutes). Increase for large codebases or slower models.
Call TaskOutput for all 4 agents in parallel (single message with 4 TaskOutput calls).
Once all TaskOutput calls return, read each agent's output file to collect confirmations.
**Expected confirmation format from each agent:**
```
## Mapping Complete
**Focus:** {focus}
**Documents written:**
- `.planning/codebase/{DOC1}.md` ({N} lines)
- `.planning/codebase/{DOC2}.md` ({N} lines)
Ready for orchestrator summary.
```
**What you receive:** Just file paths and line counts. NOT document contents.
If any agent failed, note the failure and continue with successful documents.
Continue to verify_output.
</step>
<step name="sequential_mapping" condition="Task tool is NOT available (e.g. Antigravity, Gemini CLI, Codex)">
When the `Task` tool is unavailable, perform codebase mapping sequentially in the current context. This replaces `spawn_agents` and `collect_confirmations`.
**IMPORTANT:** Do NOT use `browser_subagent`, `Explore`, or any browser-based tool. Use only file system tools (Read, Bash, Write, Grep, Glob, list_dir, view_file, grep_search, or equivalent tools available in your runtime).
**IMPORTANT:** Use `{date}` from init context for all `[YYYY-MM-DD]` date placeholders in documents. NEVER guess the date.
**SCOPE:** When `${PATH_SCOPE_HINT}` is non-empty (i.e. `--paths` was supplied), restrict every pass below to the validated path prefixes in `${SCOPED_PATHS}`. Do NOT scan files outside those prefixes. When `${PATH_SCOPE_HINT}` is empty, perform a full-repo scan.
Perform all 4 mapping passes sequentially:
**Pass 1: Tech Focus**
- Explore package.json/Cargo.toml/go.mod/requirements.txt, config files, dependency trees
- Write `.planning/codebase/STACK.md` — Languages, runtime, frameworks, dependencies, configuration
- Write `.planning/codebase/INTEGRATIONS.md` — External APIs, databases, auth providers, webhooks
**Pass 2: Architecture Focus**
- Explore directory structure, entry points, module boundaries, data flow
- Write `.planning/codebase/ARCHITECTURE.md` — Pattern, layers, data flow, abstractions, entry points
- Write `.planning/codebase/STRUCTURE.md` — Directory layout, key locations, naming conventions
**Pass 3: Quality Focus**
- Explore code style, error handling patterns, test files, CI config
- Write `.planning/codebase/CONVENTIONS.md` — Code style, naming, patterns, error handling
- Write `.planning/codebase/TESTING.md` — Framework, structure, mocking, coverage
**Pass 4: Concerns Focus**
- Explore TODOs, known issues, fragile areas, security patterns
- Write `.planning/codebase/CONCERNS.md` — Tech debt, bugs, security, performance, fragile areas
Use the same document templates as the `gsd-codebase-mapper` agent. Include actual file paths formatted with backticks.
Continue to verify_output.
</step>
<step name="verify_output">
Verify all documents created successfully:
```bash
ls -la .planning/codebase/
wc -l .planning/codebase/*.md
```
**Verification checklist:**
- All 7 documents exist
- No empty documents (each should have >20 lines)
If any documents missing or empty, note which agents may have failed.
Continue to scan_for_secrets.
</step>
<step name="scan_for_secrets">
**CRITICAL SECURITY CHECK:** Scan output files for accidentally leaked secrets before committing.
Run secret pattern detection:
```bash
# Check for common API key patterns in generated docs
grep -E '(sk-[a-zA-Z0-9]{20,}|sk_live_[a-zA-Z0-9]+|sk_test_[a-zA-Z0-9]+|ghp_[a-zA-Z0-9]{36}|gho_[a-zA-Z0-9]{36}|glpat-[a-zA-Z0-9_-]+|AKIA[A-Z0-9]{16}|xox[baprs]-[a-zA-Z0-9-]+|-----BEGIN.*PRIVATE KEY|eyJ[a-zA-Z0-9_-]+\.eyJ[a-zA-Z0-9_-]+\.)' .planning/codebase/*.md 2>/dev/null && SECRETS_FOUND=true || SECRETS_FOUND=false
```
**If SECRETS_FOUND=true:**
```
⚠️ SECURITY ALERT: Potential secrets detected in codebase documents!
Found patterns that look like API keys or tokens in:
[show grep output]
This would expose credentials if committed.
**Action required:**
1. Review the flagged content above
2. If these are real secrets, they must be removed before committing
3. Consider adding sensitive files to Claude Code "Deny" permissions
Pausing before commit. Reply "safe to proceed" if the flagged content is not actually sensitive, or edit the files first.
```
Wait for user confirmation before continuing to commit_codebase_map.
**If SECRETS_FOUND=false:**
Continue to commit_codebase_map.
</step>
<step name="commit_codebase_map">
Commit the codebase map:
```bash
gsd-sdk query commit "docs: map existing codebase" .planning/codebase/*.md
```
Continue to offer_next.
</step>
<step name="offer_next">
Present completion summary and next steps.
**Get line counts:**
```bash
wc -l .planning/codebase/*.md
```
**Output format:**
```
Codebase mapping complete.
Created .planning/codebase/:
- STACK.md ([N] lines) - Technologies and dependencies
- ARCHITECTURE.md ([N] lines) - System design and patterns
- STRUCTURE.md ([N] lines) - Directory layout and organization
- CONVENTIONS.md ([N] lines) - Code style and patterns
- TESTING.md ([N] lines) - Test structure and practices
- INTEGRATIONS.md ([N] lines) - External services and APIs
- CONCERNS.md ([N] lines) - Technical debt and issues
---
## ▶ Next Up — [${PROJECT_CODE}] ${PROJECT_TITLE}
**Initialize project** — use codebase context for planning
`/clear` then:
`/gsd:new-project`
---
**Also available:**
- Re-run mapping: `/gsd:map-codebase`
- Review specific file: `cat .planning/codebase/STACK.md`
- Edit any document before proceeding
---
```
End workflow.
</step>
</process>
<success_criteria>
- .planning/codebase/ directory created
- If Task tool available: 4 parallel gsd-codebase-mapper agents spawned with run_in_background=true
- If Task tool NOT available: 4 sequential mapping passes performed inline (never using browser_subagent)
- All 7 codebase documents exist
- No empty documents (each should have >20 lines)
- Clear completion summary with line counts
- User offered clear next steps in GSD style
</success_criteria>