diff --git a/.planning/phases/04-semantic-intelligence/04-02-PLAN.md b/.planning/phases/04-semantic-intelligence/04-02-PLAN.md index 134f66245..23074e0dc 100644 --- a/.planning/phases/04-semantic-intelligence/04-02-PLAN.md +++ b/.planning/phases/04-semantic-intelligence/04-02-PLAN.md @@ -13,6 +13,7 @@ must_haves: - "Summary includes dependency hotspots queried from SQLite" - "Summary shows file purposes, not just file counts" - "Transitive dependents queryable via recursive CTE" + - "SessionStart hook injects graph-backed summary into context" artifacts: - path: "hooks/gsd-intel-index.js" provides: "Graph-backed summary generation" @@ -24,6 +25,10 @@ must_haves: to: ".planning/intel/graph.db" via: "SQL queries for hotspots" pattern: "SELECT.*FROM edges.*GROUP BY" + - from: "hooks/gsd-intel-session.js" + to: ".planning/intel/summary.md" + via: "fs.readFileSync on startup/resume" + pattern: "readFileSync.*summary\\.md" --- @@ -52,6 +57,11 @@ Summary generation requirements (from research): - Group by type from node body - Include file purposes from entity content - Target < 500 tokens for context injection + +Existing wiring (from Phase 2): +- hooks/gsd-intel-session.js reads summary.md on startup/resume +- Injects content as tag into Claude's context +- This wiring already exists - we verify it works with new graph format @@ -269,7 +279,7 @@ Key design notes: - Task 3: Integrate graph summary into regeneration flow + Task 3: Integrate graph summary into regeneration flow and verify SessionStart wiring hooks/gsd-intel-index.js Modify regenerateEntitySummary() to prefer graph summary when available. @@ -333,7 +343,7 @@ if (isEntityFile(filePath)) { Note: regenerateEntitySummary becomes async because it calls generateGraphSummary. -Test the full flow: +Test the full flow including SessionStart injection: ```bash cd /Users/lexchristopherson/Developer/claude-code-resources/get-shit-done @@ -393,18 +403,21 @@ done # Check summary.md has graph-based content cat .planning/intel/summary.md - # Should show: # - "Dependency Hotspots" section # - test-db with 2 dependents (auth and api both depend on it) +# Verify SessionStart hook reads new summary format +echo '{"source":"startup"}' | node hooks/gsd-intel-session.js +# Should output ... with hotspots + # Cleanup rm .planning/intel/entities/test-*.md rm .planning/intel/graph.db rm .planning/intel/summary.md ``` - Summary generation prefers graph when available, falls back to file-based + Summary generation prefers graph when available, SessionStart hook confirmed to inject graph-backed summary @@ -428,6 +441,12 @@ After all tasks complete: - Simulate entity writes - Check summary.md has "Dependency Hotspots" section - Hotspot counts are accurate + +4. SessionStart wiring verified: + ```bash + # Verify SessionStart reads and injects the new format + echo '{"source":"startup"}' | node hooks/gsd-intel-session.js | grep -q "Dependency Hotspots" && echo "PASS: SessionStart injects graph summary" + ``` @@ -438,6 +457,7 @@ After all tasks complete: - [ ] regenerateEntitySummary() prefers graph when graph.db exists - [ ] Falls back gracefully to file-based summary - [ ] Summary includes dependency hotspots with accurate counts +- [ ] SessionStart hook (gsd-intel-session.js) correctly injects graph-backed summary into context diff --git a/.planning/phases/04-semantic-intelligence/04-03-PLAN.md b/.planning/phases/04-semantic-intelligence/04-03-PLAN.md index ed36ef417..4104e6ce6 100644 --- a/.planning/phases/04-semantic-intelligence/04-03-PLAN.md +++ b/.planning/phases/04-semantic-intelligence/04-03-PLAN.md @@ -16,24 +16,24 @@ must_haves: - "Batch processing handles 100+ files efficiently" artifacts: - path: "commands/gsd/analyze-codebase.md" - provides: "Semantic entity generation via Claude API" - contains: "@anthropic-ai/sdk" + provides: "Semantic entity generation instructions for Claude" + contains: "semantic entities" - path: "package.json" provides: "Anthropic SDK dependency" contains: "@anthropic-ai/sdk" key_links: - from: "commands/gsd/analyze-codebase.md" - to: "Anthropic Messages API" - via: "client.messages.create" - pattern: "messages\\.create" + to: "Task tool" + via: "Subagent spawning for entity batch processing" + pattern: "Task.*entity" --- -Enhance /gsd:analyze-codebase to create semantic entity files using Claude API. +Enhance /gsd:analyze-codebase to create semantic entity files using Claude. Purpose: Generate entity documentation that captures file PURPOSE (what it does, why it exists), not just syntax (exports/imports). This transforms "2-3 ls commands" of information into genuine semantic understanding. -Output: Updated analyze-codebase.md command with Claude API integration, @anthropic-ai/sdk dependency. +Output: Updated analyze-codebase.md command with entity generation instructions, @anthropic-ai/sdk dependency (for future direct API use). @@ -60,7 +60,14 @@ Current analyze-codebase.md: New requirement: - After indexing, optionally create entity .md files -- Use Claude to write semantic purpose, not just regex extraction +- Claude (executing the command) reads file content and generates semantic documentation +- No embedded JavaScript in command markdown - Claude IS the executor + +Execution model clarification: +- GSD command .md files contain INSTRUCTIONS for Claude to follow +- Claude reads the markdown and executes the instructions using its tools +- Commands cannot contain executable JavaScript - Claude interprets and acts on the instructions +- For batch processing, Claude uses the Task tool to spawn subagents @@ -78,7 +85,7 @@ Add @anthropic-ai/sdk to package.json dependencies: } ``` -Note: Version 0.52.0+ includes Messages API with proper TypeScript support. +Note: Version 0.52.0+ includes Messages API with proper TypeScript support. This dependency enables future direct API integration (e.g., hooks that call Claude API directly). For the /gsd:analyze-codebase command, Claude itself generates the entity content. `grep -q "@anthropic-ai/sdk" package.json` @anthropic-ai/sdk added to package.json @@ -99,20 +106,28 @@ Works standalone (without /gsd:new-project) for brownfield codebases. Creates: - index.json for file index - conventions.json for naming patterns - summary.md for context injection -- entities/*.md for semantic file documentation (optional, requires ANTHROPIC_API_KEY) +- entities/*.md for semantic file documentation (optional) Output: .planning/intel/index.json, conventions.json, summary.md, entities/*.md ``` -2. Add allowed-tools: Task (for entity generation subagent) +2. Add Task to allowed-tools (for entity generation via subagent): +```yaml +allowed-tools: + - Read + - Bash + - Glob + - Write + - Task +``` -3. Add new Step 9 after Step 8 (before completion report): +3. Add new Step 9 after Step 8 (before completion report). This step provides INSTRUCTIONS for Claude to follow: ```markdown ## Step 9: Generate semantic entities (optional) -If `ANTHROPIC_API_KEY` environment variable is set, generate semantic entity files. +After indexing, generate semantic entity files for key codebase files. ### 9a: Select key files for entity generation @@ -120,7 +135,7 @@ From the index, select files for entity generation using these criteria: - Files with 3+ exports (significant modules) - Files imported by 5+ other files (dependency hotspots) - Files in key directories: api/, lib/, utils/, services/, models/ -- Limit to 50 files maximum per run (avoid excessive API costs) +- Limit to 50 files maximum per run (context management) Skip: - Test files (*.test.*, *.spec.*) @@ -134,41 +149,30 @@ Skip: mkdir -p .planning/intel/entities ``` -### 9c: Generate entities using Claude API +### 9c: Generate entities in batches -For each selected file, use the Anthropic SDK to generate entity content: +For efficient processing, use the Task tool to spawn a subagent for batch entity generation. -```javascript -const Anthropic = require('@anthropic-ai/sdk'); -const client = new Anthropic(); // Uses ANTHROPIC_API_KEY env var +**Subagent prompt template:** -async function generateEntityContent(filePath, fileContent) { - const response = await client.messages.create({ - model: 'claude-sonnet-4-5-20250929', - max_tokens: 1500, - system: `You are a senior engineer documenting a codebase. Create entity documentation following this exact template format. Be concise - focus on PURPOSE and key relationships. +``` +Generate semantic entity documentation for the following files. -Output ONLY the markdown content, no explanations or commentary.`, - messages: [{ - role: 'user', - content: `Create entity documentation for this file. +For each file: +1. Read the file content +2. Analyze its purpose, exports, and dependencies +3. Write an entity file to .planning/intel/entities/{slug}.md -Path: ${filePath} -Content: -\`\`\` -${fileContent} -\`\`\` - -Follow this template EXACTLY: +Entity template (use EXACTLY this format): --- -path: ${filePath} -type: [module|component|util|config|test|api|hook|service|model] -updated: ${new Date().toISOString().split('T')[0]} +path: {file_path} +type: [module|component|util|config|api|hook|service|model] +updated: {today's date YYYY-MM-DD} status: active --- -# [filename] +# {filename} ## Purpose @@ -177,12 +181,12 @@ status: active ## Exports [List each export with signature and brief description] -- \`exportName(args): ReturnType\` - What it does +- `exportName(args): ReturnType` - What it does ## Dependencies [Internal deps use wiki-links, external use plain text] -- [[slugified-path]] - Why needed +- [[slugified-internal-path]] - Why needed - external-package - Why needed ## Used By @@ -191,52 +195,40 @@ TBD ## Notes -[Optional: patterns, gotchas, or important context]` - }] - }); +[Optional: patterns, gotchas, or important context] - return response.content[0].text; -} +--- + +Slug convention: `src/lib/db.ts` -> `src-lib-db` (replace / and . with -, remove extension) + +Files to process: +{list of file paths, max 10 per batch} ``` -### 9d: Write entity files +### 9d: Process in batches of 10 -For each generated entity: -1. Create slug from path: `src/lib/db.ts` -> `src-lib-db` -2. Write to `.planning/intel/entities/{slug}.md` -3. The PostToolUse hook will automatically sync to graph.db +For codebases with many key files: +1. Split the selected files into batches of 10 +2. Use Task tool for each batch with the prompt template above +3. Wait for each batch to complete before starting the next +4. This prevents context exhaustion and allows progress tracking -### 9e: Process in batches +### 9e: Verify entity creation -Process files in batches of 5 with 1 second delay between batches to respect rate limits. - -```javascript -async function processEntities(files) { - const batchSize = 5; - for (let i = 0; i < files.length; i += batchSize) { - const batch = files.slice(i, i + batchSize); - await Promise.all(batch.map(async (filePath) => { - const content = fs.readFileSync(filePath, 'utf8'); - const entityContent = await generateEntityContent(filePath, content); - const slug = filePath.replace(/^\//, '').replace(/[\/\.]/g, '-').replace(/-[jt]sx?$/, ''); - fs.writeFileSync(`.planning/intel/entities/${slug}.md`, entityContent); - })); - if (i + batchSize < files.length) { - await new Promise(r => setTimeout(r, 1000)); // Rate limit - } - } -} -``` +After batch processing: +- Count entities created: `ls .planning/intel/entities/*.md | wc -l` +- Verify they have semantic content (Purpose section, not just syntax) +- The PostToolUse hook will automatically sync new entities to graph.db ``` -4. Update Step 10 (completion report) to include entity stats: +4. Update Step 10 (renumber from Step 8) to include entity stats: ```markdown ## Step 10: Report completion Display summary statistics: -\`\`\` +``` Codebase Analysis Complete Files indexed: [N] @@ -248,17 +240,17 @@ Conventions detected: - Directories: [list] - Patterns: [list] -Entities created: [N] (if ANTHROPIC_API_KEY set) +Entities created: [N] (if entity generation ran) - Skipped: [N] (already existed or filtered) Files created: - .planning/intel/index.json - .planning/intel/conventions.json - .planning/intel/summary.md -- .planning/intel/entities/*.md (if API key set) +- .planning/intel/entities/*.md (if entities generated) Next: Intel hooks will continue incremental learning as you code. -\`\`\` +``` ``` 5. Update success criteria to include entity generation: @@ -270,7 +262,8 @@ Next: Intel hooks will continue incremental learning as you code. - [ ] index.json populated with exports and imports for each file - [ ] conventions.json has detected patterns (naming, directories, suffixes) - [ ] summary.md is concise (< 500 tokens) -- [ ] entities/*.md created for key files (if ANTHROPIC_API_KEY set) +- [ ] entities/*.md created for key files (if Step 9 executed) +- [ ] Entity files have semantic Purpose sections (not just syntax extraction) - [ ] Statistics reported to user ``` @@ -280,63 +273,55 @@ Next: Intel hooks will continue incremental learning as you code. # Check command has entity generation step grep -q "Generate semantic entities" commands/gsd/analyze-codebase.md && echo "PASS: Step 9 exists" -# Check mentions Anthropic SDK -grep -q "@anthropic-ai/sdk" commands/gsd/analyze-codebase.md && echo "PASS: SDK mentioned" +# Check mentions Task tool for batching +grep -q "Task tool" commands/gsd/analyze-codebase.md && echo "PASS: Task batching documented" -# Check has batch processing -grep -q "batchSize" commands/gsd/analyze-codebase.md && echo "PASS: Batch processing" +# Check entity template is present +grep -q "## Purpose" commands/gsd/analyze-codebase.md && echo "PASS: Entity template included" + +# Check batch size documented +grep -q "batches of 10" commands/gsd/analyze-codebase.md && echo "PASS: Batch processing" ``` - analyze-codebase.md includes semantic entity generation via Claude API + analyze-codebase.md includes semantic entity generation via Claude + Task tool batching - Task 3: Add fallback messaging for missing API key + Task 3: Add context section explaining execution model commands/gsd/analyze-codebase.md -Add clear messaging when ANTHROPIC_API_KEY is not set. +Update the context section to explain how entity generation works. In the context section, add: ```markdown -**Entity generation (optional):** -Requires `ANTHROPIC_API_KEY` environment variable. If not set, only index/conventions/summary are created. Set with: -```bash -export ANTHROPIC_API_KEY=sk-ant-... +**Entity generation:** +Step 9 generates semantic entity files using Claude's understanding of code purpose. Unlike regex-based extraction (exports/imports), this captures WHY code exists. + +For large codebases (50+ key files), entity generation uses the Task tool to spawn subagents that process files in batches of 10. This: +- Prevents context exhaustion +- Allows progress tracking +- Enables parallel processing + +Entity files are written to `.planning/intel/entities/` and automatically synced to the graph database by the PostToolUse hook. + +**When to skip entity generation:** +- Quick index-only scan: Stop after Step 8 +- Already have entities: Existing entities won't be overwritten +- Small codebase: May not need formal entities ``` -Entity generation costs approximately $0.01-0.02 per file (using claude-sonnet-4-5-20250929). -``` - -In Step 9, add at the beginning: - -```markdown -## Step 9: Generate semantic entities (optional) - -**Check for API key:** -```javascript -if (!process.env.ANTHROPIC_API_KEY) { - console.log('Skipping entity generation: ANTHROPIC_API_KEY not set'); - console.log('To enable, run: export ANTHROPIC_API_KEY=sk-ant-...'); - // Skip to Step 10 -} -``` - -If no API key, skip directly to Step 10 with message: -``` -Entity generation skipped (no ANTHROPIC_API_KEY). -To enable semantic entities, set ANTHROPIC_API_KEY and re-run. -``` -``` - -This ensures the command still works without API key (backwards compatible) while clearly explaining how to enable entity generation. +This clarifies: +1. Claude generates entity content (not embedded JavaScript) +2. Task tool handles batching for large codebases +3. Users can skip Step 9 if they just want the index ```bash -grep -q "ANTHROPIC_API_KEY not set" commands/gsd/analyze-codebase.md && echo "PASS: Fallback messaging exists" +grep -q "Task tool to spawn subagents" commands/gsd/analyze-codebase.md && echo "PASS: Execution model explained" ``` - Command gracefully handles missing API key with clear instructions + Command explains entity generation execution model clearly @@ -352,36 +337,37 @@ After all tasks complete: 2. Command has entity generation: ```bash grep -q "Step 9" commands/gsd/analyze-codebase.md && echo "PASS" - grep -q "generateEntityContent" commands/gsd/analyze-codebase.md && echo "PASS" + grep -q "semantic entities" commands/gsd/analyze-codebase.md && echo "PASS" ``` -3. Fallback works: +3. Task tool batching: ```bash - grep -q "ANTHROPIC_API_KEY not set" commands/gsd/analyze-codebase.md && echo "PASS" + grep -q "Task tool" commands/gsd/analyze-codebase.md && echo "PASS" + grep -q "batches of 10" commands/gsd/analyze-codebase.md && echo "PASS" ``` -4. Batch processing: +4. Entity template present: ```bash - grep -q "batchSize" commands/gsd/analyze-codebase.md && echo "PASS" + grep -q "## Purpose" commands/gsd/analyze-codebase.md && echo "PASS" ``` -Manual test (requires API key): +Manual test: ```bash -export ANTHROPIC_API_KEY=sk-ant-... # Run /gsd:analyze-codebase on a test project -# Verify .planning/intel/entities/*.md created with semantic content +# Verify Steps 1-8 produce index.json, conventions.json, summary.md +# If Step 9 runs, verify .planning/intel/entities/*.md created with semantic content ``` - [ ] @anthropic-ai/sdk added to package.json -- [ ] Step 9 added for entity generation +- [ ] Step 9 added for entity generation (instructions for Claude, not embedded JS) +- [ ] Task tool documented for batch processing subagents - [ ] File selection criteria documented (3+ exports, 5+ dependents, key dirs) -- [ ] 50 file limit per run to control costs -- [ ] Batch processing with rate limiting (5 files, 1s delay) +- [ ] 50 file limit per run to manage context +- [ ] Batch processing with batches of 10 via Task tool - [ ] Entity slug convention documented -- [ ] Graceful fallback when ANTHROPIC_API_KEY not set -- [ ] Cost estimate included (~$0.01-0.02 per file) +- [ ] Execution model explained (Claude generates content, not script execution) - [ ] Updated success criteria includes entity generation