feat(intel): add self-evolving codebase intelligence

- Add entity.md template for documenting code understanding - Add update_intel_entity step to executor workflow - Modify gsd-intel-index.js to parse [[wiki-links]] from entities - Hook regenerates summary.md with hotspots, modules, recent updates Executor now creates/updates entity files after each task completion. Summary automatically reflects accumulated semantic understanding. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 02:05:10 -06:00
parent 4218f8666a
commit 3a82930579
3 changed files with 537 additions and 4 deletions
--- a/get-shit-done/templates/entity.md
+++ b/get-shit-done/templates/entity.md
@@ -0,0 +1,173 @@
+# Entity Template
+
+Template for `.planning/codebase/{entity-slug}.md` - file-level intelligence documentation.
+
+---
+
+## File Template
+
+```markdown
+---
+path: {path}
+type: {type}
+updated: {updated}
+status: {status}
+---
+
+# {filename}
+
+## Purpose
+
+{purpose}
+
+## Exports
+
+{exports}
+
+## Dependencies
+
+{dependencies}
+
+## Used By
+
+{used_by}
+
+## Notes
+
+{notes}
+```
+
+---
+
+## Field Reference
+
+### Frontmatter
+
+| Field | Values | Description |
+|-------|--------|-------------|
+| `path` | Absolute path | Full path to the file |
+| `type` | module, component, util, config, test, api, hook | Primary classification |
+| `updated` | YYYY-MM-DD | Last time this entity was updated |
+| `status` | active, deprecated, stub | Current state |
+
+### Sections
+
+**Purpose** (required)
+1-3 sentences covering:
+- What this file does
+- Why it exists
+- Who/what uses it (high-level)
+
+**Exports** (required for modules with exports)
+List each export with signature and description:
+```markdown
+- `functionName(arg: Type): ReturnType` - What it does
+- `ClassName` - What it represents
+- `CONSTANT_NAME` - What value it holds
+```
+
+For files without exports (config, tests), write "None" or describe what the file defines.
+
+**Dependencies** (required)
+Internal dependencies use wiki-links (slugified paths):
+```markdown
+- [[src-lib-db]] - Database client
+- [[src-types-user]] - User type definitions
+```
+
+External dependencies use plain text:
+```markdown
+- react - Component framework
+- jose - JWT handling
+```
+
+**Used By** (grows over time)
+Files that import this one, using wiki-links:
+```markdown
+- [[src-app-api-auth-route]]
+- [[src-components-dashboard]]
+```
+
+Initially may be empty or incomplete. Updated as Claude encounters imports.
+
+**Notes** (optional)
+Patterns, gotchas, or context:
+```markdown
+- Uses singleton pattern for connection pooling
+- WARNING: Must call `init()` before any other method
+- Related: See [[src-lib-cache]] for caching layer
+```
+
+---
+
+## Slug Convention
+
+Entity slugs are derived from file paths:
+- `src/lib/db.ts` becomes `src-lib-db`
+- `src/app/api/auth/route.ts` becomes `src-app-api-auth-route`
+
+Rule: Replace `/` and `.` with `-`, drop file extension.
+
+---
+
+## Example
+
+```markdown
+---
+path: /project/src/lib/auth.ts
+type: util
+updated: 2025-01-15
+status: active
+---
+
+# auth.ts
+
+## Purpose
+
+JWT token management using jose library. Handles token creation, verification, and refresh rotation. Used by all protected API routes via middleware.
+
+## Exports
+
+- `createAccessToken(userId: string): Promise<string>` - Creates 15-min access token
+- `createRefreshToken(userId: string): Promise<string>` - Creates 7-day refresh token
+- `verifyToken(token: string): Promise<TokenPayload>` - Validates and decodes token
+- `rotateRefresh(oldToken: string): Promise<TokenPair>` - Issues new token pair
+
+## Dependencies
+
+- [[src-lib-db]] - Stores refresh tokens for revocation
+- [[src-types-auth]] - TokenPayload, TokenPair types
+- jose - JWT signing and verification
+- bcrypt - Password hashing
+
+## Used By
+
+- [[src-middleware]]
+- [[src-app-api-auth-login-route]]
+- [[src-app-api-auth-logout-route]]
+- [[src-app-api-auth-refresh-route]]
+
+## Notes
+
+- Access tokens are stateless; refresh tokens stored in DB for revocation
+- Uses RS256 algorithm with keys from environment
+- WARNING: Never log token values, even in debug mode
+```
+
+---
+
+## Guidelines
+
+**When to create/update:**
+- After modifying a file during plan execution
+- When encountering a file that lacks documentation
+- When relationships change (new imports, exports)
+
+**Minimal viable entity:**
+At minimum, an entity needs frontmatter + Purpose. Other sections can be "TBD" if unknown.
+
+**Accuracy over completeness:**
+Better to have partial accurate info than complete guesses. Mark unknowns explicitly.
+
+**Link discovery:**
+The hook that processes entities extracts all `[[wiki-links]]` to build the relationship graph. Ensure links use correct slug format.
--- a/get-shit-done/workflows/execute-plan.md
+++ b/get-shit-done/workflows/execute-plan.md
@@ -598,7 +598,8 @@ Execute each task in the prompt. **Deviations are normal** - handle them automat
   - Continue implementing, applying rules as needed
   - Run the verification
   - Confirm done criteria met
-   - **Commit the task** (see `<task_commit>` below)
+   - **Update intel entities** for modified files (see `<update_intel_entity>` below)
+   - **Commit the task** (see `<task_commit>` below) - includes entity files in commit
   - Track task completion and commit hash for Summary documentation
   - Continue to next task

@@ -927,6 +928,148 @@ None - plan executed exactly as written.

 </deviation_documentation>

+<update_intel_entity>
+
+## Update Codebase Intelligence Entity
+
+**Trigger:** After each task's `<done>` criteria met, BEFORE committing.
+
+This step documents your understanding of modified files. The knowledge base self-evolves as you work.
+
+**1. Get files from the just-completed task:**
+
+Check the task's `<files>` list. If no `<files>` attribute, use `git status --short` to identify modified files.
+
+**2. For each file, determine if significant:**
+
+**Skip these patterns (not worth indexing):**
+```bash
+# Build/generated
+node_modules/*, .next/*, dist/*, build/*, .git/*
+
+# Tests (their targets are more valuable)
+*.test.*, *.spec.*, __tests__/*, __mocks__/*
+
+# Config files (change rarely, low context value)
+package.json, package-lock.json, tsconfig.json, *.config.*, *.config.js, *.config.ts
+
+# Environment and locks
+.env*, *.lock, *.log, yarn.lock, pnpm-lock.yaml
+```
+
+If file matches skip pattern: continue to next file.
+
+**3. Derive entity path:**
+
+```bash
+# Convert file path to entity filename
+# src/lib/auth.ts → src-lib-auth.md
+# app/api/users/route.ts → app-api-users-route.md
+
+FILE_PATH="$1"
+ENTITY_NAME=$(echo "$FILE_PATH" | sed 's|^[./]*||' | tr '/' '-' | sed 's/\.[^.]*$//')
+ENTITY_PATH=".planning/intel/entities/${ENTITY_NAME}.md"
+```
+
+**4. Create intel directory if needed:**
+
+```bash
+mkdir -p .planning/intel/entities
+```
+
+**5. Check if entity exists:**
+
+```bash
+if [ -f "$ENTITY_PATH" ]; then
+  ACTION="update"
+else
+  ACTION="create"
+fi
+```
+
+**6. Create or update entity:**
+
+Use template from `~/.claude/get-shit-done/templates/entity.md`.
+
+Fill fields based on what you just built:
+
+| Field | Source |
+|-------|--------|
+| `path` | Absolute path to file |
+| `type` | Infer: module, component, util, config, api, hook |
+| `updated` | Today's date (YYYY-MM-DD) |
+| `status` | active (unless you're deprecating) |
+| **Purpose** | What you understand this file does |
+| **Exports** | Extract from the code you just wrote |
+| **Dependencies** | `[[slugified-path]]` for internal, plain for external |
+| **Used By** | Add callers if you know them from this session |
+| **Notes** | Gotchas, patterns, warnings discovered |
+
+**If updating existing entity:**
+- Read current content first
+- Preserve Used By entries you didn't touch
+- Update sections that changed
+- Don't remove information unless it's wrong
+
+**7. Write entity file:**
+
+```bash
+# Write the entity content
+cat > "$ENTITY_PATH" << 'EOF'
+---
+path: {path}
+type: {type}
+updated: {today}
+status: active
+---
+
+# {filename}
+
+## Purpose
+
+{purpose}
+
+## Exports
+
+{exports}
+
+## Dependencies
+
+{dependencies}
+
+## Used By
+
+{used_by}
+
+## Notes
+
+{notes}
+EOF
+```
+
+**8. Verify entity was created/updated:**
+
+```bash
+head -10 "$ENTITY_PATH"
+```
+
+**9. Stage entity file:**
+
+Entity files are staged along with task code files in the task commit.
+
+```bash
+git add "$ENTITY_PATH"
+```
+
+**Error handling:**
+- If entity creation fails: Log warning, continue to task_commit
+- Entity update is NOT blocking - a failed entity shouldn't stop code from being committed
+- Log: `"Warning: Could not update entity for {file}: {reason}"`
+
+**Target size:** Keep entities concise (30-50 lines). Purpose and Exports are most valuable.
+
+</update_intel_entity>
+
 <tdd_plan_execution>
 ## TDD Plan Execution

--- a/hooks/gsd-intel-index.js
+++ b/hooks/gsd-intel-index.js
@@ -1,6 +1,7 @@
 #!/usr/bin/env node
 // Codebase Intelligence - PostToolUse Indexing Hook
 // Indexes file exports/imports when Claude writes or edits JS/TS files
+// Also handles entity files for semantic codebase understanding

 const fs = require('fs');
 const path = require('path');
@@ -8,6 +9,9 @@ const path = require('path');
 // JS/TS file extensions to index
 const INDEXABLE_EXTENSIONS = ['.js', '.ts', '.jsx', '.tsx', '.mjs', '.cjs'];

+// Entity file location
+const ENTITY_DIR = '.planning/intel/entities';
+
 // Convention detection thresholds
 const MIN_SAMPLES = 5;
 const MIN_MATCH_RATE = 0.70;
@@ -378,6 +382,214 @@ function generateSummary(index, conventions) {
  return lines.join('\n');
 }

+/**
+ * Check if a file path is an entity file
+ */
+function isEntityFile(filePath) {
+  return filePath.includes('.planning/intel/entities/') &&
+         filePath.endsWith('.md');
+}
+
+/**
+ * Check if a file is a code file we should index
+ */
+function isCodeFile(filePath) {
+  const ext = path.extname(filePath).toLowerCase();
+  return INDEXABLE_EXTENSIONS.includes(ext);
+}
+
+/**
+ * Extract [[wiki-links]] from entity content
+ * Returns array of linked entity names (e.g., 'src-lib-db', 'src-api-auth')
+ */
+function extractWikiLinks(content) {
+  const links = [];
+  const regex = /\[\[([^\]]+)\]\]/g;
+  let match;
+  while ((match = regex.exec(content)) !== null) {
+    links.push(match[1]);
+  }
+  return links;
+}
+
+/**
+ * Parse frontmatter from entity content
+ * Returns object with frontmatter fields
+ */
+function parseEntityFrontmatter(content) {
+  const frontmatter = {};
+  const fmMatch = content.match(/^---\n([\s\S]*?)\n---/);
+  if (fmMatch) {
+    const fmLines = fmMatch[1].split('\n');
+    for (const line of fmLines) {
+      const colonIdx = line.indexOf(':');
+      if (colonIdx > 0) {
+        const key = line.slice(0, colonIdx).trim();
+        const value = line.slice(colonIdx + 1).trim();
+        frontmatter[key] = value;
+      }
+    }
+  }
+  return frontmatter;
+}
+
+/**
+ * Extract purpose from entity content (first paragraph after ## Purpose)
+ */
+function extractPurpose(content) {
+  const purposeMatch = content.match(/## Purpose\s*\n+([^\n#]+)/);
+  if (purposeMatch) {
+    return purposeMatch[1].trim();
+  }
+  return null;
+}
+
+/**
+ * Regenerate summary.md from all entity files
+ * Creates a semantic overview of the codebase
+ */
+function regenerateEntitySummary() {
+  const intelDir = path.join(process.cwd(), '.planning', 'intel');
+  const entitiesDir = path.join(intelDir, 'entities');
+  const summaryPath = path.join(intelDir, 'summary.md');
+
+  // Check directories exist
+  if (!fs.existsSync(entitiesDir)) {
+    return;
+  }
+
+  // Read all entity files
+  const entityFiles = fs.readdirSync(entitiesDir)
+    .filter(f => f.endsWith('.md'))
+    .map(f => path.join(entitiesDir, f));
+
+  if (entityFiles.length === 0) {
+    return;
+  }
+
+  // Parse all entities
+  const entities = [];
+  const dependentCounts = {}; // Track how many things depend on each entity
+
+  for (const filePath of entityFiles) {
+    try {
+      const content = fs.readFileSync(filePath, 'utf8');
+      const frontmatter = parseEntityFrontmatter(content);
+      const purpose = extractPurpose(content);
+      const links = extractWikiLinks(content);
+
+      // Count dependencies (things this entity links to)
+      for (const link of links) {
+        dependentCounts[link] = (dependentCounts[link] || 0) + 1;
+      }
+
+      entities.push({
+        name: path.basename(filePath, '.md'),
+        path: frontmatter.path || null,
+        type: frontmatter.type || 'unknown',
+        updated: frontmatter.updated || null,
+        status: frontmatter.status || 'unknown',
+        purpose,
+        links
+      });
+    } catch (e) {
+      // Skip unreadable files
+    }
+  }
+
+  // Generate summary
+  const lines = [];
+  lines.push('# Codebase Intelligence');
+  lines.push('');
+  lines.push(`**Files indexed:** ${entities.length}`);
+  lines.push(`**Last updated:** ${new Date().toISOString().split('T')[0]}`);
+  lines.push('');
+
+  // Group by type
+  const byType = {};
+  for (const entity of entities) {
+    const type = entity.type || 'other';
+    if (!byType[type]) {
+      byType[type] = [];
+    }
+    byType[type].push(entity);
+  }
+
+  // Key Modules section
+  lines.push('## Key Modules');
+  lines.push('');
+
+  const typeLabels = {
+    'api': 'API Layer',
+    'component': 'Components',
+    'util': 'Utilities',
+    'hook': 'Hooks',
+    'service': 'Services',
+    'model': 'Models',
+    'config': 'Configuration',
+    'other': 'Other'
+  };
+
+  for (const [type, typeEntities] of Object.entries(byType)) {
+    if (typeEntities.length === 0) continue;
+
+    const label = typeLabels[type] || type.charAt(0).toUpperCase() + type.slice(1);
+    lines.push(`### ${label}`);
+
+    // Show up to 5 entities per type, prioritize those with purposes
+    const withPurpose = typeEntities.filter(e => e.purpose);
+    const shown = withPurpose.slice(0, 5);
+    if (shown.length < 5) {
+      const remaining = typeEntities.filter(e => !e.purpose).slice(0, 5 - shown.length);
+      shown.push(...remaining);
+    }
+
+    for (const entity of shown) {
+      const desc = entity.purpose || 'No description';
+      lines.push(`- **${entity.path || entity.name}** - ${desc}`);
+    }
+    lines.push('');
+  }
+
+  // Dependency Hotspots (most depended-on files)
+  const hotspots = Object.entries(dependentCounts)
+    .sort((a, b) => b[1] - a[1])
+    .slice(0, 5);
+
+  if (hotspots.length > 0) {
+    lines.push('## Dependency Hotspots');
+    lines.push('');
+    lines.push('Files with most dependents (change carefully):');
+
+    for (const [entityName, count] of hotspots) {
+      const entity = entities.find(e => e.name === entityName);
+      const desc = entity?.purpose || '';
+      const pathStr = entity?.path || entityName;
+      lines.push(`1. \`${pathStr}\` (${count} dependents)${desc ? ' - ' + desc : ''}`);
+    }
+    lines.push('');
+  }
+
+  // Recent Updates (last 5 by date)
+  const withDates = entities
+    .filter(e => e.updated)
+    .sort((a, b) => b.updated.localeCompare(a.updated))
+    .slice(0, 5);
+
+  if (withDates.length > 0) {
+    lines.push('## Recent Updates');
+    lines.push('');
+    for (const entity of withDates) {
+      const desc = entity.purpose ? ` - ${entity.purpose}` : '';
+      lines.push(`- ${entity.name}.md (${entity.updated})${desc}`);
+    }
+    lines.push('');
+  }
+
+  // Write summary
+  fs.writeFileSync(summaryPath, lines.join('\n'));
+}
+
 /**
 * Update the index.json file with new file entry
 * Uses read-modify-write pattern with synchronous operations
@@ -447,9 +659,14 @@ process.stdin.on('end', () => {
      process.exit(0);
    }

-    // Only index JS/TS files
-    const ext = path.extname(filePath).toLowerCase();
-    if (!INDEXABLE_EXTENSIONS.includes(ext)) {
+    // Handle entity file writes - regenerate summary
+    if (isEntityFile(filePath)) {
+      regenerateEntitySummary();
+      process.exit(0);
+    }
+
+    // Handle code file writes - existing behavior
+    if (!isCodeFile(filePath)) {
      process.exit(0);
    }