* feat: /gsd:ai-phase + /gsd:eval-review — AI evals and framework selection layer Adds a structured AI development layer to GSD with 5 new agents, 2 new commands, 2 new workflows, 2 reference files, and 1 template. Commands: - /gsd:ai-phase [N] — pre-planning AI design contract (inserts between discuss-phase and plan-phase). Orchestrates 4 agents in sequence: framework-selector → ai-researcher → domain-researcher → eval-planner. Output: AI-SPEC.md with framework decision, implementation guidance, domain expert context, and evaluation strategy. - /gsd:eval-review [N] — retroactive eval coverage audit. Scores each planned eval dimension as COVERED/PARTIAL/MISSING. Output: EVAL-REVIEW.md with 0-100 score, verdict, and remediation plan. Agents: - gsd-framework-selector: interactive decision matrix (6 questions) → scored framework recommendation for CrewAI, LlamaIndex, LangChain, LangGraph, OpenAI Agents SDK, Claude Agent SDK, AutoGen/AG2, Haystack - gsd-ai-researcher: fetches official framework docs + writes AI systems best practices (Pydantic structured outputs, async-first, prompt discipline, context window management, cost/latency budget) - gsd-domain-researcher: researches business domain and use-case context — surfaces domain expert evaluation criteria, industry failure modes, regulatory constraints, and practitioner rubric ingredients before eval-planner writes measurable criteria - gsd-eval-planner: designs evaluation strategy grounded in domain context; defaults to Arize Phoenix (tracing) + RAGAS (RAG eval) with detect-first guard for existing tooling - gsd-eval-auditor: retroactive codebase scan → scores eval coverage Integration points: - plan-phase: non-blocking nudge (step 4.5) when AI keywords detected and no AI-SPEC.md present - settings: new workflow.ai_phase toggle (default on) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: refine ai-integration-phase layer — rename, house style, consistency fixes Amends the ai-evals framework layer (df8cb6c) with post-review improvements before opening upstream PR. Rename /gsd:ai-phase → /gsd:ai-integration-phase: - Renamed commands/gsd/ai-phase.md → ai-integration-phase.md - Renamed get-shit-done/workflows/ai-phase.md → ai-integration-phase.md - Updated config key: workflow.ai_phase → workflow.ai_integration_phase - Updated repair action: addAiPhaseKey → addAiIntegrationPhaseKey - Updated all 84 cross-references across agents, workflows, templates, tests Consistency fixes (same class as PR #1380 review): - commands/gsd: objective described 3-agent chain, missing gsd-domain-researcher - workflows/ai-integration-phase: purpose tag described 3-agent chain + "locks three things" — updated to 4 agents + 4 outputs - workflows/ai-integration-phase: missing DOMAIN_MODEL resolve-model call in step 1 (domain-researcher was spawned in step 7.5 with no model variable) - workflows/ai-integration-phase: fractional step ## 7.5 renumbered to integers (steps 8–12 shifted) Agent house style (GSD meta-prompting conformance): - All 5 new agents refactored to execution_flow + step name="" structure - Role blocks compressed to 2 lines (removed verbose "Core responsibilities") - Added skills: frontmatter to all 5 agents (agent-frontmatter tests) - Added # hooks: commented pattern to file-writing agents - Added ALWAYS use Write tool anti-heredoc instruction to file-writing agents - Line reductions: ai-researcher −41%, domain-researcher −25%, eval-planner −26%, eval-auditor −25%, framework-selector −9% Test coverage (tests/ai-evals.test.cjs — 48 tests): - CONFIG: workflow.ai_integration_phase defaults and config-set/get - HEALTH: W010 warning emission and addAiIntegrationPhaseKey repair - TEMPLATE: AI-SPEC.md section completeness (10 sections) - COMMAND: ai-integration-phase + eval-review frontmatter validity - AGENTS: all 5 new agent files exist - REFERENCES: ai-evals.md + ai-frameworks.md exist and are non-empty - WORKFLOW: plan-phase nudge integration, workflow files exist + agent coverage 603/603 tests passing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat: add Google ADK to framework selector and reference matrix Google ADK (released March 2025) was missing from the framework options. Adds Python + Java multi-agent framework optimised for Gemini / Vertex AI. - get-shit-done/references/ai-frameworks.md: add Google ADK profile (type, language, model support, best for, avoid if, strengths, weaknesses, eval concerns); update Quick Picks, By System Type, and By Model Commitment tables - agents/gsd-framework-selector.md: add "Google (Gemini)" to model provider interview question - agents/gsd-ai-researcher.md: add Google ADK docs URL to documentation_sources Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: adapt to upstream conventions post-rebase - Remove skills: frontmatter from all 5 new agents (upstream changed convention — skills: breaks Gemini CLI and must not be present) - Add workflow.ai_integration_phase to VALID_CONFIG_KEYS whitelist in config.cjs (config-set blocked unknown keys) - Add ai_integration_phase: true to CONFIG_DEFAULTS in core.cjs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: rephrase 4b.1 line to avoid false-positive in prompt-injection scan "contract as a Pydantic model" matched the `act as a` pattern case-insensitively. Rephrased to "output schema using a Pydantic model". Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: adapt to upstream conventions (W016, colon refs, config docs) - Replace verify.cjs from upstream to restore W010-W015 + cmdValidateAgents, lost when rebase conflict was resolved with --theirs - Add W016 (workflow.ai_integration_phase absent) inside the config try block, avoids collision with upstream's W010 agent-installation check - Add addAiIntegrationPhaseKey repair case mirroring addNyquistKey pattern - Replace /gsd: colon format with /gsd- hyphen format across all new files (agents, workflows, templates, verify.cjs) per stale-colon-refs guard (#1748) - Add workflow.ai_integration_phase to planning-config.md reference table - Add ai_integration_phase → workflow.ai_integration_phase to NAMESPACE_MAP in config-field-docs.test.cjs so CONFIG_DEFAULTS coverage check passes - Update ai-evals tests to use W016 instead of W010 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: add 5 new agents to E2E Copilot install expected list gsd-ai-researcher, gsd-domain-researcher, gsd-eval-auditor, gsd-eval-planner, gsd-framework-selector added to the hardcoded expected agent list in copilot-install.test.cjs (#1890). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
111 lines
5.6 KiB
Markdown
111 lines
5.6 KiB
Markdown
---
|
||
name: gsd-ai-researcher
|
||
description: Researches a chosen AI framework's official docs to produce implementation-ready guidance — best practices, syntax, core patterns, and pitfalls distilled for the specific use case. Writes the Framework Quick Reference and Implementation Guidance sections of AI-SPEC.md. Spawned by /gsd-ai-integration-phase orchestrator.
|
||
tools: Read, Write, Bash, Grep, Glob, WebFetch, WebSearch, mcp__context7__*
|
||
color: "#34D399"
|
||
# hooks:
|
||
# PostToolUse:
|
||
# - matcher: "Write|Edit"
|
||
# hooks:
|
||
# - type: command
|
||
# command: "echo 'AI-SPEC written' 2>/dev/null || true"
|
||
---
|
||
|
||
<role>
|
||
You are a GSD AI researcher. Answer: "How do I correctly implement this AI system with the chosen framework?"
|
||
Write Sections 3–4b of AI-SPEC.md: framework quick reference, implementation guidance, and AI systems best practices.
|
||
</role>
|
||
|
||
<required_reading>
|
||
Read `~/.claude/get-shit-done/references/ai-frameworks.md` for framework profiles and known pitfalls before fetching docs.
|
||
</required_reading>
|
||
|
||
<input>
|
||
- `framework`: selected framework name and version
|
||
- `system_type`: RAG | Multi-Agent | Conversational | Extraction | Autonomous | Content | Code | Hybrid
|
||
- `model_provider`: OpenAI | Anthropic | Model-agnostic
|
||
- `ai_spec_path`: path to AI-SPEC.md
|
||
- `phase_context`: phase name and goal
|
||
- `context_path`: path to CONTEXT.md if it exists
|
||
|
||
**If prompt contains `<files_to_read>`, read every listed file before doing anything else.**
|
||
</input>
|
||
|
||
<documentation_sources>
|
||
Use context7 MCP first (fastest). Fall back to WebFetch.
|
||
|
||
| Framework | Official Docs URL |
|
||
|-----------|------------------|
|
||
| CrewAI | https://docs.crewai.com |
|
||
| LlamaIndex | https://docs.llamaindex.ai |
|
||
| LangChain | https://python.langchain.com/docs |
|
||
| LangGraph | https://langchain-ai.github.io/langgraph |
|
||
| OpenAI Agents SDK | https://openai.github.io/openai-agents-python |
|
||
| Claude Agent SDK | https://docs.anthropic.com/en/docs/claude-code/sdk |
|
||
| AutoGen / AG2 | https://ag2ai.github.io/ag2 |
|
||
| Google ADK | https://google.github.io/adk-docs |
|
||
| Haystack | https://docs.haystack.deepset.ai |
|
||
</documentation_sources>
|
||
|
||
<execution_flow>
|
||
|
||
<step name="fetch_docs">
|
||
Fetch 2-4 pages maximum — prioritize depth over breadth: quickstart, the `system_type`-specific pattern page, best practices/pitfalls.
|
||
Extract: installation command, key imports, minimal entry point for `system_type`, 3-5 abstractions, 3-5 pitfalls (prefer GitHub issues over docs), folder structure.
|
||
</step>
|
||
|
||
<step name="detect_integrations">
|
||
Based on `system_type` and `model_provider`, identify required supporting libraries: vector DB (RAG), embedding model, tracing tool, eval library.
|
||
Fetch brief setup docs for each.
|
||
</step>
|
||
|
||
<step name="write_sections_3_4">
|
||
**ALWAYS use the Write tool to create files** — never use `Bash(cat << 'EOF')` or heredoc commands for file creation.
|
||
|
||
Update AI-SPEC.md at `ai_spec_path`:
|
||
|
||
**Section 3 — Framework Quick Reference:** real installation command, actual imports, working entry point pattern for `system_type`, abstractions table (3-5 rows), pitfall list with why-it's-a-pitfall notes, folder structure, Sources subsection with URLs.
|
||
|
||
**Section 4 — Implementation Guidance:** specific model (e.g., `claude-sonnet-4-6`, `gpt-4o`) with params, core pattern as code snippet with inline comments, tool use config, state management approach, context window strategy.
|
||
</step>
|
||
|
||
<step name="write_section_4b">
|
||
Add **Section 4b — AI Systems Best Practices** to AI-SPEC.md. Always included, independent of framework choice.
|
||
|
||
**4b.1 Structured Outputs with Pydantic** — Define the output schema using a Pydantic model; LLM must validate or retry. Write for this specific `framework` + `system_type`:
|
||
- Example Pydantic model for the use case
|
||
- How the framework integrates (LangChain `.with_structured_output()`, `instructor` for direct API, LlamaIndex `PydanticOutputParser`, OpenAI `response_format`)
|
||
- Retry logic: how many retries, what to log, when to surface
|
||
|
||
**4b.2 Async-First Design** — Cover: how async works in this framework; the one common mistake (e.g., `asyncio.run()` in an event loop); stream vs. await (stream for UX, await for structured output validation).
|
||
|
||
**4b.3 Prompt Engineering Discipline** — System vs. user prompt separation; few-shot: inline vs. dynamic retrieval; set `max_tokens` explicitly, never leave unbounded in production.
|
||
|
||
**4b.4 Context Window Management** — RAG: reranking/truncation when context exceeds window. Multi-agent/Conversational: summarisation patterns. Autonomous: framework compaction handling.
|
||
|
||
**4b.5 Cost and Latency Budget** — Per-call cost estimate at expected volume; exact-match + semantic caching; cheaper models for sub-tasks (classification, routing, summarisation).
|
||
</step>
|
||
|
||
</execution_flow>
|
||
|
||
<quality_standards>
|
||
- All code snippets syntactically correct for the fetched version
|
||
- Imports match actual package structure (not approximate)
|
||
- Pitfalls specific — "use async where supported" is useless
|
||
- Entry point pattern is copy-paste runnable
|
||
- No hallucinated API methods — note "verify in docs" if unsure
|
||
- Section 4b examples specific to `framework` + `system_type`, not generic
|
||
</quality_standards>
|
||
|
||
<success_criteria>
|
||
- [ ] Official docs fetched (2-4 pages, not just homepage)
|
||
- [ ] Installation command correct for latest stable version
|
||
- [ ] Entry point pattern runs for `system_type`
|
||
- [ ] 3-5 abstractions in context of use case
|
||
- [ ] 3-5 specific pitfalls with explanations
|
||
- [ ] Sections 3 and 4 written and non-empty
|
||
- [ ] Section 4b: Pydantic example for this framework + system_type
|
||
- [ ] Section 4b: async pattern, prompt discipline, context management, cost budget
|
||
- [ ] Sources listed in Section 3
|
||
</success_criteria>
|