33575ba91d
* feat: /gsd:ai-phase + /gsd:eval-review — AI evals and framework selection layer Adds a structured AI development layer to GSD with 5 new agents, 2 new commands, 2 new workflows, 2 reference files, and 1 template. Commands: - /gsd:ai-phase [N] — pre-planning AI design contract (inserts between discuss-phase and plan-phase). Orchestrates 4 agents in sequence: framework-selector → ai-researcher → domain-researcher → eval-planner. Output: AI-SPEC.md with framework decision, implementation guidance, domain expert context, and evaluation strategy. - /gsd:eval-review [N] — retroactive eval coverage audit. Scores each planned eval dimension as COVERED/PARTIAL/MISSING. Output: EVAL-REVIEW.md with 0-100 score, verdict, and remediation plan. Agents: - gsd-framework-selector: interactive decision matrix (6 questions) → scored framework recommendation for CrewAI, LlamaIndex, LangChain, LangGraph, OpenAI Agents SDK, Claude Agent SDK, AutoGen/AG2, Haystack - gsd-ai-researcher: fetches official framework docs + writes AI systems best practices (Pydantic structured outputs, async-first, prompt discipline, context window management, cost/latency budget) - gsd-domain-researcher: researches business domain and use-case context — surfaces domain expert evaluation criteria, industry failure modes, regulatory constraints, and practitioner rubric ingredients before eval-planner writes measurable criteria - gsd-eval-planner: designs evaluation strategy grounded in domain context; defaults to Arize Phoenix (tracing) + RAGAS (RAG eval) with detect-first guard for existing tooling - gsd-eval-auditor: retroactive codebase scan → scores eval coverage Integration points: - plan-phase: non-blocking nudge (step 4.5) when AI keywords detected and no AI-SPEC.md present - settings: new workflow.ai_phase toggle (default on) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: refine ai-integration-phase layer — rename, house style, consistency fixes Amends the ai-evals framework layer (df8cb6c) with post-review improvements before opening upstream PR. Rename /gsd:ai-phase → /gsd:ai-integration-phase: - Renamed commands/gsd/ai-phase.md → ai-integration-phase.md - Renamed get-shit-done/workflows/ai-phase.md → ai-integration-phase.md - Updated config key: workflow.ai_phase → workflow.ai_integration_phase - Updated repair action: addAiPhaseKey → addAiIntegrationPhaseKey - Updated all 84 cross-references across agents, workflows, templates, tests Consistency fixes (same class as PR #1380 review): - commands/gsd: objective described 3-agent chain, missing gsd-domain-researcher - workflows/ai-integration-phase: purpose tag described 3-agent chain + "locks three things" — updated to 4 agents + 4 outputs - workflows/ai-integration-phase: missing DOMAIN_MODEL resolve-model call in step 1 (domain-researcher was spawned in step 7.5 with no model variable) - workflows/ai-integration-phase: fractional step ## 7.5 renumbered to integers (steps 8–12 shifted) Agent house style (GSD meta-prompting conformance): - All 5 new agents refactored to execution_flow + step name="" structure - Role blocks compressed to 2 lines (removed verbose "Core responsibilities") - Added skills: frontmatter to all 5 agents (agent-frontmatter tests) - Added # hooks: commented pattern to file-writing agents - Added ALWAYS use Write tool anti-heredoc instruction to file-writing agents - Line reductions: ai-researcher −41%, domain-researcher −25%, eval-planner −26%, eval-auditor −25%, framework-selector −9% Test coverage (tests/ai-evals.test.cjs — 48 tests): - CONFIG: workflow.ai_integration_phase defaults and config-set/get - HEALTH: W010 warning emission and addAiIntegrationPhaseKey repair - TEMPLATE: AI-SPEC.md section completeness (10 sections) - COMMAND: ai-integration-phase + eval-review frontmatter validity - AGENTS: all 5 new agent files exist - REFERENCES: ai-evals.md + ai-frameworks.md exist and are non-empty - WORKFLOW: plan-phase nudge integration, workflow files exist + agent coverage 603/603 tests passing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat: add Google ADK to framework selector and reference matrix Google ADK (released March 2025) was missing from the framework options. Adds Python + Java multi-agent framework optimised for Gemini / Vertex AI. - get-shit-done/references/ai-frameworks.md: add Google ADK profile (type, language, model support, best for, avoid if, strengths, weaknesses, eval concerns); update Quick Picks, By System Type, and By Model Commitment tables - agents/gsd-framework-selector.md: add "Google (Gemini)" to model provider interview question - agents/gsd-ai-researcher.md: add Google ADK docs URL to documentation_sources Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: adapt to upstream conventions post-rebase - Remove skills: frontmatter from all 5 new agents (upstream changed convention — skills: breaks Gemini CLI and must not be present) - Add workflow.ai_integration_phase to VALID_CONFIG_KEYS whitelist in config.cjs (config-set blocked unknown keys) - Add ai_integration_phase: true to CONFIG_DEFAULTS in core.cjs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: rephrase 4b.1 line to avoid false-positive in prompt-injection scan "contract as a Pydantic model" matched the `act as a` pattern case-insensitively. Rephrased to "output schema using a Pydantic model". Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: adapt to upstream conventions (W016, colon refs, config docs) - Replace verify.cjs from upstream to restore W010-W015 + cmdValidateAgents, lost when rebase conflict was resolved with --theirs - Add W016 (workflow.ai_integration_phase absent) inside the config try block, avoids collision with upstream's W010 agent-installation check - Add addAiIntegrationPhaseKey repair case mirroring addNyquistKey pattern - Replace /gsd: colon format with /gsd- hyphen format across all new files (agents, workflows, templates, verify.cjs) per stale-colon-refs guard (#1748) - Add workflow.ai_integration_phase to planning-config.md reference table - Add ai_integration_phase → workflow.ai_integration_phase to NAMESPACE_MAP in config-field-docs.test.cjs so CONFIG_DEFAULTS coverage check passes - Update ai-evals tests to use W016 instead of W010 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: add 5 new agents to E2E Copilot install expected list gsd-ai-researcher, gsd-domain-researcher, gsd-eval-auditor, gsd-eval-planner, gsd-framework-selector added to the hardcoded expected agent list in copilot-install.test.cjs (#1890). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
187 lines
11 KiB
Markdown
187 lines
11 KiB
Markdown
# AI Framework Decision Matrix
|
||
|
||
> Reference used by `gsd-framework-selector` and `gsd-ai-researcher`.
|
||
> Distilled from official docs, benchmarks, and developer reports (2026).
|
||
|
||
---
|
||
|
||
## Quick Picks
|
||
|
||
| Situation | Pick |
|
||
|-----------|------|
|
||
| Simplest path to a working agent (OpenAI) | OpenAI Agents SDK |
|
||
| Simplest path to a working agent (model-agnostic) | CrewAI |
|
||
| Production RAG / document Q&A | LlamaIndex |
|
||
| Complex stateful workflows with branching | LangGraph |
|
||
| Multi-agent teams with defined roles | CrewAI |
|
||
| Code-aware autonomous agents (Anthropic) | Claude Agent SDK |
|
||
| "I don't know my requirements yet" | LangChain |
|
||
| Regulated / audit-trail required | LangGraph |
|
||
| Enterprise Microsoft/.NET shops | AutoGen/AG2 |
|
||
| Google Cloud / Gemini-committed teams | Google ADK |
|
||
| Pure NLP pipelines with explicit control | Haystack |
|
||
|
||
---
|
||
|
||
## Framework Profiles
|
||
|
||
### CrewAI
|
||
- **Type:** Multi-agent orchestration
|
||
- **Language:** Python only
|
||
- **Model support:** Model-agnostic
|
||
- **Learning curve:** Beginner (role/task/crew maps to real teams)
|
||
- **Best for:** Content pipelines, research automation, business process workflows, rapid prototyping
|
||
- **Avoid if:** Fine-grained state management, TypeScript, fault-tolerant checkpointing, complex conditional branching
|
||
- **Strengths:** Fastest multi-agent prototyping, 5.76x faster than LangGraph on QA tasks, built-in memory (short/long/entity/contextual), Flows architecture, standalone (no LangChain dep)
|
||
- **Weaknesses:** Limited checkpointing, coarse error handling, Python only
|
||
- **Eval concerns:** Task decomposition accuracy, inter-agent handoff, goal completion rate, loop detection
|
||
|
||
### LlamaIndex
|
||
- **Type:** RAG and data ingestion
|
||
- **Language:** Python + TypeScript
|
||
- **Model support:** Model-agnostic
|
||
- **Learning curve:** Intermediate
|
||
- **Best for:** Legal research, internal knowledge assistants, enterprise document search, any system where retrieval quality is the #1 priority
|
||
- **Avoid if:** Primary need is agent orchestration, multi-agent collaboration, or chatbot conversation flow
|
||
- **Strengths:** Best-in-class document parsing (LlamaParse), 35% retrieval accuracy improvement, 20-30% faster queries, mixed retrieval strategies (vector + graph + reranker)
|
||
- **Weaknesses:** Data framework first — agent orchestration is secondary
|
||
- **Eval concerns:** Context faithfulness, hallucination, answer relevance, retrieval precision/recall
|
||
|
||
### LangChain
|
||
- **Type:** General-purpose LLM framework
|
||
- **Language:** Python + TypeScript
|
||
- **Model support:** Model-agnostic (widest ecosystem)
|
||
- **Learning curve:** Intermediate–Advanced
|
||
- **Best for:** Evolving requirements, many third-party integrations, teams wanting one framework for everything, RAG + agents + chains
|
||
- **Avoid if:** Simple well-defined use case, RAG-primary (use LlamaIndex), complex stateful workflows (use LangGraph), performance at scale is critical
|
||
- **Strengths:** Largest community and integration ecosystem, 25% faster development vs scratch, covers RAG/agents/chains/memory
|
||
- **Weaknesses:** Abstraction overhead, p99 latency degrades under load, complexity creep risk
|
||
- **Eval concerns:** End-to-end task completion, chain correctness, retrieval quality
|
||
|
||
### LangGraph
|
||
- **Type:** Stateful agent workflows (graph-based)
|
||
- **Language:** Python + TypeScript (full parity)
|
||
- **Model support:** Model-agnostic (inherits LangChain integrations)
|
||
- **Learning curve:** Intermediate–Advanced (graph mental model)
|
||
- **Best for:** Production-grade stateful workflows, regulated industries, audit trails, human-in-the-loop flows, fault-tolerant multi-step agents
|
||
- **Avoid if:** Simple chatbot, purely linear workflow, rapid prototyping
|
||
- **Strengths:** Best checkpointing (every node), time-travel debugging, native Postgres/Redis persistence, streaming support, chosen by 62% of developers for stateful agent work (2026)
|
||
- **Weaknesses:** More upfront scaffolding, steeper curve, overkill for simple cases
|
||
- **Eval concerns:** State transition correctness, goal completion rate, tool use accuracy, safety guardrails
|
||
|
||
### OpenAI Agents SDK
|
||
- **Type:** Native OpenAI agent framework
|
||
- **Language:** Python + TypeScript
|
||
- **Model support:** Optimized for OpenAI (supports 100+ via Chat Completions compatibility)
|
||
- **Learning curve:** Beginner (4 primitives: Agents, Handoffs, Guardrails, Tracing)
|
||
- **Best for:** OpenAI-committed teams, rapid agent prototyping, voice agents (gpt-realtime), teams wanting visual builder (AgentKit)
|
||
- **Avoid if:** Model flexibility needed, complex multi-agent collaboration, persistent state management required, vendor lock-in concern
|
||
- **Strengths:** Simplest mental model, built-in tracing and guardrails, Handoffs for agent delegation, Realtime Agents for voice
|
||
- **Weaknesses:** OpenAI vendor lock-in, no built-in persistent state, younger ecosystem
|
||
- **Eval concerns:** Instruction following, safety guardrails, escalation accuracy, tone consistency
|
||
|
||
### Claude Agent SDK (Anthropic)
|
||
- **Type:** Code-aware autonomous agent framework
|
||
- **Language:** Python + TypeScript
|
||
- **Model support:** Claude models only
|
||
- **Learning curve:** Intermediate (18 hook events, MCP, tool decorators)
|
||
- **Best for:** Developer tooling, code generation/review agents, autonomous coding assistants, MCP-heavy architectures, safety-critical applications
|
||
- **Avoid if:** Model flexibility needed, stable/mature API required, use case unrelated to code/tool-use
|
||
- **Strengths:** Deepest MCP integration, built-in filesystem/shell access, 18 lifecycle hooks, automatic context compaction, extended thinking, safety-first design
|
||
- **Weaknesses:** Claude-only vendor lock-in, newer/evolving API, smaller community
|
||
- **Eval concerns:** Tool use correctness, safety, code quality, instruction following
|
||
|
||
### AutoGen / AG2 / Microsoft Agent Framework
|
||
- **Type:** Multi-agent conversational framework
|
||
- **Language:** Python (AG2), Python + .NET (Microsoft Agent Framework)
|
||
- **Model support:** Model-agnostic
|
||
- **Learning curve:** Intermediate–Advanced
|
||
- **Best for:** Research applications, conversational problem-solving, code generation + execution loops, Microsoft/.NET shops
|
||
- **Avoid if:** You want ecosystem stability, deterministic workflows, or "safest long-term bet" (fragmentation risk)
|
||
- **Strengths:** Most sophisticated conversational agent patterns, code generation + execution loop, async event-driven (v0.4+), cross-language interop (Microsoft Agent Framework)
|
||
- **Weaknesses:** Ecosystem fragmented (AutoGen maintenance mode, AG2 fork, Microsoft Agent Framework preview) — genuine long-term risk
|
||
- **Eval concerns:** Conversation goal completion, consensus quality, code execution correctness
|
||
|
||
### Google ADK (Agent Development Kit)
|
||
- **Type:** Multi-agent orchestration framework
|
||
- **Language:** Python + Java
|
||
- **Model support:** Optimized for Gemini; supports other models via LiteLLM
|
||
- **Learning curve:** Intermediate (agent/tool/session model, familiar if you know LangGraph)
|
||
- **Best for:** Google Cloud / Vertex AI shops, multi-agent workflows needing built-in session management and memory, teams already committed to Gemini, agent pipelines that need Google Search / BigQuery tool integration
|
||
- **Avoid if:** Model flexibility is required beyond Gemini, no Google Cloud dependency acceptable, TypeScript-only stack
|
||
- **Strengths:** First-party Google support, built-in session/memory/artifact management, tight Vertex AI and Google Search integration, own eval framework (RAGAS-compatible), multi-agent by design (sequential, parallel, loop patterns), Java SDK for enterprise teams
|
||
- **Weaknesses:** Gemini vendor lock-in in practice, younger community than LangChain/LlamaIndex, less third-party integration depth
|
||
- **Eval concerns:** Multi-agent task decomposition, tool use correctness, session state consistency, goal completion rate
|
||
|
||
### Haystack
|
||
- **Type:** NLP pipeline framework
|
||
- **Language:** Python
|
||
- **Model support:** Model-agnostic
|
||
- **Learning curve:** Intermediate
|
||
- **Best for:** Explicit, auditable NLP pipelines, document processing with fine-grained control, enterprise search, regulated industries needing transparency
|
||
- **Avoid if:** Rapid prototyping, multi-agent workflows, or you want a large community
|
||
- **Strengths:** Explicit pipeline control, strong for structured data pipelines, good documentation
|
||
- **Weaknesses:** Smaller community, less agent-oriented than alternatives
|
||
- **Eval concerns:** Extraction accuracy, pipeline output validity, retrieval quality
|
||
|
||
---
|
||
|
||
## Decision Dimensions
|
||
|
||
### By System Type
|
||
|
||
| System Type | Primary Framework(s) | Key Eval Concerns |
|
||
|-------------|---------------------|-------------------|
|
||
| RAG / Knowledge Q&A | LlamaIndex, LangChain | Context faithfulness, hallucination, retrieval precision/recall |
|
||
| Multi-agent orchestration | CrewAI, LangGraph, Google ADK | Task decomposition, handoff quality, goal completion |
|
||
| Conversational assistants | OpenAI Agents SDK, Claude Agent SDK | Tone, safety, instruction following, escalation |
|
||
| Structured data extraction | LangChain, LlamaIndex | Schema compliance, extraction accuracy |
|
||
| Autonomous task agents | LangGraph, OpenAI Agents SDK | Safety guardrails, tool correctness, cost adherence |
|
||
| Content generation | Claude Agent SDK, OpenAI Agents SDK | Brand voice, factual accuracy, tone |
|
||
| Code automation | Claude Agent SDK | Code correctness, safety, test pass rate |
|
||
|
||
### By Team Size and Stage
|
||
|
||
| Context | Recommendation |
|
||
|---------|----------------|
|
||
| Solo dev, prototyping | OpenAI Agents SDK or CrewAI (fastest to running) |
|
||
| Solo dev, RAG | LlamaIndex (batteries included) |
|
||
| Team, production, stateful | LangGraph (best fault tolerance) |
|
||
| Team, evolving requirements | LangChain (broadest escape hatches) |
|
||
| Team, multi-agent | CrewAI (simplest role abstraction) |
|
||
| Enterprise, .NET | AutoGen AG2 / Microsoft Agent Framework |
|
||
|
||
### By Model Commitment
|
||
|
||
| Preference | Framework |
|
||
|-----------|-----------|
|
||
| OpenAI-only | OpenAI Agents SDK |
|
||
| Anthropic/Claude-only | Claude Agent SDK |
|
||
| Google/Gemini-committed | Google ADK |
|
||
| Model-agnostic (full flexibility) | LangChain, LlamaIndex, CrewAI, LangGraph, Haystack |
|
||
|
||
---
|
||
|
||
## Anti-Patterns
|
||
|
||
1. **Using LangChain for simple chatbots** — Direct SDK call is less code, faster, and easier to debug
|
||
2. **Using CrewAI for complex stateful workflows** — Checkpointing gaps will bite you in production
|
||
3. **Using OpenAI Agents SDK with non-OpenAI models** — Loses the integration benefits you chose it for
|
||
4. **Using LlamaIndex as a multi-agent framework** — It can do agents, but that's not its strength
|
||
5. **Defaulting to LangChain without evaluating alternatives** — "Everyone uses it" ≠ right for your use case
|
||
6. **Starting a new project on AutoGen (not AG2)** — AutoGen is in maintenance mode; use AG2 or wait for Microsoft Agent Framework GA
|
||
7. **Choosing LangGraph for simple linear flows** — The graph overhead is not worth it; use LangChain chains instead
|
||
8. **Ignoring vendor lock-in** — Provider-native SDKs (OpenAI, Claude) trade flexibility for integration depth; decide consciously
|
||
|
||
---
|
||
|
||
## Combination Plays (Multi-Framework Stacks)
|
||
|
||
| Production Pattern | Stack |
|
||
|-------------------|-------|
|
||
| RAG with observability | LlamaIndex + LangSmith or Langfuse |
|
||
| Stateful agent with RAG | LangGraph + LlamaIndex |
|
||
| Multi-agent with tracing | CrewAI + Langfuse |
|
||
| OpenAI agents with evals | OpenAI Agents SDK + Promptfoo or Braintrust |
|
||
| Claude agents with MCP | Claude Agent SDK + LangSmith or Arize Phoenix |
|