feat: /gsd-ai-integration-phase + /gsd-eval-review — AI framework selection and eval coverage layer (#1971)
* feat: /gsd:ai-phase + /gsd:eval-review — AI evals and framework selection layer Adds a structured AI development layer to GSD with 5 new agents, 2 new commands, 2 new workflows, 2 reference files, and 1 template. Commands: - /gsd:ai-phase [N] — pre-planning AI design contract (inserts between discuss-phase and plan-phase). Orchestrates 4 agents in sequence: framework-selector → ai-researcher → domain-researcher → eval-planner. Output: AI-SPEC.md with framework decision, implementation guidance, domain expert context, and evaluation strategy. - /gsd:eval-review [N] — retroactive eval coverage audit. Scores each planned eval dimension as COVERED/PARTIAL/MISSING. Output: EVAL-REVIEW.md with 0-100 score, verdict, and remediation plan. Agents: - gsd-framework-selector: interactive decision matrix (6 questions) → scored framework recommendation for CrewAI, LlamaIndex, LangChain, LangGraph, OpenAI Agents SDK, Claude Agent SDK, AutoGen/AG2, Haystack - gsd-ai-researcher: fetches official framework docs + writes AI systems best practices (Pydantic structured outputs, async-first, prompt discipline, context window management, cost/latency budget) - gsd-domain-researcher: researches business domain and use-case context — surfaces domain expert evaluation criteria, industry failure modes, regulatory constraints, and practitioner rubric ingredients before eval-planner writes measurable criteria - gsd-eval-planner: designs evaluation strategy grounded in domain context; defaults to Arize Phoenix (tracing) + RAGAS (RAG eval) with detect-first guard for existing tooling - gsd-eval-auditor: retroactive codebase scan → scores eval coverage Integration points: - plan-phase: non-blocking nudge (step 4.5) when AI keywords detected and no AI-SPEC.md present - settings: new workflow.ai_phase toggle (default on) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: refine ai-integration-phase layer — rename, house style, consistency fixes Amends the ai-evals framework layer (df8cb6c) with post-review improvements before opening upstream PR. Rename /gsd:ai-phase → /gsd:ai-integration-phase: - Renamed commands/gsd/ai-phase.md → ai-integration-phase.md - Renamed get-shit-done/workflows/ai-phase.md → ai-integration-phase.md - Updated config key: workflow.ai_phase → workflow.ai_integration_phase - Updated repair action: addAiPhaseKey → addAiIntegrationPhaseKey - Updated all 84 cross-references across agents, workflows, templates, tests Consistency fixes (same class as PR #1380 review): - commands/gsd: objective described 3-agent chain, missing gsd-domain-researcher - workflows/ai-integration-phase: purpose tag described 3-agent chain + "locks three things" — updated to 4 agents + 4 outputs - workflows/ai-integration-phase: missing DOMAIN_MODEL resolve-model call in step 1 (domain-researcher was spawned in step 7.5 with no model variable) - workflows/ai-integration-phase: fractional step ## 7.5 renumbered to integers (steps 8–12 shifted) Agent house style (GSD meta-prompting conformance): - All 5 new agents refactored to execution_flow + step name="" structure - Role blocks compressed to 2 lines (removed verbose "Core responsibilities") - Added skills: frontmatter to all 5 agents (agent-frontmatter tests) - Added # hooks: commented pattern to file-writing agents - Added ALWAYS use Write tool anti-heredoc instruction to file-writing agents - Line reductions: ai-researcher −41%, domain-researcher −25%, eval-planner −26%, eval-auditor −25%, framework-selector −9% Test coverage (tests/ai-evals.test.cjs — 48 tests): - CONFIG: workflow.ai_integration_phase defaults and config-set/get - HEALTH: W010 warning emission and addAiIntegrationPhaseKey repair - TEMPLATE: AI-SPEC.md section completeness (10 sections) - COMMAND: ai-integration-phase + eval-review frontmatter validity - AGENTS: all 5 new agent files exist - REFERENCES: ai-evals.md + ai-frameworks.md exist and are non-empty - WORKFLOW: plan-phase nudge integration, workflow files exist + agent coverage 603/603 tests passing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat: add Google ADK to framework selector and reference matrix Google ADK (released March 2025) was missing from the framework options. Adds Python + Java multi-agent framework optimised for Gemini / Vertex AI. - get-shit-done/references/ai-frameworks.md: add Google ADK profile (type, language, model support, best for, avoid if, strengths, weaknesses, eval concerns); update Quick Picks, By System Type, and By Model Commitment tables - agents/gsd-framework-selector.md: add "Google (Gemini)" to model provider interview question - agents/gsd-ai-researcher.md: add Google ADK docs URL to documentation_sources Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: adapt to upstream conventions post-rebase - Remove skills: frontmatter from all 5 new agents (upstream changed convention — skills: breaks Gemini CLI and must not be present) - Add workflow.ai_integration_phase to VALID_CONFIG_KEYS whitelist in config.cjs (config-set blocked unknown keys) - Add ai_integration_phase: true to CONFIG_DEFAULTS in core.cjs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: rephrase 4b.1 line to avoid false-positive in prompt-injection scan "contract as a Pydantic model" matched the `act as a` pattern case-insensitively. Rephrased to "output schema using a Pydantic model". Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: adapt to upstream conventions (W016, colon refs, config docs) - Replace verify.cjs from upstream to restore W010-W015 + cmdValidateAgents, lost when rebase conflict was resolved with --theirs - Add W016 (workflow.ai_integration_phase absent) inside the config try block, avoids collision with upstream's W010 agent-installation check - Add addAiIntegrationPhaseKey repair case mirroring addNyquistKey pattern - Replace /gsd: colon format with /gsd- hyphen format across all new files (agents, workflows, templates, verify.cjs) per stale-colon-refs guard (#1748) - Add workflow.ai_integration_phase to planning-config.md reference table - Add ai_integration_phase → workflow.ai_integration_phase to NAMESPACE_MAP in config-field-docs.test.cjs so CONFIG_DEFAULTS coverage check passes - Update ai-evals tests to use W016 instead of W010 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: add 5 new agents to E2E Copilot install expected list gsd-ai-researcher, gsd-domain-researcher, gsd-eval-auditor, gsd-eval-planner, gsd-framework-selector added to the hardcoded expected agent list in copilot-install.test.cjs (#1890). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
282
get-shit-done/workflows/ai-integration-phase.md
Normal file
282
get-shit-done/workflows/ai-integration-phase.md
Normal file
@@ -0,0 +1,282 @@
|
||||
<purpose>
|
||||
Generate an AI design contract (AI-SPEC.md) for phases that involve building AI systems. Orchestrates gsd-framework-selector → gsd-ai-researcher → gsd-domain-researcher → gsd-eval-planner with a validation gate. Inserts between discuss-phase and plan-phase in the GSD lifecycle.
|
||||
|
||||
AI-SPEC.md locks four things before the planner creates tasks:
|
||||
1. Framework selection (with rationale and alternatives)
|
||||
2. Implementation guidance (correct syntax, patterns, pitfalls from official docs)
|
||||
3. Domain context (practitioner rubric ingredients, failure modes, regulatory constraints)
|
||||
4. Evaluation strategy (dimensions, rubrics, tooling, reference dataset, guardrails)
|
||||
|
||||
This prevents the two most common AI development failures: choosing the wrong framework for the use case, and treating evaluation as an afterthought.
|
||||
</purpose>
|
||||
|
||||
<required_reading>
|
||||
@~/.claude/get-shit-done/references/ai-frameworks.md
|
||||
@~/.claude/get-shit-done/references/ai-evals.md
|
||||
</required_reading>
|
||||
|
||||
<process>
|
||||
|
||||
## 1. Initialize
|
||||
|
||||
```bash
|
||||
INIT=$(node "$HOME/.claude/get-shit-done/bin/gsd-tools.cjs" init plan-phase "$PHASE")
|
||||
if [[ "$INIT" == @file:* ]]; then INIT=$(cat "${INIT#@file:}"); fi
|
||||
```
|
||||
|
||||
Parse JSON for: `phase_dir`, `phase_number`, `phase_name`, `phase_slug`, `padded_phase`, `has_context`, `has_research`, `commit_docs`.
|
||||
|
||||
**File paths:** `state_path`, `roadmap_path`, `requirements_path`, `context_path`.
|
||||
|
||||
Resolve agent models:
|
||||
```bash
|
||||
SELECTOR_MODEL=$(node "$HOME/.claude/get-shit-done/bin/gsd-tools.cjs" resolve-model gsd-framework-selector --raw)
|
||||
RESEARCHER_MODEL=$(node "$HOME/.claude/get-shit-done/bin/gsd-tools.cjs" resolve-model gsd-ai-researcher --raw)
|
||||
DOMAIN_MODEL=$(node "$HOME/.claude/get-shit-done/bin/gsd-tools.cjs" resolve-model gsd-domain-researcher --raw)
|
||||
PLANNER_MODEL=$(node "$HOME/.claude/get-shit-done/bin/gsd-tools.cjs" resolve-model gsd-eval-planner --raw)
|
||||
```
|
||||
|
||||
Check config:
|
||||
```bash
|
||||
AI_PHASE_ENABLED=$(node "$HOME/.claude/get-shit-done/bin/gsd-tools.cjs" config-get workflow.ai_integration_phase 2>/dev/null || echo "true")
|
||||
```
|
||||
|
||||
**If `AI_PHASE_ENABLED` is `false`:**
|
||||
```
|
||||
AI phase is disabled in config. Enable via /gsd-settings.
|
||||
```
|
||||
Exit workflow.
|
||||
|
||||
**If `planning_exists` is false:** Error — run `/gsd-new-project` first.
|
||||
|
||||
## 2. Parse and Validate Phase
|
||||
|
||||
Extract phase number from $ARGUMENTS. If not provided, detect next unplanned phase.
|
||||
|
||||
```bash
|
||||
PHASE_INFO=$(node "$HOME/.claude/get-shit-done/bin/gsd-tools.cjs" roadmap get-phase "${PHASE}")
|
||||
```
|
||||
|
||||
**If `found` is false:** Error with available phases.
|
||||
|
||||
## 3. Check Prerequisites
|
||||
|
||||
**If `has_context` is false:**
|
||||
```
|
||||
No CONTEXT.md found for Phase {N}.
|
||||
Recommended: run /gsd-discuss-phase {N} first to capture framework preferences.
|
||||
Continuing without user decisions — framework selector will ask all questions.
|
||||
```
|
||||
Continue (non-blocking).
|
||||
|
||||
## 4. Check Existing AI-SPEC
|
||||
|
||||
```bash
|
||||
AI_SPEC_FILE=$(ls "${PHASE_DIR}"/*-AI-SPEC.md 2>/dev/null | head -1)
|
||||
```
|
||||
|
||||
**If exists:** Use AskUserQuestion:
|
||||
- header: "Existing AI-SPEC"
|
||||
- question: "AI-SPEC.md already exists for Phase {N}. What would you like to do?"
|
||||
- options:
|
||||
- "Update — re-run with existing as baseline"
|
||||
- "View — display current AI-SPEC and exit"
|
||||
- "Skip — keep current AI-SPEC and exit"
|
||||
|
||||
If "View": display file contents, exit.
|
||||
If "Skip": exit.
|
||||
If "Update": continue to step 5.
|
||||
|
||||
## 5. Spawn gsd-framework-selector
|
||||
|
||||
Display:
|
||||
```
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
GSD ► AI DESIGN CONTRACT — PHASE {N}: {name}
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
◆ Step 1/4 — Framework Selection...
|
||||
```
|
||||
|
||||
Spawn `gsd-framework-selector` with:
|
||||
```markdown
|
||||
Read ~/.claude/agents/gsd-framework-selector.md for instructions.
|
||||
|
||||
<objective>
|
||||
Select the right AI framework for Phase {phase_number}: {phase_name}
|
||||
Goal: {phase_goal}
|
||||
</objective>
|
||||
|
||||
<files_to_read>
|
||||
{context_path if exists}
|
||||
{requirements_path if exists}
|
||||
</files_to_read>
|
||||
|
||||
<phase_context>
|
||||
Phase: {phase_number} — {phase_name}
|
||||
Goal: {phase_goal}
|
||||
</phase_context>
|
||||
```
|
||||
|
||||
Parse selector output for: `primary_framework`, `system_type`, `model_provider`, `eval_concerns`, `alternative_framework`.
|
||||
|
||||
**If selector fails or returns empty:** Exit with error — "Framework selection failed. Re-run /gsd-ai-integration-phase {N} or answer the framework question in /gsd-discuss-phase {N} first."
|
||||
|
||||
## 6. Initialize AI-SPEC.md
|
||||
|
||||
Copy template:
|
||||
```bash
|
||||
cp "$HOME/.claude/get-shit-done/templates/AI-SPEC.md" "${PHASE_DIR}/${PADDED_PHASE}-AI-SPEC.md"
|
||||
```
|
||||
|
||||
Fill in header fields:
|
||||
- Phase number and name
|
||||
- System classification (from selector)
|
||||
- Selected framework (from selector)
|
||||
- Alternative considered (from selector)
|
||||
|
||||
## 7. Spawn gsd-ai-researcher
|
||||
|
||||
Display:
|
||||
```
|
||||
◆ Step 2/4 — Researching {primary_framework} docs + AI systems best practices...
|
||||
```
|
||||
|
||||
Spawn `gsd-ai-researcher` with:
|
||||
```markdown
|
||||
Read ~/.claude/agents/gsd-ai-researcher.md for instructions.
|
||||
|
||||
<objective>
|
||||
Research {primary_framework} for Phase {phase_number}: {phase_name}
|
||||
Write Sections 3 and 4 of AI-SPEC.md
|
||||
</objective>
|
||||
|
||||
<files_to_read>
|
||||
{ai_spec_path}
|
||||
{context_path if exists}
|
||||
</files_to_read>
|
||||
|
||||
<input>
|
||||
framework: {primary_framework}
|
||||
system_type: {system_type}
|
||||
model_provider: {model_provider}
|
||||
ai_spec_path: {ai_spec_path}
|
||||
phase_context: Phase {phase_number}: {phase_name} — {phase_goal}
|
||||
</input>
|
||||
```
|
||||
|
||||
## 8. Spawn gsd-domain-researcher
|
||||
|
||||
Display:
|
||||
```
|
||||
◆ Step 3/4 — Researching domain context and expert evaluation criteria...
|
||||
```
|
||||
|
||||
Spawn `gsd-domain-researcher` with:
|
||||
```markdown
|
||||
Read ~/.claude/agents/gsd-domain-researcher.md for instructions.
|
||||
|
||||
<objective>
|
||||
Research the business domain and expert evaluation criteria for Phase {phase_number}: {phase_name}
|
||||
Write Section 1b (Domain Context) of AI-SPEC.md
|
||||
</objective>
|
||||
|
||||
<files_to_read>
|
||||
{ai_spec_path}
|
||||
{context_path if exists}
|
||||
{requirements_path if exists}
|
||||
</files_to_read>
|
||||
|
||||
<input>
|
||||
system_type: {system_type}
|
||||
phase_name: {phase_name}
|
||||
phase_goal: {phase_goal}
|
||||
ai_spec_path: {ai_spec_path}
|
||||
</input>
|
||||
```
|
||||
|
||||
## 9. Spawn gsd-eval-planner
|
||||
|
||||
Display:
|
||||
```
|
||||
◆ Step 4/4 — Designing evaluation strategy from domain + technical context...
|
||||
```
|
||||
|
||||
Spawn `gsd-eval-planner` with:
|
||||
```markdown
|
||||
Read ~/.claude/agents/gsd-eval-planner.md for instructions.
|
||||
|
||||
<objective>
|
||||
Design evaluation strategy for Phase {phase_number}: {phase_name}
|
||||
Write Sections 5, 6, and 7 of AI-SPEC.md
|
||||
AI-SPEC.md now contains domain context (Section 1b) — use it as your rubric starting point.
|
||||
</objective>
|
||||
|
||||
<files_to_read>
|
||||
{ai_spec_path}
|
||||
{context_path if exists}
|
||||
{requirements_path if exists}
|
||||
</files_to_read>
|
||||
|
||||
<input>
|
||||
system_type: {system_type}
|
||||
framework: {primary_framework}
|
||||
model_provider: {model_provider}
|
||||
phase_name: {phase_name}
|
||||
phase_goal: {phase_goal}
|
||||
ai_spec_path: {ai_spec_path}
|
||||
</input>
|
||||
```
|
||||
|
||||
## 10. Validate AI-SPEC Completeness
|
||||
|
||||
Read the completed AI-SPEC.md. Check that:
|
||||
- Section 2 has a framework name (not placeholder)
|
||||
- Section 1b has at least one domain rubric ingredient (Good/Bad/Stakes)
|
||||
- Section 3 has a non-empty code block (entry point pattern)
|
||||
- Section 4b has a Pydantic example
|
||||
- Section 5 has at least one row in the dimensions table
|
||||
- Section 6 has at least one guardrail or explicit "N/A for internal tool" note
|
||||
- Checklist section at end has 3+ items checked
|
||||
|
||||
**If validation fails:** Display specific missing sections. Ask user if they want to re-run the specific step or continue anyway.
|
||||
|
||||
## 11. Commit
|
||||
|
||||
**If `commit_docs` is true:**
|
||||
```bash
|
||||
git add "${AI_SPEC_FILE}"
|
||||
git commit -m "docs({phase_slug}): generate AI-SPEC.md — {primary_framework} + domain context + eval strategy"
|
||||
```
|
||||
|
||||
## 12. Display Completion
|
||||
|
||||
```
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
GSD ► AI-SPEC COMPLETE — PHASE {N}: {name}
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
◆ Framework: {primary_framework}
|
||||
◆ System Type: {system_type}
|
||||
◆ Domain: {domain_vertical from Section 1b}
|
||||
◆ Eval Dimensions: {eval_concerns}
|
||||
◆ Tracing Default: Arize Phoenix (or detected existing tool)
|
||||
◆ Output: {ai_spec_path}
|
||||
|
||||
Next step:
|
||||
/gsd-plan-phase {N} — planner will consume AI-SPEC.md
|
||||
```
|
||||
|
||||
</process>
|
||||
|
||||
<success_criteria>
|
||||
- [ ] Framework selected with rationale (Section 2)
|
||||
- [ ] AI-SPEC.md created from template
|
||||
- [ ] Framework docs + AI best practices researched (Sections 3, 4, 4b populated)
|
||||
- [ ] Domain context + expert rubric ingredients researched (Section 1b populated)
|
||||
- [ ] Eval strategy grounded in domain context (Sections 5-7 populated)
|
||||
- [ ] Arize Phoenix (or detected tool) set as tracing default in Section 7
|
||||
- [ ] AI-SPEC.md validated (Sections 1b, 2, 3, 4b, 5, 6 all non-empty)
|
||||
- [ ] Committed if commit_docs enabled
|
||||
- [ ] Next step surfaced to user
|
||||
</success_criteria>
|
||||
153
get-shit-done/workflows/eval-review.md
Normal file
153
get-shit-done/workflows/eval-review.md
Normal file
@@ -0,0 +1,153 @@
|
||||
<purpose>
|
||||
Retroactive audit of an implemented AI phase's evaluation coverage. Standalone command that works on any GSD-managed AI phase. Produces a scored EVAL-REVIEW.md with gap analysis and remediation plan.
|
||||
|
||||
Use after /gsd-execute-phase to verify that the evaluation strategy from AI-SPEC.md was actually implemented. Mirrors the pattern of /gsd-ui-review and /gsd-validate-phase.
|
||||
</purpose>
|
||||
|
||||
<required_reading>
|
||||
@~/.claude/get-shit-done/references/ai-evals.md
|
||||
</required_reading>
|
||||
|
||||
<process>
|
||||
|
||||
## 0. Initialize
|
||||
|
||||
```bash
|
||||
INIT=$(node "$HOME/.claude/get-shit-done/bin/gsd-tools.cjs" init phase-op "${PHASE_ARG}")
|
||||
if [[ "$INIT" == @file:* ]]; then INIT=$(cat "${INIT#@file:}"); fi
|
||||
```
|
||||
|
||||
Parse: `phase_dir`, `phase_number`, `phase_name`, `phase_slug`, `padded_phase`, `commit_docs`.
|
||||
|
||||
```bash
|
||||
AUDITOR_MODEL=$(node "$HOME/.claude/get-shit-done/bin/gsd-tools.cjs" resolve-model gsd-eval-auditor --raw)
|
||||
```
|
||||
|
||||
Display banner:
|
||||
```
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
GSD ► EVAL AUDIT — PHASE {N}: {name}
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
```
|
||||
|
||||
## 1. Detect Input State
|
||||
|
||||
```bash
|
||||
SUMMARY_FILES=$(ls "${PHASE_DIR}"/*-SUMMARY.md 2>/dev/null)
|
||||
AI_SPEC_FILE=$(ls "${PHASE_DIR}"/*-AI-SPEC.md 2>/dev/null | head -1)
|
||||
EVAL_REVIEW_FILE=$(ls "${PHASE_DIR}"/*-EVAL-REVIEW.md 2>/dev/null | head -1)
|
||||
```
|
||||
|
||||
**State A** — AI-SPEC.md + SUMMARY.md exist: Full audit against spec
|
||||
**State B** — SUMMARY.md exists, no AI-SPEC.md: Audit against general best practices
|
||||
**State C** — No SUMMARY.md: Exit — "Phase {N} not executed. Run /gsd-execute-phase {N} first."
|
||||
|
||||
**If `EVAL_REVIEW_FILE` non-empty:** Use AskUserQuestion:
|
||||
- header: "Existing Eval Review"
|
||||
- question: "EVAL-REVIEW.md already exists for Phase {N}."
|
||||
- options:
|
||||
- "Re-audit — run fresh audit"
|
||||
- "View — display current review and exit"
|
||||
|
||||
If "View": display file, exit.
|
||||
If "Re-audit": continue.
|
||||
|
||||
**If State B (no AI-SPEC.md):** Warn:
|
||||
```
|
||||
No AI-SPEC.md found for Phase {N}.
|
||||
Audit will evaluate against general AI eval best practices rather than a phase-specific plan.
|
||||
Consider running /gsd-ai-integration-phase {N} before implementation next time.
|
||||
```
|
||||
Continue (non-blocking).
|
||||
|
||||
## 2. Gather Context Paths
|
||||
|
||||
Build file list for auditor:
|
||||
- AI-SPEC.md (if exists — the planned eval strategy)
|
||||
- All SUMMARY.md files in phase dir
|
||||
- All PLAN.md files in phase dir
|
||||
|
||||
## 3. Spawn gsd-eval-auditor
|
||||
|
||||
```
|
||||
◆ Spawning eval auditor...
|
||||
```
|
||||
|
||||
Build prompt:
|
||||
|
||||
```markdown
|
||||
Read ~/.claude/agents/gsd-eval-auditor.md for instructions.
|
||||
|
||||
<objective>
|
||||
Conduct evaluation coverage audit of Phase {phase_number}: {phase_name}
|
||||
{If AI-SPEC exists: "Audit against AI-SPEC.md evaluation plan."}
|
||||
{If no AI-SPEC: "Audit against general AI eval best practices."}
|
||||
</objective>
|
||||
|
||||
<files_to_read>
|
||||
- {summary_paths}
|
||||
- {plan_paths}
|
||||
- {ai_spec_path if exists}
|
||||
</files_to_read>
|
||||
|
||||
<input>
|
||||
ai_spec_path: {ai_spec_path or "none"}
|
||||
phase_dir: {phase_dir}
|
||||
phase_number: {phase_number}
|
||||
phase_name: {phase_name}
|
||||
padded_phase: {padded_phase}
|
||||
state: {A or B}
|
||||
</input>
|
||||
```
|
||||
|
||||
Spawn as Task with model `AUDITOR_MODEL`.
|
||||
|
||||
## 4. Parse Auditor Result
|
||||
|
||||
Read the written EVAL-REVIEW.md. Extract:
|
||||
- `overall_score`
|
||||
- `verdict` (PRODUCTION READY | NEEDS WORK | SIGNIFICANT GAPS | NOT IMPLEMENTED)
|
||||
- `critical_gap_count`
|
||||
|
||||
## 5. Display Summary
|
||||
|
||||
```
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
GSD ► EVAL AUDIT COMPLETE — PHASE {N}: {name}
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
◆ Score: {overall_score}/100
|
||||
◆ Verdict: {verdict}
|
||||
◆ Critical Gaps: {critical_gap_count}
|
||||
◆ Output: {eval_review_path}
|
||||
|
||||
{If PRODUCTION READY:}
|
||||
Next step: /gsd-plan-phase (next phase) or deploy
|
||||
|
||||
{If NEEDS WORK:}
|
||||
Address critical gaps in EVAL-REVIEW.md, then re-run /gsd-eval-review {N}
|
||||
|
||||
{If SIGNIFICANT GAPS or NOT IMPLEMENTED:}
|
||||
Review AI-SPEC.md evaluation plan. Critical eval dimensions are not implemented.
|
||||
Do not deploy until gaps are addressed.
|
||||
```
|
||||
|
||||
## 6. Commit
|
||||
|
||||
**If `commit_docs` is true:**
|
||||
```bash
|
||||
git add "${EVAL_REVIEW_FILE}"
|
||||
git commit -m "docs({phase_slug}): add EVAL-REVIEW.md — score {overall_score}/100 ({verdict})"
|
||||
```
|
||||
|
||||
</process>
|
||||
|
||||
<success_criteria>
|
||||
- [ ] Phase execution state detected correctly
|
||||
- [ ] AI-SPEC.md presence handled (with or without)
|
||||
- [ ] gsd-eval-auditor spawned with correct context
|
||||
- [ ] EVAL-REVIEW.md written (by auditor)
|
||||
- [ ] Score and verdict displayed to user
|
||||
- [ ] Appropriate next steps surfaced based on verdict
|
||||
- [ ] Committed if commit_docs enabled
|
||||
</success_criteria>
|
||||
@@ -241,6 +241,46 @@ If "Run discuss-phase first":
|
||||
```
|
||||
**Exit the plan-phase workflow. Do not continue.**
|
||||
|
||||
## 4.5. Check AI-SPEC
|
||||
|
||||
**Skip if:** `ai_integration_phase_enabled` from config is false, or `--skip-ai-spec` flag provided.
|
||||
|
||||
```bash
|
||||
AI_SPEC_FILE=$(ls "${PHASE_DIR}"/*-AI-SPEC.md 2>/dev/null | head -1)
|
||||
AI_PHASE_CFG=$(node "$HOME/.claude/get-shit-done/bin/gsd-tools.cjs" config-get workflow.ai_integration_phase 2>/dev/null || echo "true")
|
||||
```
|
||||
|
||||
**Skip if `AI_PHASE_CFG` is `false`.**
|
||||
|
||||
**If `AI_SPEC_FILE` is empty:** Check phase goal for AI keywords:
|
||||
```bash
|
||||
echo "${phase_goal}" | grep -qi "agent\|llm\|rag\|chatbot\|embedding\|langchain\|llamaindex\|crewai\|langgraph\|openai\|anthropic\|vector\|eval\|ai system"
|
||||
```
|
||||
|
||||
**If AI keywords detected AND no AI-SPEC.md:**
|
||||
```
|
||||
◆ Note: This phase appears to involve AI system development.
|
||||
Consider running /gsd-ai-integration-phase {N} before planning to:
|
||||
- Select the right framework for your use case
|
||||
- Research its docs and best practices
|
||||
- Design an evaluation strategy
|
||||
|
||||
Continue planning without AI-SPEC? (non-blocking — /gsd-ai-integration-phase can be run after)
|
||||
```
|
||||
|
||||
Use AskUserQuestion with options:
|
||||
- "Continue — plan without AI-SPEC"
|
||||
- "Stop — I'll run /gsd-ai-integration-phase {N} first"
|
||||
|
||||
If "Stop": Exit with `/gsd-ai-integration-phase {N}` reminder.
|
||||
If "Continue": Proceed. (Non-blocking — planner will note AI-SPEC is absent.)
|
||||
|
||||
**If `AI_SPEC_FILE` is non-empty:** Extract framework for planner context:
|
||||
```bash
|
||||
FRAMEWORK_LINE=$(grep "Selected Framework:" "${AI_SPEC_FILE}" | head -1)
|
||||
```
|
||||
Pass `ai_spec_path` and `framework_line` to planner in step 7 so it can reference the AI design contract.
|
||||
|
||||
## 5. Handle Research
|
||||
|
||||
**Skip if:** `--gaps` flag or `--skip-research` flag or `--reviews` flag.
|
||||
|
||||
@@ -32,6 +32,7 @@ Parse current values (default to `true` if not present):
|
||||
- `workflow.nyquist_validation` — validation architecture research during plan-phase (default: true if absent)
|
||||
- `workflow.ui_phase` — generate UI-SPEC.md design contracts for frontend phases (default: true if absent)
|
||||
- `workflow.ui_safety_gate` — prompt to run /gsd-ui-phase before planning frontend phases (default: true if absent)
|
||||
- `workflow.ai_integration_phase` — framework selection + eval strategy for AI phases (default: true if absent)
|
||||
- `model_profile` — which model each agent uses (default: `balanced`)
|
||||
- `git.branching_strategy` — branching approach (default: `"none"`)
|
||||
- `workflow.use_worktrees` — whether parallel executor agents run in worktree isolation (default: `true`)
|
||||
@@ -118,6 +119,15 @@ AskUserQuestion([
|
||||
{ label: "No", description: "No prompt — plan-phase proceeds without UI-SPEC check." }
|
||||
]
|
||||
},
|
||||
{
|
||||
question: "Enable AI Phase? (framework selection + eval strategy for AI phases)",
|
||||
header: "AI Phase",
|
||||
multiSelect: false,
|
||||
options: [
|
||||
{ label: "Yes (Recommended)", description: "Run /gsd-ai-phase before planning AI system phases. Surfaces the right framework, researches its docs, and designs the evaluation strategy." },
|
||||
{ label: "No", description: "Skip AI design contract. Good for non-AI phases or when framework is already decided." }
|
||||
]
|
||||
},
|
||||
{
|
||||
question: "Git branching strategy?",
|
||||
header: "Branching",
|
||||
@@ -183,6 +193,7 @@ Merge new settings into existing config.json:
|
||||
"nyquist_validation": true/false,
|
||||
"ui_phase": true/false,
|
||||
"ui_safety_gate": true/false,
|
||||
"ai_integration_phase": true/false,
|
||||
"text_mode": true/false,
|
||||
"research_before_questions": true/false,
|
||||
"discuss_mode": "discuss" | "assumptions",
|
||||
@@ -244,6 +255,7 @@ Write `~/.gsd/defaults.json` with:
|
||||
"nyquist_validation": <current>,
|
||||
"ui_phase": <current>,
|
||||
"ui_safety_gate": <current>,
|
||||
"ai_integration_phase": <current>,
|
||||
"skip_discuss": <current>
|
||||
}
|
||||
}
|
||||
@@ -268,6 +280,7 @@ Display:
|
||||
| Nyquist Validation | {On/Off} |
|
||||
| UI Phase | {On/Off} |
|
||||
| UI Safety Gate | {On/Off} |
|
||||
| AI Integration Phase | {On/Off} |
|
||||
| Git Branching | {None/Per Phase/Per Milestone} |
|
||||
| Skip Discuss | {On/Off} |
|
||||
| Context Warnings | {On/Off} |
|
||||
@@ -287,7 +300,7 @@ Quick commands:
|
||||
|
||||
<success_criteria>
|
||||
- [ ] Current config read
|
||||
- [ ] User presented with 10 settings (profile + 8 workflow toggles + git branching)
|
||||
- [ ] User presented with 14 settings (profile + 11 workflow toggles + git branching + ctx warnings)
|
||||
- [ ] Config updated with model_profile, workflow, and git sections
|
||||
- [ ] User offered to save as global defaults (~/.gsd/defaults.json)
|
||||
- [ ] Changes confirmed to user
|
||||
|
||||
Reference in New Issue
Block a user