msd-core/get-shit-done/templates/AI-SPEC.md

# AI-SPEC — Phase {N}: {phase_name}

> AI design contract generated by `/gsd-ai-integration-phase`. Consumed by `gsd-planner` and `gsd-eval-auditor`.
> Locks framework selection, implementation guidance, and evaluation strategy before planning begins.

---

## 1. System Classification

**System Type:** <!-- RAG | Multi-Agent | Conversational | Extraction | Autonomous Agent | Content Generation | Code Automation | Hybrid -->

**Description:**
<!-- One-paragraph description of what this AI system does, who uses it, and what "good" looks like -->

**Critical Failure Modes:**
<!-- The 3-5 behaviors that absolutely cannot go wrong in this system -->
1.
2.
3.

---

## 1b. Domain Context

> Researched by `gsd-domain-researcher`. Grounds the evaluation strategy in domain expert knowledge.

**Industry Vertical:** <!-- healthcare | legal | finance | customer service | education | developer tooling | e-commerce | etc. -->

**User Population:** <!-- who uses this system and in what context -->

**Stakes Level:** <!-- Low | Medium | High | Critical -->

**Output Consequence:** <!-- what happens downstream when the AI output is acted on -->

### What Domain Experts Evaluate Against

<!-- Domain-specific rubric ingredients — in practitioner language, not AI jargon -->
<!-- Format: Dimension / Good (expert accepts) / Bad (expert flags) / Stakes / Source -->

### Known Failure Modes in This Domain

<!-- Domain-specific failure modes from research — not generic hallucination, but how it manifests here -->

### Regulatory / Compliance Context

<!-- Relevant regulations or constraints — or "None identified" if genuinely none apply -->

### Domain Expert Roles for Evaluation

| Role | Responsibility |
|------|---------------|
| <!-- e.g., Senior practitioner --> | <!-- Dataset labeling / rubric calibration / production sampling --> |

---

## 2. Framework Decision

**Selected Framework:** <!-- e.g., LlamaIndex v0.10.x -->

**Version:** <!-- Pin the version -->

**Rationale:**
<!-- Why this framework fits this system type, team context, and production requirements -->

**Alternatives Considered:**

| Framework | Ruled Out Because |
|-----------|------------------|
| | |

**Vendor Lock-In Accepted:** <!-- Yes / No / Partial — document the trade-off consciously -->

---

## 3. Framework Quick Reference

> Fetched from official docs by `gsd-ai-researcher`. Distilled for this specific use case.

### Installation
```bash
# Install command(s)
```

### Core Imports
```python
# Key imports for this use case
```

### Entry Point Pattern
```python
# Minimal working example for this system type
```

### Key Abstractions
<!-- Framework-specific concepts the developer must understand before coding -->
| Concept | What It Is | When You Use It |
|---------|-----------|-----------------|
| | | |

### Common Pitfalls
<!-- Gotchas specific to this framework and system type — from docs, issues, and community reports -->
1.
2.
3.

### Recommended Project Structure
```
project/
├── # Framework-specific folder layout
```

---

## 4. Implementation Guidance

**Model Configuration:**
<!-- Which model(s), temperature, max tokens, and other key parameters -->

**Core Pattern:**
<!-- The primary implementation pattern for this system type in this framework -->

**Tool Use:**
<!-- Tools/integrations needed and how to configure them -->

**State Management:**
<!-- How state is persisted, retrieved, and updated -->

**Context Window Strategy:**
<!-- How to manage context limits for this system type -->

---

## 4b. AI Systems Best Practices

> Written by `gsd-ai-researcher`. Cross-cutting patterns every developer building AI systems needs — independent of framework choice.

### Structured Outputs with Pydantic

<!-- Framework-specific Pydantic integration pattern for this use case -->
<!-- Include: output model definition, how the framework uses it, retry logic on validation failure -->

```python
# Pydantic output model for this system type
```

### Async-First Design

<!-- How async is handled in this framework, the one common mistake, and when to stream vs. await -->

### Prompt Engineering Discipline

<!-- System vs. user prompt separation, few-shot guidance, token budget strategy -->

### Context Window Management

<!-- Strategy specific to this system type: RAG chunking / conversation summarisation / agent compaction -->

### Cost and Latency Budget

<!-- Per-call cost estimate, caching strategy, sub-task model routing -->

---

## 5. Evaluation Strategy

### Dimensions

| Dimension | Rubric (Pass/Fail or 1-5) | Measurement Approach | Priority |
|-----------|--------------------------|---------------------|----------|
| | | Code / LLM Judge / Human | Critical / High / Medium |

### Eval Tooling

**Primary Tool:** <!-- e.g., RAGAS + Langfuse -->

**Setup:**
```bash
# Install and configure
```

**CI/CD Integration:**
```bash
# Command to run evals in CI/CD pipeline
```

### Reference Dataset

**Size:** <!-- e.g., 20 examples to start -->

**Composition:**
<!-- What scenario types the dataset covers: critical paths, edge cases, failure modes -->

**Labeling:**
<!-- Who labels examples and how (domain expert, LLM judge with calibration, etc.) -->

---

## 6. Guardrails

### Online (Real-Time)

| Guardrail | Trigger | Intervention |
|-----------|---------|--------------|
| | | Block / Escalate / Flag |

### Offline (Flywheel)

| Metric | Sampling Strategy | Action on Degradation |
|--------|------------------|----------------------|
| | | |

---

## 7. Production Monitoring

**Tracing Tool:** <!-- e.g., Langfuse self-hosted -->

**Key Metrics to Track:**
<!-- 3-5 metrics that will be monitored in production -->

**Alert Thresholds:**
<!-- When to page/alert -->

**Smart Sampling Strategy:**
<!-- How to select interactions for human review — signal-based filters -->

---

## Checklist

- [ ] System type classified
- [ ] Critical failure modes identified (≥ 3)
- [ ] Domain context researched (Section 1b: vertical, stakes, expert criteria, failure modes)
- [ ] Regulatory/compliance context identified or explicitly noted as none
- [ ] Domain expert roles defined for evaluation involvement
- [ ] Framework selected with rationale documented
- [ ] Alternatives considered and ruled out
- [ ] Framework quick reference written (install, imports, pattern, pitfalls)
- [ ] AI systems best practices written (Section 4b: Pydantic, async, prompt discipline, context)
- [ ] Evaluation dimensions grounded in domain rubric ingredients
- [ ] Each eval dimension has a concrete rubric (Good/Bad in domain language)
- [ ] Eval tooling selected — Arize Phoenix default confirmed or override noted
- [ ] Reference dataset spec written (size ≥ 10, composition + labeling defined)
- [ ] CI/CD eval integration specified
- [ ] Online guardrails defined
- [ ] Production monitoring configured (tracing tool + sampling strategy)