msd-core/gsd-core/references/ai-evals.md

AI Evaluation Reference

Reference used by gsd-eval-planner and gsd-eval-auditor. Based on "AI Evals for Everyone" course (Reganti & Badam) + industry practice.

---

Core Concepts

Why Evals Exist

AI systems are non-deterministic. Input X does not reliably produce output Y across runs, users, or edge cases. Evals are the continuous process of assessing whether your system's behavior meets expectations under real-world conditions — unit tests and integration tests alone are insufficient.

Model vs. Product Evaluation

• Model evals (MMLU, HumanEval, GSM8K) — measure general capability in standardized conditions. Use as initial filter only.
• Product evals — measure behavior inside your specific system, with your data, your users, your domain rules. This is where 80% of eval effort belongs.

The Three Components of Every Eval

• Input — everything affecting the system: query, history, retrieved docs, system prompt, config
• Expected — what good behavior looks like, defined through rubrics
• Actual — what the system produced, including intermediate steps, tool calls, and reasoning traces

Three Measurement Approaches

1. Code-based metrics — deterministic checks: JSON validation, required disclaimers, performance thresholds, classification flags. Fast, cheap, reliable. Use first.
2. LLM judges — one model evaluates another against a rubric. Powerful for subjective qualities (tone, reasoning, escalation). Requires calibration against human judgment before trusting.
3. Human evaluation — gold standard for nuanced judgment. Doesn't scale. Use for calibration, edge cases, periodic sampling, and high-stakes decisions.

Most effective systems combine all three.

---

Evaluation Dimensions

Pre-Deployment (Development Phase)

Dimension	What It Measures	When It Matters
Factual accuracy	Correctness of claims against ground truth	RAG, knowledge bases, any factual assertions
Context faithfulness	Response grounded in provided context vs. fabricated	RAG pipelines, document Q&A, retrieval-augmented systems
Hallucination detection	Plausible but unsupported claims	All generative systems, high-stakes domains
Escalation accuracy	Correct identification of when human intervention needed	Customer service, healthcare, financial advisory
Policy compliance	Adherence to business rules, legal requirements, disclaimers	Regulated industries, enterprise deployments
Tone/style appropriateness	Match with brand voice, audience expectations, emotional context	Customer-facing systems, content generation
Output structure validity	Schema compliance, required fields, format correctness	Structured extraction, API integrations, data pipelines
Task completion	Whether the system accomplished the stated goal	Agentic workflows, multi-step tasks
Tool use correctness	Correct selection and invocation of tools	Agent systems with tool calls
Safety	Absence of harmful, biased, or inappropriate outputs	All user-facing systems

Production Monitoring

Dimension	Monitoring Approach
Safety violations	Online guardrail — real-time, immediate intervention
Compliance failures	Online guardrail — block or escalate before user sees output
Quality degradation trends	Offline flywheel — batch analysis of sampled interactions
Emerging failure modes	Signal-metric divergence — when user behavior signals diverge from metric scores, investigate manually
Cost/latency drift	Code-based metrics — automated threshold alerts

---

The Guardrail vs. Flywheel Decision

Ask: "If this behavior goes wrong, would it be catastrophic for my business?"

• Yes → Guardrail — run online, real-time, with immediate intervention (block, escalate, hand off). Be selective: guardrails add latency.
• No → Flywheel — run offline as batch analysis feeding system refinements over time.

---

Rubric Design

Generic metrics are meaningless without context. "Helpfulness" in real estate means summarizing listings clearly. In healthcare it means knowing when not to answer.

A rubric must define:

1. The dimension being measured
2. What scores 1, 3, and 5 on a 5-point scale (or pass/fail criteria)
3. Domain-specific examples of acceptable vs. unacceptable behavior

Without rubrics, LLM judges produce noise rather than signal.

---

Reference Dataset Guidelines

• Start with 10-20 high-quality examples — not 200 mediocre ones
• Cover: critical success scenarios, common user workflows, known edge cases, historical failure modes
• Have domain experts label the examples (not just engineers)
• Expand based on what you learn in production — don't build for hypothetical coverage

---

Eval Tooling Guide

Tool	Type	Best For	Key Strength
RAGAS	Python library	RAG evaluation	Purpose-built metrics: faithfulness, answer relevance, context precision/recall
Langfuse	Platform (open-source, self-hostable)	All system types	Strong tracing, prompt management, good for teams wanting infrastructure control
LangSmith	Platform (commercial)	LangChain/LangGraph ecosystems	Tightest integration with LangChain; best if already in that ecosystem
Arize Phoenix	Platform (open-source + hosted)	RAG + multi-agent tracing	Strong RAG eval + trace visualization; open-source with hosted option
Braintrust	Platform (commercial)	Model-agnostic evaluation	Dataset and experiment management; good for comparing across frameworks
Promptfoo	CLI tool (open-source)	Prompt testing, CI/CD	CLI-first, excellent for CI/CD prompt regression testing

Tool Selection by System Type

System Type	Recommended Tooling
RAG / Knowledge Q&A	RAGAS + Arize Phoenix or Braintrust
Multi-agent systems	Langfuse + Arize Phoenix
Conversational / single-model	Promptfoo + Braintrust
Structured extraction	Promptfoo + code-based validators
LangChain/LangGraph projects	LangSmith (native integration)
Production monitoring (all types)	Langfuse, Arize Phoenix, or LangSmith

---

Evals in the Development Lifecycle

Plan Phase (Evaluation-Aware Design)

Before writing code, define:

1. What type of AI system is being built → determines framework and dominant eval concerns
2. Critical failure modes (3-5 behaviors that cannot go wrong)
3. Rubrics — explicit definitions of acceptable/unacceptable behavior per dimension
4. Evaluation strategy — which dimensions use code metrics, LLM judges, or human review
5. Reference dataset requirements — size, composition, labeling approach
6. Eval tooling selection

Output: EVALS-SPEC section of AI-SPEC.md

Execute Phase (Instrument While Building)

• Add tracing from day one (Langfuse, Arize Phoenix, or LangSmith)
• Build reference dataset concurrently with implementation
• Implement code-based checks first; add LLM judges only for subjective dimensions
• Run evals in CI/CD via Promptfoo or Braintrust

Verify Phase (Pre-Deployment Validation)

• Run full reference dataset against all metrics
• Conduct human review of edge cases and LLM judge disagreements
• Calibrate LLM judges against human scores (target ≥ 0.7 correlation before trusting)
• Define and configure production guardrails
• Establish monitoring baseline

Monitor Phase (Production Evaluation Loop)

• Smart sampling — weight toward interactions with concerning signals (retries, unusual length, explicit escalations)
• Online guardrails on every interaction
• Offline flywheel on sampled batch
• Watch for signal-metric divergence — the early warning system for evaluation gaps

---

Common Pitfalls

1. Assuming benchmarks predict product success — they don't; model evals are a filter, not a verdict
2. Engineering evals in isolation — domain experts must co-define rubrics; engineers alone miss critical nuances
3. Building comprehensive coverage on day one — start small (10-20 examples), expand from real failure modes
4. Trusting uncalibrated LLM judges — validate against human judgment before relying on them
5. Measuring everything — only track metrics that drive decisions; "collect it all" produces noise
6. Treating evaluation as one-time setup — user behavior evolves, requirements change, failure modes emerge; evaluation is continuous