diff --git a/agents/gsd-verifier.md b/agents/gsd-verifier.md index 2ad4d5376..393767cb2 100644 --- a/agents/gsd-verifier.md +++ b/agents/gsd-verifier.md @@ -22,6 +22,10 @@ If the prompt contains a `` block, you MUST use the `Read` tool t **Critical mindset:** Do NOT trust SUMMARY.md claims. SUMMARYs document what Claude SAID it did. You verify what ACTUALLY exists in the code. These often differ. + +@~/.claude/get-shit-done/references/verification-overrides.md + + Before verifying, discover project context: @@ -154,7 +158,42 @@ For each truth: 1. Identify supporting artifacts 2. Check artifact status (Step 4) 3. Check wiring status (Step 5) -4. Determine truth status +4. **Before marking FAIL:** Check for override (Step 3b) +5. Determine truth status + +## Step 3b: Check Verification Overrides + +Before marking any must-have as FAILED, check the VERIFICATION.md frontmatter for an `overrides:` entry that matches this must-have. + +**Override check procedure:** + +1. Parse `overrides:` array from VERIFICATION.md frontmatter (if present) +2. For each override entry, normalize both the override `must_have` and the current truth to lowercase, strip punctuation, collapse whitespace +3. Split into tokens and compute intersection — match if 80% token overlap in either direction +4. Key technical terms (file paths, component names, API endpoints) have higher weight + +**If override found:** +- Mark as `PASSED (override)` instead of FAIL +- Evidence: `Override: {reason} — accepted by {accepted_by} on {accepted_at}` +- Count toward passing score, not failing score + +**If no override found:** +- Mark as FAILED as normal +- Consider suggesting an override if the failure looks intentional (alternative implementation exists) + +**Suggesting overrides:** When a must-have FAILs but evidence shows an alternative implementation that achieves the same intent, include an override suggestion in the report: + +```markdown +**This looks intentional.** To accept this deviation, add to VERIFICATION.md frontmatter: + +```yaml +overrides: + - must_have: "{must-have text}" + reason: "{why this deviation is acceptable}" + accepted_by: "{name}" + accepted_at: "{ISO timestamp}" +``` +``` ## Step 4: Verify Artifacts (Three Levels) @@ -541,6 +580,12 @@ phase: XX-name verified: YYYY-MM-DDTHH:MM:SSZ status: passed | gaps_found | human_needed score: N/M must-haves verified +overrides_applied: 0 # Count of PASSED (override) items included in score +overrides: # Only if overrides exist — carried forward or newly added + - must_have: "Must-have text that was overridden" + reason: "Why deviation is acceptable" + accepted_by: "username" + accepted_at: "ISO timestamp" re_verification: # Only if previous VERIFICATION.md existed previous_status: gaps_found previous_score: 2/5 diff --git a/get-shit-done/references/verification-overrides.md b/get-shit-done/references/verification-overrides.md new file mode 100644 index 000000000..e7ffed876 --- /dev/null +++ b/get-shit-done/references/verification-overrides.md @@ -0,0 +1,227 @@ +# Verification Overrides + +Mechanism for intentionally accepting must-have failures when the deviation is known and acceptable. Prevents verification loops on items that will never pass as originally specified. + + + +## Override Format + +Overrides are declared in the VERIFICATION.md frontmatter under an `overrides:` key: + +```yaml +--- +phase: 03-authentication +verified: 2026-04-05T12:00:00Z +status: passed +score: 5/5 +overrides_applied: 2 +overrides: + - must_have: "OAuth2 PKCE flow implemented" + reason: "Using session-based auth instead — PKCE unnecessary for server-rendered app" + accepted_by: "dave" + accepted_at: "2026-04-04T15:30:00Z" + - must_have: "Rate limiting on login endpoint" + reason: "Deferred to Phase 5 (infrastructure) — tracked in ROADMAP.md" + accepted_by: "dave" + accepted_at: "2026-04-04T15:30:00Z" +--- +``` + +### Required Fields + +| Field | Type | Description | +|-------|------|-------------| +| `must_have` | string | The must-have truth, artifact description, or key link being overridden. Does not need to be an exact match — fuzzy matching applies. | +| `reason` | string | Why this deviation is acceptable. Must be specific — not just "not needed". | +| `accepted_by` | string | Who accepted the override (username or role). Required. | +| `accepted_at` | string | ISO timestamp of when the override was accepted. Required. | + + + +## When to Use + +Overrides apply when a phase intentionally deviated from the original plan during execution — for example, a requirement was descoped, an alternative approach was chosen, or a dependency changed. + +Without overrides, the verifier reports these as FAIL even though the deviation was intentional. Overrides let the developer mark specific items as `PASSED (override)` with a documented reason. + +Overrides are appropriate when: +- A requirement changed after planning but ROADMAP.md hasn't been updated yet +- An alternative implementation satisfies the intent but not the literal wording +- A must-have is deferred to a later phase with explicit tracking +- External constraints make the original must-have impossible or unnecessary + +## When NOT to Use + +Overrides are NOT appropriate when: +- The implementation is simply incomplete — fix it instead +- The must-have is unclear — clarify it instead +- The developer wants to skip verification — that undermines the process +- Multiple must-haves are failing for the same phase — if more than 2-3 items need overrides, revisit the plan instead of overriding in bulk + + + +## Matching Rules + +Override matching uses **fuzzy matching**, not exact string comparison. This accommodates minor wording differences between how must-haves are phrased in ROADMAP.md, PLAN.md frontmatter, and the override entry. + +### Matching Algorithm + +1. **Normalize both strings:** case-insensitive comparison — lowercase both strings, strip punctuation, collapse whitespace +2. **Token overlap:** split into words, compute intersection +3. **Match threshold:** 80% token overlap in EITHER direction (override tokens found in must-have, OR must-have tokens found in override) +4. **Key noun priority:** nouns and technical terms (file paths, component names, API endpoints) are weighted higher than common words + +### Examples + +| Must-Have | Override `must_have` | Match? | Reason | +|-----------|---------------------|--------|--------| +| "User can authenticate via OAuth2 PKCE" | "OAuth2 PKCE flow implemented" | Yes | Key terms `OAuth2` and `PKCE` overlap, 80% threshold met | +| "Rate limiting on /api/auth/login" | "Rate limiting on login endpoint" | Yes | `rate limiting` + `login` overlap | +| "Chat component renders messages" | "OAuth2 PKCE flow implemented" | No | No meaningful token overlap | +| "src/components/Chat.tsx provides message list" | "Chat.tsx message list rendering" | Yes | `Chat.tsx` + `message` + `list` overlap | + +### Ambiguity Resolution + +If an override matches multiple must-haves, apply it to the **most specific match** (highest token overlap percentage). If still ambiguous, apply to the first match and log a warning. + + + + + +## Verifier Behavior with Overrides + +### Check Order + +The override check happens **before marking a must-have as FAIL**. The flow is: + +1. Evaluate must-have against codebase (Steps 3-5 of verification process) +2. If evaluation result is FAIL or UNCERTAIN: + a. Check `overrides:` array in VERIFICATION.md frontmatter for a fuzzy match + b. If override found: mark as `PASSED (override)` instead of FAIL + c. If no override found: mark as FAIL as normal +3. If evaluation result is PASS: mark as VERIFIED (overrides are irrelevant) + +### Output Format + +Overridden items appear with distinct status in all verification tables: + +```markdown +| # | Truth | Status | Evidence | +|---|-------|--------|----------| +| 1 | User can authenticate | VERIFIED | OAuth session flow working | +| 2 | OAuth2 PKCE flow | PASSED (override) | Override: Using session-based auth — accepted by dave on 2026-04-04 | +| 3 | Chat renders messages | FAILED | Component returns placeholder | +``` + +The `PASSED (override)` status must be visually distinct from both `VERIFIED` and `FAILED`. In the evidence column, include the override reason and who accepted it. + +### Impact on Overall Status + +- `PASSED (override)` items count toward the passing score, not the failing score +- A phase with all items either VERIFIED or PASSED (override) can have status `passed` +- Overrides do NOT suppress `human_needed` items — those still require human testing + +### Frontmatter Score + +The score and override count in frontmatter reflect applied overrides: + +```yaml +score: 5/5 # includes 2 overrides +overrides_applied: 2 +``` + + + + + +## Creating Overrides + +### Interactive Override Suggestion + +When the verifier marks a must-have as FAIL and the failure looks intentional (e.g., alternative implementation exists, or the code explicitly handles the case differently), the verifier should suggest creating an override: + +```markdown +### F-002: OAuth2 PKCE flow + +**Status:** FAILED +**Evidence:** No PKCE implementation found. Session-based auth used instead. + +**This looks intentional.** The codebase uses session-based authentication which achieves the same goal differently. To accept this deviation, add an override to VERIFICATION.md frontmatter: + +```yaml +overrides: + - must_have: "OAuth2 PKCE flow implemented" + reason: "Using session-based auth instead — PKCE unnecessary for server-rendered app" + accepted_by: "{your name}" + accepted_at: "{current ISO timestamp}" +``` + +Then re-run verification to apply. +``` + +### Override via gsd-tools + +Overrides can also be managed through the verification workflow: + +1. Run `/gsd-verify-work` — verification finds gaps +2. Review gaps — determine which are intentional deviations +3. Add override entries to VERIFICATION.md frontmatter +4. Re-run `/gsd-verify-work` — overrides are applied, remaining gaps shown + + + + + +## Override Lifecycle + +### During Re-verification + +When a phase is re-verified (e.g., after gap closure): +- Existing overrides carry forward automatically +- If the underlying code now satisfies the must-have, the override becomes unnecessary — mark as VERIFIED instead +- Overrides are never removed automatically; they persist as documentation + +### At Milestone Completion + +During `/gsd-audit-milestone`, overrides are surfaced in the audit report: + +``` +### Verification Overrides ({count} across {phase_count} phases) + +| Phase | Must-Have | Reason | Accepted By | +|-------|----------|--------|-------------| +| 03 | OAuth2 PKCE | Session-based auth used instead | dave | +``` + +This gives the team visibility into all accepted deviations before closing the milestone. + +### Cleanup + +Stale overrides (where the must-have was later implemented or removed from ROADMAP.md) can be cleaned up during milestone completion. They are informational — leaving them causes no harm. + + + +## Example VERIFICATION.md + +```markdown +--- +phase: 03-api-layer +verified: 2026-04-05T12:00:00Z +status: passed +score: 3/3 +overrides_applied: 1 +overrides: + - must_have: "paginated API responses" + reason: "Descoped — dataset under 100 items, pagination adds complexity without value" + accepted_by: "dave" + accepted_at: "2026-04-04T15:30:00Z" +--- + +## Phase 3: API Layer — Verification + +| # | Truth | Status | Evidence | +|---|-------|--------|----------| +| 1 | REST endpoints return JSON | VERIFIED | curl tests confirm | +| 2 | Paginated API responses | PASSED (override) | Descoped — see override: dataset under 100 items | +| 3 | Authentication middleware | VERIFIED | JWT validation working | +``` diff --git a/tests/verification-overrides.test.cjs b/tests/verification-overrides.test.cjs new file mode 100644 index 000000000..9a53fe6d6 --- /dev/null +++ b/tests/verification-overrides.test.cjs @@ -0,0 +1,259 @@ +/** + * Tests for verification overrides reference document (#1747) + * + * Verifies that the verification-overrides.md reference exists, documents + * the YAML frontmatter override format, and is referenced by gsd-verifier.md. + */ + +'use strict'; + +const { test, describe } = require('node:test'); +const assert = require('node:assert/strict'); +const fs = require('fs'); +const path = require('path'); + +const ROOT = path.resolve(__dirname, '..'); + +describe('verification overrides reference (#1747)', () => { + + // ── Reference document ──────────────────────────────────────────────────── + + describe('get-shit-done/references/verification-overrides.md', () => { + const refPath = path.join(ROOT, 'get-shit-done', 'references', 'verification-overrides.md'); + let content; + + test('file exists', () => { + assert.ok(fs.existsSync(refPath), 'verification-overrides.md should exist'); + content = fs.readFileSync(refPath, 'utf-8'); + }); + + test('contains Override Format section', () => { + content = content || fs.readFileSync(refPath, 'utf-8'); + assert.ok( + content.includes('## Override Format'), + 'should contain an "Override Format" section' + ); + }); + + test('contains Matching Rules section', () => { + content = content || fs.readFileSync(refPath, 'utf-8'); + assert.ok( + content.includes('## Matching Rules'), + 'should contain a "Matching Rules" section' + ); + }); + + test('contains Verifier Behavior section', () => { + content = content || fs.readFileSync(refPath, 'utf-8'); + assert.ok( + content.includes('## Verifier Behavior'), + 'should contain a "Verifier Behavior" section' + ); + }); + + test('documents YAML frontmatter overrides block with must_have field', () => { + content = content || fs.readFileSync(refPath, 'utf-8'); + assert.ok( + content.includes('overrides:') && content.includes('must_have:') && content.includes('reason:'), + 'should document the YAML frontmatter format with overrides, must_have, and reason fields' + ); + }); + + test('does not use criterion field (must use must_have)', () => { + content = content || fs.readFileSync(refPath, 'utf-8'); + // criterion: as a YAML field name should not appear; must_have: is the correct field + const lines = content.split('\n'); + const criterionLines = lines.filter((l) => /^\s+criterion:/.test(l)); + assert.strictEqual( + criterionLines.length, + 0, + 'should use must_have field, not criterion field' + ); + }); + + test('documents required accepted_by field', () => { + content = content || fs.readFileSync(refPath, 'utf-8'); + assert.ok( + content.includes('accepted_by'), + 'should document the accepted_by field' + ); + }); + + test('documents required accepted_at field', () => { + content = content || fs.readFileSync(refPath, 'utf-8'); + assert.ok( + content.includes('accepted_at'), + 'should document the accepted_at field' + ); + }); + + test('marks accepted_by as required (not optional)', () => { + content = content || fs.readFileSync(refPath, 'utf-8'); + // The field table should list accepted_by as required, not optional + assert.ok( + content.includes('accepted_by') && content.includes('Required'), + 'accepted_by should be described as a required field' + ); + }); + + test('marks accepted_at as required (not optional)', () => { + content = content || fs.readFileSync(refPath, 'utf-8'); + assert.ok( + content.includes('accepted_at') && content.includes('Required'), + 'accepted_at should be described as a required field' + ); + }); + + test('describes fuzzy matching behavior', () => { + content = content || fs.readFileSync(refPath, 'utf-8'); + assert.ok( + content.includes('fuzzy matching') || content.includes('fuzzy-match'), + 'should describe fuzzy matching for pairing overrides with must-haves' + ); + }); + + test('describes case-insensitive matching', () => { + content = content || fs.readFileSync(refPath, 'utf-8'); + assert.ok( + content.toLowerCase().includes('case-insensitive'), + 'should describe case-insensitive matching' + ); + }); + + test('describes 80% word overlap matching threshold', () => { + content = content || fs.readFileSync(refPath, 'utf-8'); + assert.ok( + content.includes('80%'), + 'should describe 80% word overlap matching threshold' + ); + }); + + test('does not use a threshold lower than 80%', () => { + content = content || fs.readFileSync(refPath, 'utf-8'); + // 60% threshold is too loose — should not appear as the matching threshold + assert.ok( + !content.includes('60% token overlap') && !content.includes('60% word overlap'), + 'should not describe a 60% matching threshold (too loose — use 80%)' + ); + }); + + test('documents PASSED (override) status', () => { + content = content || fs.readFileSync(refPath, 'utf-8'); + assert.ok( + content.includes('PASSED (override)'), + 'should document the PASSED (override) status marker' + ); + }); + + test('includes example VERIFICATION.md', () => { + content = content || fs.readFileSync(refPath, 'utf-8'); + assert.ok( + content.includes('## Example VERIFICATION.md'), + 'should include an example VERIFICATION.md section' + ); + }); + + test('documents When to Use guidance', () => { + content = content || fs.readFileSync(refPath, 'utf-8'); + assert.ok( + content.includes('## When to Use'), + 'should contain a "When to Use" section' + ); + }); + + test('documents When NOT to Use guardrails', () => { + content = content || fs.readFileSync(refPath, 'utf-8'); + assert.ok( + content.includes('When NOT to Use') || content.includes('NOT appropriate'), + 'should contain guardrails explaining when overrides are not appropriate' + ); + }); + + test('documents overrides_applied counter field', () => { + content = content || fs.readFileSync(refPath, 'utf-8'); + assert.ok( + content.includes('overrides_applied'), + 'should document the overrides_applied counter field in frontmatter' + ); + }); + + test('documents re-verification carryforward behavior', () => { + content = content || fs.readFileSync(refPath, 'utf-8'); + assert.ok( + content.includes('carry forward') || content.includes('carryforward') || content.includes('Re-verification') || content.includes('re-verification'), + 'should document that overrides carry forward during re-verification' + ); + }); + + test('documents milestone audit surfacing', () => { + content = content || fs.readFileSync(refPath, 'utf-8'); + assert.ok( + content.includes('gsd-audit-milestone') || content.includes('audit-milestone') || content.includes('milestone'), + 'should document that overrides are surfaced during milestone audit' + ); + }); + }); + + // ── Verifier agent reference ────────────────────────────────────────────── + + describe('agents/gsd-verifier.md references overrides', () => { + const verifierPath = path.join(ROOT, 'agents', 'gsd-verifier.md'); + let verifierContent; + + test('gsd-verifier.md exists', () => { + assert.ok(fs.existsSync(verifierPath), 'gsd-verifier.md should exist'); + verifierContent = fs.readFileSync(verifierPath, 'utf-8'); + }); + + test('references verification-overrides.md in required_reading', () => { + verifierContent = verifierContent || fs.readFileSync(verifierPath, 'utf-8'); + assert.ok( + verifierContent.includes('verification-overrides.md'), + 'gsd-verifier.md should reference verification-overrides.md' + ); + }); + + test('required_reading block is between and ', () => { + verifierContent = verifierContent || fs.readFileSync(verifierPath, 'utf-8'); + const roleEnd = verifierContent.indexOf(''); + const projectCtx = verifierContent.indexOf(''); + const reqReading = verifierContent.indexOf(''); + assert.ok(roleEnd > -1, ' tag should exist'); + assert.ok(projectCtx > -1, ' tag should exist'); + assert.ok(reqReading > -1, ' tag should exist'); + assert.ok( + reqReading > roleEnd && reqReading < projectCtx, + ' should appear between and ' + ); + }); + + test('verifier includes Step 3b for override check before FAIL', () => { + verifierContent = verifierContent || fs.readFileSync(verifierPath, 'utf-8'); + assert.ok( + verifierContent.includes('Step 3b'), + 'gsd-verifier.md should include a Step 3b override check' + ); + }); + + test('verifier Step 3b uses must_have field (not criterion)', () => { + verifierContent = verifierContent || fs.readFileSync(verifierPath, 'utf-8'); + // Find the Step 3b section + const step3bStart = verifierContent.indexOf('## Step 3b'); + assert.ok(step3bStart > -1, 'Step 3b section should exist'); + const step3bEnd = verifierContent.indexOf('\n## Step 4', step3bStart); + const step3bSection = verifierContent.slice(step3bStart, step3bEnd > -1 ? step3bEnd : undefined); + assert.ok( + step3bSection.includes('must_have'), + 'Step 3b should reference the must_have field' + ); + }); + + test('verifier frontmatter template includes overrides_applied', () => { + verifierContent = verifierContent || fs.readFileSync(verifierPath, 'utf-8'); + assert.ok( + verifierContent.includes('overrides_applied'), + 'gsd-verifier.md frontmatter template should include overrides_applied counter' + ); + }); + }); +});