feat(115): secret-scan exclusion governance + --strict reduced-scan mode (#134)

* test(115): add failing tests for secret-scan exclusion lint + strict mode

Adds tests/secret-scan-lint.test.cjs covering all 7 acceptance criteria
for issue #115 (secret-scan exclusion governance):

  1. Lint exits 0 on fully-annotated .secretscanignore fixture
  2. Lint exits 1 on fixture missing required key (reason/owner/expires)
  3. Lint exits 1 on fixture with expires date in the past
  4. Lint exits 1 on wildcard pattern without rule-id
  5. Lint exits 0 on grandfathered entry (default mode), exits 1 under --strict
  6. secret-scan --strict does not honour grandfathered exclusions
     (temp workspace fixture: file with real AWS-key pattern excluded by a
     grandfathered entry → default exits 0, strict exits 1)
  7. secret-scan default mode behaviour unchanged for existing .secretscanignore
     entries (regression test)

All 24 tests confirmed RED on origin/main before any implementation.
Test helpers use spawnSync throughout so both stdout and stderr are always
captured regardless of exit code (fixes the execFileSync/stderr gap from
the existing security-scan.test.cjs pattern).

Design references cited in test file:
  - GitGuardian exclusion annotation convention:
    https://docs.gitguardian.com/internal-repositories-monitoring/integrations/cli/secrets
  - CNCF Security TAG threat-model exception lifecycle:
    https://github.com/cncf/tag-security/blob/main/community/working-groups/threat-modeling/templates/threats.md

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(115): add secret-scan-lint.sh + --strict mode + annotation parser

Implements secret-scan exclusion governance for issue #115.

## secret-scan-lint.sh (new script)

Exit codes (match secret-scan.sh convention):
  0 = all exclusions valid (or grandfathered with warning)
  1 = annotation violation: missing key, expired date, wildcard without rule-id,
      or (under --strict) any grandfathered entry
  2 = config error (file not found, bad args)

Annotation format (sidecar comment, immediately preceding the path):
  # allow: <pattern>  reason="..."  owner="..."  expires="YYYY-MM-DD"  [rule-id="..."]
  <pattern>

Required keys: reason, owner, expires
Optional key:  rule-id — required when pattern contains * wildcards

Grandfathered entries (plain comment, no structured keys):
  - Default mode: exit 0 + deprecation warning to stderr
  - --strict mode: exit 1

## secret-scan.sh (modified: --strict flag)

--strict flag for release/security-review CI lanes:
  - Grandfathered entries are NOT applied (file is scanned, not skipped)
  - Exclusions whose expires date is past are NOT applied
  - Default mode behaviour is fully preserved

load_ignorelist() now parses annotations:
  - Reads prev_comment to determine annotation status per entry
  - Uses date comparison (YYYY-MM-DD lexicographic) for expires checks
  - Emits DEPRECATION WARNING to stderr for grandfathered entries in default mode
  - Emits WARNING under --strict when skipping a grandfathered entry

Design references:
  - GitGuardian exclusion annotation convention:
    https://docs.gitguardian.com/internal-repositories-monitoring/integrations/cli/secrets
  - CNCF Security TAG threat-model exception lifecycle:
    https://github.com/cncf/tag-security/blob/main/community/working-groups/threat-modeling/templates/threats.md
  - TruffleHog / GitLeaks wildcard-exclusion risk informed the rule-id requirement
    for wildcard entries (unguarded wildcards can accidentally suppress real findings)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore(115): annotate existing .secretscanignore entries + wire CI lint step

## .secretscanignore migration

Existing entry `get-shit-done/workflows/plan-phase.md` has been migrated
from a bare plain comment to a fully-structured annotation:

  # allow: get-shit-done/workflows/plan-phase.md
  #   reason="contains illustrative DATABASE_URL/REDIS_URL example strings
  #           used as documentation placeholders — not real credentials"
  #   owner="@open-gsd/maintainers"
  #   expires="2027-06-30"

This entry now passes lint (exit 0) in both default and --strict modes.
The expiration date of 2027-06-30 gives the team ~13 months to review
whether the file still needs to be excluded before the entry expires.

## CI workflow change (.github/workflows/security-scan.yml)

Added step "Secret scan exclusion lint" immediately before the existing
"Planning directory check" step:

  - name: Secret scan exclusion lint
    run: |
      chmod +x scripts/secret-scan-lint.sh
      scripts/secret-scan-lint.sh --file .secretscanignore

The step has no ${{ }} context interpolation in its run block (no
injection surface). It runs on every PR targeting main, release/**, hotfix/**.

This implements CI acceptance criterion from issue #115:
"CI lint fails for unmanaged wildcard exclusions"
"CI enforces policy format"

## Header added to .secretscanignore

Added governance documentation block explaining annotation format,
required/optional keys, and references to design sources:
  - GitGuardian exclusion annotation convention:
    https://docs.gitguardian.com/internal-repositories-monitoring/integrations/cli/secrets
  - CNCF Security TAG threat-model exception lifecycle:
    https://github.com/cncf/tag-security/blob/main/community/working-groups/threat-modeling/templates/threats.md

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs(115): document exclusion governance + periodic reduced-scan procedure

Updates SECURITY.md with a new section "Secret-Scan Exclusion Governance"
covering:

  1. Annotation format (required/optional keys, wildcard rule)
  2. Local lint command
  3. Periodic reduced-exclusion scan procedure using --strict mode

The procedure section explicitly states when to run (every release +
scheduled security review), what --strict does differently, and what to do
when --strict finds findings that default mode does not.

No runbooks/security-audit*.md exists in this repo. SECURITY.md is the
correct location as it is what secret-scan.sh references in its header
docstring (via the "See SECURITY.md" note pattern common in this codebase).

References cited:
  - GitGuardian exclusion annotation convention:
    https://docs.gitguardian.com/internal-repositories-monitoring/integrations/cli/secrets
  - CNCF Security TAG threat-model exception lifecycle:
    https://github.com/cncf/tag-security/blob/main/community/working-groups/threat-modeling/templates/threats.md

Closes #115 (together with feat and chore commits on this branch)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(#115): exclude scanner's own test fixtures from diff-mode scan

Add */secret-scan-lint.test.cjs to should_skip_file(), consistent with
the existing exclusions for security-scan.test.cjs and
security-prompt-injection.test.cjs. The test fixture at line 465
contains a DATABASE_URL credential-shaped string that exercises the
Env Variable Leak detector — scanning it as live code is a false positive.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
2026-05-23 10:58:14 -04:00
committed by GitHub
parent 38d44a6ff8
commit 75287effb9
6 changed files with 906 additions and 14 deletions

231
scripts/secret-scan-lint.sh Executable file
View File

@@ -0,0 +1,231 @@
#!/usr/bin/env bash
# secret-scan-lint.sh — Lint governance policy for .secretscanignore exclusions
#
# Usage:
# scripts/secret-scan-lint.sh --file <path-to-.secretscanignore>
# scripts/secret-scan-lint.sh --file <path-to-.secretscanignore> --strict
#
# Exit codes:
# 0 = every exclusion has full annotation OR is grandfathered (with deprecation warning)
# 1 = annotation violation: missing required key, expired date, or unguarded wildcard
# without rule-id; OR (under --strict) any grandfathered entry is present
# 2 = usage/config error (file not found, invalid arguments)
#
# Annotation syntax (sidecar comment, must immediately precede the path line):
# # allow: <pattern> reason="..." owner="..." expires="YYYY-MM-DD" [rule-id="..."]
# <pattern>
#
# Required annotation keys: reason, owner, expires
# Optional annotation key: rule-id (REQUIRED when pattern contains '*' wildcards)
#
# Grandfathered entries: paths with any preceding plain comment (not a structured
# annotation) are treated as grandfathered in default mode — exit 0 with a
# deprecation warning to stderr. Under --strict, grandfathered entries cause exit 1.
#
# Design references:
# - GitGuardian exclusion annotation convention:
# https://docs.gitguardian.com/internal-repositories-monitoring/integrations/cli/secrets
# - CNCF Security TAG threat-model exception lifecycle:
# https://github.com/cncf/tag-security/blob/main/community/working-groups/threat-modeling/templates/threats.md
#
# Exit-code alignment with secret-scan.sh:
# Both scripts use 0=clean, 1=policy-violation/findings, 2=usage/config-error.
# The symmetry is intentional — CI can treat either non-zero as a gate failure.
set -euo pipefail
# ─── Argument Parsing ─────────────────────────────────────────────────────────
STRICT=false
IGNOREFILE=""
usage() {
echo "Usage: $0 --file <path-to-.secretscanignore> [--strict]" >&2
exit 2
}
while [[ $# -gt 0 ]]; do
case "$1" in
--file)
shift
[[ $# -eq 0 ]] && usage
IGNOREFILE="$1"
shift
;;
--strict)
STRICT=true
shift
;;
-h|--help)
usage
;;
*)
echo "Error: unknown argument: $1" >&2
usage
;;
esac
done
if [[ -z "$IGNOREFILE" ]]; then
usage
fi
if [[ ! -f "$IGNOREFILE" ]]; then
echo "Error: file not found: $IGNOREFILE" >&2
exit 2
fi
# ─── Date Helpers ─────────────────────────────────────────────────────────────
# Returns today's date as YYYY-MM-DD (portable across macOS and Linux)
today_date() {
date +%Y-%m-%d
}
# Returns true (0) if date1 < date2 (both YYYY-MM-DD strings)
date_is_past() {
local check_date="$1"
local today
today=$(today_date)
# Lexicographic comparison works for ISO-8601 dates
[[ "$check_date" < "$today" ]]
}
# ─── Annotation Parser ────────────────────────────────────────────────────────
# Returns value of a key="value" or key='value' pair from a string.
# Usage: extract_key <string> <key>
extract_key() {
local str="$1"
local key="$2"
# Match key="value" or key='value'
local val
val=$(echo "$str" | grep -oE "${key}=['\"][^'\"]+['\"]" | head -1 | sed "s/${key}=['\"]//;s/['\"]$//") || true
echo "$val"
}
# Returns true (0) if the string contains a wildcard glob character (* or **)
contains_wildcard() {
local pattern="$1"
[[ "$pattern" == *"*"* ]]
}
# Returns true (0) if a comment line is a structured annotation
# (must start with "# allow:" prefix)
is_structured_annotation() {
local comment="$1"
[[ "$comment" =~ ^#[[:space:]]+allow:[[:space:]] ]]
}
# ─── Main Lint Logic ──────────────────────────────────────────────────────────
VIOLATIONS=0
WARNINGS=0
# We process the file line-by-line, tracking the comment immediately preceding
# each path entry. If the preceding line was a comment, we inspect it.
prev_comment=""
lineno=0
while IFS= read -r line || [[ -n "$line" ]]; do
lineno=$((lineno + 1))
# Skip empty lines (reset prev_comment to avoid false association)
if [[ -z "${line// }" ]]; then
prev_comment=""
continue
fi
# Accumulate comment lines
if [[ "$line" =~ ^[[:space:]]*# ]]; then
prev_comment="$line"
continue
fi
# This is a path/pattern entry.
local_path="$line"
# ── Case 1: No preceding comment at all ────────────────────────────────────
if [[ -z "$prev_comment" ]]; then
echo "VIOLATION (line $lineno): '$local_path' has no annotation comment." >&2
echo " Required: # allow: <pattern> reason=\"...\" owner=\"...\" expires=\"YYYY-MM-DD\"" >&2
VIOLATIONS=$((VIOLATIONS + 1))
prev_comment=""
continue
fi
# ── Case 2: Preceding comment exists — check if it's a structured annotation ─
if is_structured_annotation "$prev_comment"; then
# Extract required keys
reason=$(extract_key "$prev_comment" "reason")
owner=$(extract_key "$prev_comment" "owner")
expires=$(extract_key "$prev_comment" "expires")
rule_id=$(extract_key "$prev_comment" "rule-id")
local_ok=true
if [[ -z "$reason" ]]; then
echo "VIOLATION (line $lineno): '$local_path' annotation missing required key: reason" >&2
local_ok=false
fi
if [[ -z "$owner" ]]; then
echo "VIOLATION (line $lineno): '$local_path' annotation missing required key: owner" >&2
local_ok=false
fi
if [[ -z "$expires" ]]; then
echo "VIOLATION (line $lineno): '$local_path' annotation missing required key: expires" >&2
local_ok=false
elif date_is_past "$expires"; then
echo "VIOLATION (line $lineno): '$local_path' annotation 'expires' date is in the past: $expires" >&2
echo " Review this exclusion and update or remove it." >&2
local_ok=false
fi
if contains_wildcard "$local_path" && [[ -z "$rule_id" ]]; then
echo "VIOLATION (line $lineno): '$local_path' uses a wildcard but is missing required key: rule-id" >&2
echo " Wildcard exclusions (** *.ext) require an explicit rule-id for auditability." >&2
local_ok=false
fi
if [[ "$local_ok" == false ]]; then
VIOLATIONS=$((VIOLATIONS + 1))
fi
else
# ── Case 3: Preceding comment is plain (not structured) — grandfathered ──
if [[ "$STRICT" == true ]]; then
echo "VIOLATION (line $lineno): '$local_path' is grandfathered (no structured annotation) — rejected under --strict mode." >&2
echo " Add: # allow: $local_path reason=\"...\" owner=\"...\" expires=\"YYYY-MM-DD\"" >&2
VIOLATIONS=$((VIOLATIONS + 1))
else
echo "WARNING (line $lineno): '$local_path' is grandfathered (missing structured annotation)." >&2
echo " DEPRECATION: migrate to structured annotation before removing grandfather status." >&2
echo " Required: # allow: $local_path reason=\"...\" owner=\"...\" expires=\"YYYY-MM-DD\"" >&2
echo " See: https://docs.gitguardian.com/internal-repositories-monitoring/integrations/cli/secrets" >&2
WARNINGS=$((WARNINGS + 1))
fi
fi
prev_comment=""
done < "$IGNOREFILE"
# ─── Summary ──────────────────────────────────────────────────────────────────
if [[ $VIOLATIONS -gt 0 ]]; then
echo "secret-scan-lint: $VIOLATIONS violation(s) found" >&2
if [[ $STRICT == true ]]; then
echo "secret-scan-lint: --strict mode active — grandfathered entries are not permitted" >&2
fi
exit 1
fi
if [[ $WARNINGS -gt 0 ]]; then
echo "secret-scan-lint: $WARNINGS grandfathered entry/entries (deprecation warning)" >&2
fi
echo "secret-scan-lint: OK"
exit 0

View File

@@ -2,16 +2,46 @@
# secret-scan.sh — Check files for accidentally committed secrets/credentials
#
# Usage:
# scripts/secret-scan.sh --diff origin/main # CI mode: scan changed files
# scripts/secret-scan.sh --file path/to/file # Scan a single file
# scripts/secret-scan.sh --dir agents/ # Scan all files in a directory
# scripts/secret-scan.sh --diff origin/main # CI mode: scan changed files
# scripts/secret-scan.sh --file path/to/file # Scan a single file
# scripts/secret-scan.sh --dir agents/ # Scan all files in a directory
# scripts/secret-scan.sh --diff origin/main --strict # Strict/release mode
#
# Flags:
# --strict Reduced-exclusion mode for release and security-audit CI lanes.
# Under --strict:
# - Grandfathered (un-annotated) .secretscanignore entries are
# treated as FAILURES rather than silently honoured.
# - Exclusions whose 'expires' date is in the past are ignored
# (the file IS scanned, not skipped).
# This flag does not change secret-detection logic — only which
# exclusions are applied.
#
# Exit codes:
# 0 = clean
# 1 = findings detected
# 2 = usage error
#
# Annotation format for .secretscanignore (required for --strict compliance):
# # allow: <pattern> reason="..." owner="..." expires="YYYY-MM-DD" [rule-id="..."]
# <pattern>
#
# Design references:
# - GitGuardian exclusion annotation convention:
# https://docs.gitguardian.com/internal-repositories-monitoring/integrations/cli/secrets
# - CNCF Security TAG threat-model exception lifecycle:
# https://github.com/cncf/tag-security/blob/main/community/working-groups/threat-modeling/templates/threats.md
#
# Periodic reduced-exclusion scan procedure:
# Run this script with --strict on every release branch and during scheduled
# security reviews. This mode intentionally skips grandfathered entries and
# expired exclusions so that accumulated technical debt in the ignore-list
# cannot permanently hide secrets. See SECURITY.md for the audit runbook.
set -euo pipefail
# ─── Global mode flag ─────────────────────────────────────────────────────────
STRICT_MODE=false
# ─── Secret Patterns ─────────────────────────────────────────────────────────
# Format: "LABEL:::REGEX"
# Each entry is a human label paired with a POSIX extended regex.
@@ -57,18 +87,100 @@ SECRET_PATTERNS=(
)
# ─── Ignorelist ──────────────────────────────────────────────────────────────
#
# Entries in IGNORED_FILES are loaded from .secretscanignore.
# In --strict mode, only fully-annotated entries with a future 'expires' date
# are loaded. Grandfathered entries and expired entries are skipped (the
# corresponding files ARE scanned, not excluded).
#
# Annotation format (structured comment must immediately precede the path):
# # allow: <pattern> reason="..." owner="..." expires="YYYY-MM-DD" [rule-id="..."]
# <pattern>
#
# Entries without a structured annotation are grandfathered:
# - Default mode: accepted (file excluded), deprecation warning emitted
# - Strict mode: rejected (file scanned, no exclusion applied)
IGNOREFILE=".secretscanignore"
IGNORED_FILES=()
# Returns value of key="value" annotation pair from a string
_extract_annotation_key() {
local str="$1"
local key="$2"
echo "$str" | grep -oE "${key}=['\"][^'\"]+['\"]" | head -1 | sed "s/${key}=['\"]//;s/['\"]$//" || true
}
# Returns today as YYYY-MM-DD
_today() {
date +%Y-%m-%d
}
# Returns 0 (true) if a date string YYYY-MM-DD is strictly in the past
_date_is_past() {
local d="$1"
[[ "$d" < "$(_today)" ]]
}
load_ignorelist() {
if [[ -f "$IGNOREFILE" ]]; then
while IFS= read -r line; do
[[ "$line" =~ ^[[:space:]]*# ]] && continue
[[ -z "${line// }" ]] && continue
IGNORED_FILES+=("$line")
done < "$IGNOREFILE"
if [[ ! -f "$IGNOREFILE" ]]; then
return
fi
local prev_comment=""
while IFS= read -r line || [[ -n "$line" ]]; do
# Empty line resets context
if [[ -z "${line// }" ]]; then
prev_comment=""
continue
fi
# Accumulate comment
if [[ "$line" =~ ^[[:space:]]*# ]]; then
prev_comment="$line"
continue
fi
# This is a path entry
local pattern="$line"
# Determine if preceding comment is a structured annotation
local is_structured=false
if [[ "$prev_comment" =~ ^#[[:space:]]+allow:[[:space:]] ]]; then
is_structured=true
fi
if [[ "$is_structured" == true ]]; then
# Parse structured annotation
local expires
expires=$(_extract_annotation_key "$prev_comment" "expires")
if [[ -n "$expires" ]] && _date_is_past "$expires"; then
# Expired exclusion — never apply, regardless of mode
echo "secret-scan: WARNING: exclusion '$pattern' has expired (expires=$expires) — entry ignored" >&2
prev_comment=""
continue
fi
# Valid structured annotation — always apply
IGNORED_FILES+=("$pattern")
else
# Grandfathered (plain comment or no comment)
if [[ "$STRICT_MODE" == true ]]; then
# Strict mode: do NOT apply grandfathered exclusion
echo "secret-scan: WARNING (--strict): grandfathered exclusion '$pattern' not applied" >&2
else
# Default mode: apply but warn
echo "secret-scan: DEPRECATION WARNING: '$pattern' has no structured annotation — grandfather applied" >&2
echo " Migrate to: # allow: $pattern reason=\"...\" owner=\"...\" expires=\"YYYY-MM-DD\"" >&2
IGNORED_FILES+=("$pattern")
fi
fi
prev_comment=""
done < "$IGNOREFILE"
}
is_ignored() {
@@ -106,6 +218,7 @@ should_skip_file() {
# Skip the scan scripts themselves and test files
case "$file" in
*/secret-scan.sh) return 0 ;;
*/secret-scan-lint.test.cjs) return 0 ;;
*/security-scan.test.cjs) return 0 ;;
*/security-prompt-injection.test.cjs) return 0 ;;
tests/fixtures/adversarial/security/*|*/tests/fixtures/adversarial/security/*) return 0 ;;
@@ -186,7 +299,23 @@ scan_file() {
main() {
if [[ $# -eq 0 ]]; then
echo "Usage: $0 --diff [base] | --file <path> | --dir <path>" >&2
echo "Usage: $0 --diff [base] | --file <path> | --dir <path> [--strict]" >&2
exit 2
fi
# Parse --strict flag first (may appear anywhere in argv)
local remaining_args=()
for arg in "$@"; do
if [[ "$arg" == "--strict" ]]; then
STRICT_MODE=true
else
remaining_args+=("$arg")
fi
done
set -- "${remaining_args[@]}"
if [[ $# -eq 0 ]]; then
echo "Usage: $0 --diff [base] | --file <path> | --dir <path> [--strict]" >&2
exit 2
fi