feat(115): secret-scan exclusion governance + --strict reduced-scan mode (#134)

* test(115): add failing tests for secret-scan exclusion lint + strict mode

Adds tests/secret-scan-lint.test.cjs covering all 7 acceptance criteria
for issue #115 (secret-scan exclusion governance):

  1. Lint exits 0 on fully-annotated .secretscanignore fixture
  2. Lint exits 1 on fixture missing required key (reason/owner/expires)
  3. Lint exits 1 on fixture with expires date in the past
  4. Lint exits 1 on wildcard pattern without rule-id
  5. Lint exits 0 on grandfathered entry (default mode), exits 1 under --strict
  6. secret-scan --strict does not honour grandfathered exclusions
     (temp workspace fixture: file with real AWS-key pattern excluded by a
     grandfathered entry → default exits 0, strict exits 1)
  7. secret-scan default mode behaviour unchanged for existing .secretscanignore
     entries (regression test)

All 24 tests confirmed RED on origin/main before any implementation.
Test helpers use spawnSync throughout so both stdout and stderr are always
captured regardless of exit code (fixes the execFileSync/stderr gap from
the existing security-scan.test.cjs pattern).

Design references cited in test file:
  - GitGuardian exclusion annotation convention:
    https://docs.gitguardian.com/internal-repositories-monitoring/integrations/cli/secrets
  - CNCF Security TAG threat-model exception lifecycle:
    https://github.com/cncf/tag-security/blob/main/community/working-groups/threat-modeling/templates/threats.md

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(115): add secret-scan-lint.sh + --strict mode + annotation parser

Implements secret-scan exclusion governance for issue #115.

## secret-scan-lint.sh (new script)

Exit codes (match secret-scan.sh convention):
  0 = all exclusions valid (or grandfathered with warning)
  1 = annotation violation: missing key, expired date, wildcard without rule-id,
      or (under --strict) any grandfathered entry
  2 = config error (file not found, bad args)

Annotation format (sidecar comment, immediately preceding the path):
  # allow: <pattern>  reason="..."  owner="..."  expires="YYYY-MM-DD"  [rule-id="..."]
  <pattern>

Required keys: reason, owner, expires
Optional key:  rule-id — required when pattern contains * wildcards

Grandfathered entries (plain comment, no structured keys):
  - Default mode: exit 0 + deprecation warning to stderr
  - --strict mode: exit 1

## secret-scan.sh (modified: --strict flag)

--strict flag for release/security-review CI lanes:
  - Grandfathered entries are NOT applied (file is scanned, not skipped)
  - Exclusions whose expires date is past are NOT applied
  - Default mode behaviour is fully preserved

load_ignorelist() now parses annotations:
  - Reads prev_comment to determine annotation status per entry
  - Uses date comparison (YYYY-MM-DD lexicographic) for expires checks
  - Emits DEPRECATION WARNING to stderr for grandfathered entries in default mode
  - Emits WARNING under --strict when skipping a grandfathered entry

Design references:
  - GitGuardian exclusion annotation convention:
    https://docs.gitguardian.com/internal-repositories-monitoring/integrations/cli/secrets
  - CNCF Security TAG threat-model exception lifecycle:
    https://github.com/cncf/tag-security/blob/main/community/working-groups/threat-modeling/templates/threats.md
  - TruffleHog / GitLeaks wildcard-exclusion risk informed the rule-id requirement
    for wildcard entries (unguarded wildcards can accidentally suppress real findings)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore(115): annotate existing .secretscanignore entries + wire CI lint step

## .secretscanignore migration

Existing entry `get-shit-done/workflows/plan-phase.md` has been migrated
from a bare plain comment to a fully-structured annotation:

  # allow: get-shit-done/workflows/plan-phase.md
  #   reason="contains illustrative DATABASE_URL/REDIS_URL example strings
  #           used as documentation placeholders — not real credentials"
  #   owner="@open-gsd/maintainers"
  #   expires="2027-06-30"

This entry now passes lint (exit 0) in both default and --strict modes.
The expiration date of 2027-06-30 gives the team ~13 months to review
whether the file still needs to be excluded before the entry expires.

## CI workflow change (.github/workflows/security-scan.yml)

Added step "Secret scan exclusion lint" immediately before the existing
"Planning directory check" step:

  - name: Secret scan exclusion lint
    run: |
      chmod +x scripts/secret-scan-lint.sh
      scripts/secret-scan-lint.sh --file .secretscanignore

The step has no ${{ }} context interpolation in its run block (no
injection surface). It runs on every PR targeting main, release/**, hotfix/**.

This implements CI acceptance criterion from issue #115:
"CI lint fails for unmanaged wildcard exclusions"
"CI enforces policy format"

## Header added to .secretscanignore

Added governance documentation block explaining annotation format,
required/optional keys, and references to design sources:
  - GitGuardian exclusion annotation convention:
    https://docs.gitguardian.com/internal-repositories-monitoring/integrations/cli/secrets
  - CNCF Security TAG threat-model exception lifecycle:
    https://github.com/cncf/tag-security/blob/main/community/working-groups/threat-modeling/templates/threats.md

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs(115): document exclusion governance + periodic reduced-scan procedure

Updates SECURITY.md with a new section "Secret-Scan Exclusion Governance"
covering:

  1. Annotation format (required/optional keys, wildcard rule)
  2. Local lint command
  3. Periodic reduced-exclusion scan procedure using --strict mode

The procedure section explicitly states when to run (every release +
scheduled security review), what --strict does differently, and what to do
when --strict finds findings that default mode does not.

No runbooks/security-audit*.md exists in this repo. SECURITY.md is the
correct location as it is what secret-scan.sh references in its header
docstring (via the "See SECURITY.md" note pattern common in this codebase).

References cited:
  - GitGuardian exclusion annotation convention:
    https://docs.gitguardian.com/internal-repositories-monitoring/integrations/cli/secrets
  - CNCF Security TAG threat-model exception lifecycle:
    https://github.com/cncf/tag-security/blob/main/community/working-groups/threat-modeling/templates/threats.md

Closes #115 (together with feat and chore commits on this branch)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(#115): exclude scanner's own test fixtures from diff-mode scan

Add */secret-scan-lint.test.cjs to should_skip_file(), consistent with
the existing exclusions for security-scan.test.cjs and
security-prompt-injection.test.cjs. The test fixture at line 465
contains a DATABASE_URL credential-shaped string that exercises the
Env Variable Leak detector — scanning it as live code is a false positive.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
2026-05-23 10:58:14 -04:00
committed by GitHub
parent 38d44a6ff8
commit 75287effb9
6 changed files with 906 additions and 14 deletions

View File

@@ -2,16 +2,46 @@
# secret-scan.sh — Check files for accidentally committed secrets/credentials
#
# Usage:
# scripts/secret-scan.sh --diff origin/main # CI mode: scan changed files
# scripts/secret-scan.sh --file path/to/file # Scan a single file
# scripts/secret-scan.sh --dir agents/ # Scan all files in a directory
# scripts/secret-scan.sh --diff origin/main # CI mode: scan changed files
# scripts/secret-scan.sh --file path/to/file # Scan a single file
# scripts/secret-scan.sh --dir agents/ # Scan all files in a directory
# scripts/secret-scan.sh --diff origin/main --strict # Strict/release mode
#
# Flags:
# --strict Reduced-exclusion mode for release and security-audit CI lanes.
# Under --strict:
# - Grandfathered (un-annotated) .secretscanignore entries are
# treated as FAILURES rather than silently honoured.
# - Exclusions whose 'expires' date is in the past are ignored
# (the file IS scanned, not skipped).
# This flag does not change secret-detection logic — only which
# exclusions are applied.
#
# Exit codes:
# 0 = clean
# 1 = findings detected
# 2 = usage error
#
# Annotation format for .secretscanignore (required for --strict compliance):
# # allow: <pattern> reason="..." owner="..." expires="YYYY-MM-DD" [rule-id="..."]
# <pattern>
#
# Design references:
# - GitGuardian exclusion annotation convention:
# https://docs.gitguardian.com/internal-repositories-monitoring/integrations/cli/secrets
# - CNCF Security TAG threat-model exception lifecycle:
# https://github.com/cncf/tag-security/blob/main/community/working-groups/threat-modeling/templates/threats.md
#
# Periodic reduced-exclusion scan procedure:
# Run this script with --strict on every release branch and during scheduled
# security reviews. This mode intentionally skips grandfathered entries and
# expired exclusions so that accumulated technical debt in the ignore-list
# cannot permanently hide secrets. See SECURITY.md for the audit runbook.
set -euo pipefail
# ─── Global mode flag ─────────────────────────────────────────────────────────
STRICT_MODE=false
# ─── Secret Patterns ─────────────────────────────────────────────────────────
# Format: "LABEL:::REGEX"
# Each entry is a human label paired with a POSIX extended regex.
@@ -57,18 +87,100 @@ SECRET_PATTERNS=(
)
# ─── Ignorelist ──────────────────────────────────────────────────────────────
#
# Entries in IGNORED_FILES are loaded from .secretscanignore.
# In --strict mode, only fully-annotated entries with a future 'expires' date
# are loaded. Grandfathered entries and expired entries are skipped (the
# corresponding files ARE scanned, not excluded).
#
# Annotation format (structured comment must immediately precede the path):
# # allow: <pattern> reason="..." owner="..." expires="YYYY-MM-DD" [rule-id="..."]
# <pattern>
#
# Entries without a structured annotation are grandfathered:
# - Default mode: accepted (file excluded), deprecation warning emitted
# - Strict mode: rejected (file scanned, no exclusion applied)
IGNOREFILE=".secretscanignore"
IGNORED_FILES=()
# Returns value of key="value" annotation pair from a string
_extract_annotation_key() {
local str="$1"
local key="$2"
echo "$str" | grep -oE "${key}=['\"][^'\"]+['\"]" | head -1 | sed "s/${key}=['\"]//;s/['\"]$//" || true
}
# Returns today as YYYY-MM-DD
_today() {
date +%Y-%m-%d
}
# Returns 0 (true) if a date string YYYY-MM-DD is strictly in the past
_date_is_past() {
local d="$1"
[[ "$d" < "$(_today)" ]]
}
load_ignorelist() {
if [[ -f "$IGNOREFILE" ]]; then
while IFS= read -r line; do
[[ "$line" =~ ^[[:space:]]*# ]] && continue
[[ -z "${line// }" ]] && continue
IGNORED_FILES+=("$line")
done < "$IGNOREFILE"
if [[ ! -f "$IGNOREFILE" ]]; then
return
fi
local prev_comment=""
while IFS= read -r line || [[ -n "$line" ]]; do
# Empty line resets context
if [[ -z "${line// }" ]]; then
prev_comment=""
continue
fi
# Accumulate comment
if [[ "$line" =~ ^[[:space:]]*# ]]; then
prev_comment="$line"
continue
fi
# This is a path entry
local pattern="$line"
# Determine if preceding comment is a structured annotation
local is_structured=false
if [[ "$prev_comment" =~ ^#[[:space:]]+allow:[[:space:]] ]]; then
is_structured=true
fi
if [[ "$is_structured" == true ]]; then
# Parse structured annotation
local expires
expires=$(_extract_annotation_key "$prev_comment" "expires")
if [[ -n "$expires" ]] && _date_is_past "$expires"; then
# Expired exclusion — never apply, regardless of mode
echo "secret-scan: WARNING: exclusion '$pattern' has expired (expires=$expires) — entry ignored" >&2
prev_comment=""
continue
fi
# Valid structured annotation — always apply
IGNORED_FILES+=("$pattern")
else
# Grandfathered (plain comment or no comment)
if [[ "$STRICT_MODE" == true ]]; then
# Strict mode: do NOT apply grandfathered exclusion
echo "secret-scan: WARNING (--strict): grandfathered exclusion '$pattern' not applied" >&2
else
# Default mode: apply but warn
echo "secret-scan: DEPRECATION WARNING: '$pattern' has no structured annotation — grandfather applied" >&2
echo " Migrate to: # allow: $pattern reason=\"...\" owner=\"...\" expires=\"YYYY-MM-DD\"" >&2
IGNORED_FILES+=("$pattern")
fi
fi
prev_comment=""
done < "$IGNOREFILE"
}
is_ignored() {
@@ -106,6 +218,7 @@ should_skip_file() {
# Skip the scan scripts themselves and test files
case "$file" in
*/secret-scan.sh) return 0 ;;
*/secret-scan-lint.test.cjs) return 0 ;;
*/security-scan.test.cjs) return 0 ;;
*/security-prompt-injection.test.cjs) return 0 ;;
tests/fixtures/adversarial/security/*|*/tests/fixtures/adversarial/security/*) return 0 ;;
@@ -186,7 +299,23 @@ scan_file() {
main() {
if [[ $# -eq 0 ]]; then
echo "Usage: $0 --diff [base] | --file <path> | --dir <path>" >&2
echo "Usage: $0 --diff [base] | --file <path> | --dir <path> [--strict]" >&2
exit 2
fi
# Parse --strict flag first (may appear anywhere in argv)
local remaining_args=()
for arg in "$@"; do
if [[ "$arg" == "--strict" ]]; then
STRICT_MODE=true
else
remaining_args+=("$arg")
fi
done
set -- "${remaining_args[@]}"
if [[ $# -eq 0 ]]; then
echo "Usage: $0 --diff [base] | --file <path> | --dir <path> [--strict]" >&2
exit 2
fi