feat(115): secret-scan exclusion governance + --strict reduced-scan mode (#134)
* test(115): add failing tests for secret-scan exclusion lint + strict mode Adds tests/secret-scan-lint.test.cjs covering all 7 acceptance criteria for issue #115 (secret-scan exclusion governance): 1. Lint exits 0 on fully-annotated .secretscanignore fixture 2. Lint exits 1 on fixture missing required key (reason/owner/expires) 3. Lint exits 1 on fixture with expires date in the past 4. Lint exits 1 on wildcard pattern without rule-id 5. Lint exits 0 on grandfathered entry (default mode), exits 1 under --strict 6. secret-scan --strict does not honour grandfathered exclusions (temp workspace fixture: file with real AWS-key pattern excluded by a grandfathered entry → default exits 0, strict exits 1) 7. secret-scan default mode behaviour unchanged for existing .secretscanignore entries (regression test) All 24 tests confirmed RED on origin/main before any implementation. Test helpers use spawnSync throughout so both stdout and stderr are always captured regardless of exit code (fixes the execFileSync/stderr gap from the existing security-scan.test.cjs pattern). Design references cited in test file: - GitGuardian exclusion annotation convention: https://docs.gitguardian.com/internal-repositories-monitoring/integrations/cli/secrets - CNCF Security TAG threat-model exception lifecycle: https://github.com/cncf/tag-security/blob/main/community/working-groups/threat-modeling/templates/threats.md Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(115): add secret-scan-lint.sh + --strict mode + annotation parser Implements secret-scan exclusion governance for issue #115. ## secret-scan-lint.sh (new script) Exit codes (match secret-scan.sh convention): 0 = all exclusions valid (or grandfathered with warning) 1 = annotation violation: missing key, expired date, wildcard without rule-id, or (under --strict) any grandfathered entry 2 = config error (file not found, bad args) Annotation format (sidecar comment, immediately preceding the path): # allow: <pattern> reason="..." owner="..." expires="YYYY-MM-DD" [rule-id="..."] <pattern> Required keys: reason, owner, expires Optional key: rule-id — required when pattern contains * wildcards Grandfathered entries (plain comment, no structured keys): - Default mode: exit 0 + deprecation warning to stderr - --strict mode: exit 1 ## secret-scan.sh (modified: --strict flag) --strict flag for release/security-review CI lanes: - Grandfathered entries are NOT applied (file is scanned, not skipped) - Exclusions whose expires date is past are NOT applied - Default mode behaviour is fully preserved load_ignorelist() now parses annotations: - Reads prev_comment to determine annotation status per entry - Uses date comparison (YYYY-MM-DD lexicographic) for expires checks - Emits DEPRECATION WARNING to stderr for grandfathered entries in default mode - Emits WARNING under --strict when skipping a grandfathered entry Design references: - GitGuardian exclusion annotation convention: https://docs.gitguardian.com/internal-repositories-monitoring/integrations/cli/secrets - CNCF Security TAG threat-model exception lifecycle: https://github.com/cncf/tag-security/blob/main/community/working-groups/threat-modeling/templates/threats.md - TruffleHog / GitLeaks wildcard-exclusion risk informed the rule-id requirement for wildcard entries (unguarded wildcards can accidentally suppress real findings) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore(115): annotate existing .secretscanignore entries + wire CI lint step ## .secretscanignore migration Existing entry `get-shit-done/workflows/plan-phase.md` has been migrated from a bare plain comment to a fully-structured annotation: # allow: get-shit-done/workflows/plan-phase.md # reason="contains illustrative DATABASE_URL/REDIS_URL example strings # used as documentation placeholders — not real credentials" # owner="@open-gsd/maintainers" # expires="2027-06-30" This entry now passes lint (exit 0) in both default and --strict modes. The expiration date of 2027-06-30 gives the team ~13 months to review whether the file still needs to be excluded before the entry expires. ## CI workflow change (.github/workflows/security-scan.yml) Added step "Secret scan exclusion lint" immediately before the existing "Planning directory check" step: - name: Secret scan exclusion lint run: | chmod +x scripts/secret-scan-lint.sh scripts/secret-scan-lint.sh --file .secretscanignore The step has no ${{ }} context interpolation in its run block (no injection surface). It runs on every PR targeting main, release/**, hotfix/**. This implements CI acceptance criterion from issue #115: "CI lint fails for unmanaged wildcard exclusions" "CI enforces policy format" ## Header added to .secretscanignore Added governance documentation block explaining annotation format, required/optional keys, and references to design sources: - GitGuardian exclusion annotation convention: https://docs.gitguardian.com/internal-repositories-monitoring/integrations/cli/secrets - CNCF Security TAG threat-model exception lifecycle: https://github.com/cncf/tag-security/blob/main/community/working-groups/threat-modeling/templates/threats.md Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(115): document exclusion governance + periodic reduced-scan procedure Updates SECURITY.md with a new section "Secret-Scan Exclusion Governance" covering: 1. Annotation format (required/optional keys, wildcard rule) 2. Local lint command 3. Periodic reduced-exclusion scan procedure using --strict mode The procedure section explicitly states when to run (every release + scheduled security review), what --strict does differently, and what to do when --strict finds findings that default mode does not. No runbooks/security-audit*.md exists in this repo. SECURITY.md is the correct location as it is what secret-scan.sh references in its header docstring (via the "See SECURITY.md" note pattern common in this codebase). References cited: - GitGuardian exclusion annotation convention: https://docs.gitguardian.com/internal-repositories-monitoring/integrations/cli/secrets - CNCF Security TAG threat-model exception lifecycle: https://github.com/cncf/tag-security/blob/main/community/working-groups/threat-modeling/templates/threats.md Closes #115 (together with feat and chore commits on this branch) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(#115): exclude scanner's own test fixtures from diff-mode scan Add */secret-scan-lint.test.cjs to should_skip_file(), consistent with the existing exclusions for security-scan.test.cjs and security-prompt-injection.test.cjs. The test fixture at line 465 contains a DATABASE_URL credential-shaped string that exercises the Env Variable Leak detector — scanning it as live code is a false positive. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -2,16 +2,46 @@
|
||||
# secret-scan.sh — Check files for accidentally committed secrets/credentials
|
||||
#
|
||||
# Usage:
|
||||
# scripts/secret-scan.sh --diff origin/main # CI mode: scan changed files
|
||||
# scripts/secret-scan.sh --file path/to/file # Scan a single file
|
||||
# scripts/secret-scan.sh --dir agents/ # Scan all files in a directory
|
||||
# scripts/secret-scan.sh --diff origin/main # CI mode: scan changed files
|
||||
# scripts/secret-scan.sh --file path/to/file # Scan a single file
|
||||
# scripts/secret-scan.sh --dir agents/ # Scan all files in a directory
|
||||
# scripts/secret-scan.sh --diff origin/main --strict # Strict/release mode
|
||||
#
|
||||
# Flags:
|
||||
# --strict Reduced-exclusion mode for release and security-audit CI lanes.
|
||||
# Under --strict:
|
||||
# - Grandfathered (un-annotated) .secretscanignore entries are
|
||||
# treated as FAILURES rather than silently honoured.
|
||||
# - Exclusions whose 'expires' date is in the past are ignored
|
||||
# (the file IS scanned, not skipped).
|
||||
# This flag does not change secret-detection logic — only which
|
||||
# exclusions are applied.
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 = clean
|
||||
# 1 = findings detected
|
||||
# 2 = usage error
|
||||
#
|
||||
# Annotation format for .secretscanignore (required for --strict compliance):
|
||||
# # allow: <pattern> reason="..." owner="..." expires="YYYY-MM-DD" [rule-id="..."]
|
||||
# <pattern>
|
||||
#
|
||||
# Design references:
|
||||
# - GitGuardian exclusion annotation convention:
|
||||
# https://docs.gitguardian.com/internal-repositories-monitoring/integrations/cli/secrets
|
||||
# - CNCF Security TAG threat-model exception lifecycle:
|
||||
# https://github.com/cncf/tag-security/blob/main/community/working-groups/threat-modeling/templates/threats.md
|
||||
#
|
||||
# Periodic reduced-exclusion scan procedure:
|
||||
# Run this script with --strict on every release branch and during scheduled
|
||||
# security reviews. This mode intentionally skips grandfathered entries and
|
||||
# expired exclusions so that accumulated technical debt in the ignore-list
|
||||
# cannot permanently hide secrets. See SECURITY.md for the audit runbook.
|
||||
set -euo pipefail
|
||||
|
||||
# ─── Global mode flag ─────────────────────────────────────────────────────────
|
||||
STRICT_MODE=false
|
||||
|
||||
# ─── Secret Patterns ─────────────────────────────────────────────────────────
|
||||
# Format: "LABEL:::REGEX"
|
||||
# Each entry is a human label paired with a POSIX extended regex.
|
||||
@@ -57,18 +87,100 @@ SECRET_PATTERNS=(
|
||||
)
|
||||
|
||||
# ─── Ignorelist ──────────────────────────────────────────────────────────────
|
||||
#
|
||||
# Entries in IGNORED_FILES are loaded from .secretscanignore.
|
||||
# In --strict mode, only fully-annotated entries with a future 'expires' date
|
||||
# are loaded. Grandfathered entries and expired entries are skipped (the
|
||||
# corresponding files ARE scanned, not excluded).
|
||||
#
|
||||
# Annotation format (structured comment must immediately precede the path):
|
||||
# # allow: <pattern> reason="..." owner="..." expires="YYYY-MM-DD" [rule-id="..."]
|
||||
# <pattern>
|
||||
#
|
||||
# Entries without a structured annotation are grandfathered:
|
||||
# - Default mode: accepted (file excluded), deprecation warning emitted
|
||||
# - Strict mode: rejected (file scanned, no exclusion applied)
|
||||
|
||||
IGNOREFILE=".secretscanignore"
|
||||
IGNORED_FILES=()
|
||||
|
||||
# Returns value of key="value" annotation pair from a string
|
||||
_extract_annotation_key() {
|
||||
local str="$1"
|
||||
local key="$2"
|
||||
echo "$str" | grep -oE "${key}=['\"][^'\"]+['\"]" | head -1 | sed "s/${key}=['\"]//;s/['\"]$//" || true
|
||||
}
|
||||
|
||||
# Returns today as YYYY-MM-DD
|
||||
_today() {
|
||||
date +%Y-%m-%d
|
||||
}
|
||||
|
||||
# Returns 0 (true) if a date string YYYY-MM-DD is strictly in the past
|
||||
_date_is_past() {
|
||||
local d="$1"
|
||||
[[ "$d" < "$(_today)" ]]
|
||||
}
|
||||
|
||||
load_ignorelist() {
|
||||
if [[ -f "$IGNOREFILE" ]]; then
|
||||
while IFS= read -r line; do
|
||||
[[ "$line" =~ ^[[:space:]]*# ]] && continue
|
||||
[[ -z "${line// }" ]] && continue
|
||||
IGNORED_FILES+=("$line")
|
||||
done < "$IGNOREFILE"
|
||||
if [[ ! -f "$IGNOREFILE" ]]; then
|
||||
return
|
||||
fi
|
||||
|
||||
local prev_comment=""
|
||||
|
||||
while IFS= read -r line || [[ -n "$line" ]]; do
|
||||
# Empty line resets context
|
||||
if [[ -z "${line// }" ]]; then
|
||||
prev_comment=""
|
||||
continue
|
||||
fi
|
||||
|
||||
# Accumulate comment
|
||||
if [[ "$line" =~ ^[[:space:]]*# ]]; then
|
||||
prev_comment="$line"
|
||||
continue
|
||||
fi
|
||||
|
||||
# This is a path entry
|
||||
local pattern="$line"
|
||||
|
||||
# Determine if preceding comment is a structured annotation
|
||||
local is_structured=false
|
||||
if [[ "$prev_comment" =~ ^#[[:space:]]+allow:[[:space:]] ]]; then
|
||||
is_structured=true
|
||||
fi
|
||||
|
||||
if [[ "$is_structured" == true ]]; then
|
||||
# Parse structured annotation
|
||||
local expires
|
||||
expires=$(_extract_annotation_key "$prev_comment" "expires")
|
||||
|
||||
if [[ -n "$expires" ]] && _date_is_past "$expires"; then
|
||||
# Expired exclusion — never apply, regardless of mode
|
||||
echo "secret-scan: WARNING: exclusion '$pattern' has expired (expires=$expires) — entry ignored" >&2
|
||||
prev_comment=""
|
||||
continue
|
||||
fi
|
||||
|
||||
# Valid structured annotation — always apply
|
||||
IGNORED_FILES+=("$pattern")
|
||||
|
||||
else
|
||||
# Grandfathered (plain comment or no comment)
|
||||
if [[ "$STRICT_MODE" == true ]]; then
|
||||
# Strict mode: do NOT apply grandfathered exclusion
|
||||
echo "secret-scan: WARNING (--strict): grandfathered exclusion '$pattern' not applied" >&2
|
||||
else
|
||||
# Default mode: apply but warn
|
||||
echo "secret-scan: DEPRECATION WARNING: '$pattern' has no structured annotation — grandfather applied" >&2
|
||||
echo " Migrate to: # allow: $pattern reason=\"...\" owner=\"...\" expires=\"YYYY-MM-DD\"" >&2
|
||||
IGNORED_FILES+=("$pattern")
|
||||
fi
|
||||
fi
|
||||
|
||||
prev_comment=""
|
||||
done < "$IGNOREFILE"
|
||||
}
|
||||
|
||||
is_ignored() {
|
||||
@@ -106,6 +218,7 @@ should_skip_file() {
|
||||
# Skip the scan scripts themselves and test files
|
||||
case "$file" in
|
||||
*/secret-scan.sh) return 0 ;;
|
||||
*/secret-scan-lint.test.cjs) return 0 ;;
|
||||
*/security-scan.test.cjs) return 0 ;;
|
||||
*/security-prompt-injection.test.cjs) return 0 ;;
|
||||
tests/fixtures/adversarial/security/*|*/tests/fixtures/adversarial/security/*) return 0 ;;
|
||||
@@ -186,7 +299,23 @@ scan_file() {
|
||||
|
||||
main() {
|
||||
if [[ $# -eq 0 ]]; then
|
||||
echo "Usage: $0 --diff [base] | --file <path> | --dir <path>" >&2
|
||||
echo "Usage: $0 --diff [base] | --file <path> | --dir <path> [--strict]" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Parse --strict flag first (may appear anywhere in argv)
|
||||
local remaining_args=()
|
||||
for arg in "$@"; do
|
||||
if [[ "$arg" == "--strict" ]]; then
|
||||
STRICT_MODE=true
|
||||
else
|
||||
remaining_args+=("$arg")
|
||||
fi
|
||||
done
|
||||
set -- "${remaining_args[@]}"
|
||||
|
||||
if [[ $# -eq 0 ]]; then
|
||||
echo "Usage: $0 --diff [base] | --file <path> | --dir <path> [--strict]" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
|
||||
Reference in New Issue
Block a user