fix(#3631): exclude only __pycache__-resident bytecode from the consent digest (#3650)

* test(3631): failing-first coverage for bytecode-cache in the consent hash

bundleContentHash digests a walk with no exclusion, so a routine 'python3 -m unittest'
inside a Python-backed capability bundle writes __pycache__ under the bundle, the
recomputed hash stops matching the consent record, and the capability silently goes
inactive — no error, no warning, and loop render-hooks then omits its step and gate.

Two distinct triggers, and the second is the sharper one: collectBundleEntries pushes a
{kind:'dir'} entry for EVERY directory and the digest emits a TAG_DIR marker for it, so an
EMPTY __pycache__/ flips the hash before a single .pyc is written. A fix filtering only
*.pyc would leave that live. Verified by execution against the built lib: 5 of 7 probe
rows diverge from intent today, including the empty-directory row.

The anti-regression rows are the point of the shape: editing a real scripts/m.py and
adding node_modules/pkg/index.js must BOTH still change the hash. node_modules is
deliberately not excludable — its contents are required at runtime, so dropping it from
the digest would stop consent binding executable content. The symlink row pins ordering:
exclusion must apply after the lstat fail-closed rejection, never before.

Refs #3631

* fix(3631): exclude derived bytecode caches from the consent digest

RED proven at e5ba8f1fe on the remote runner: 8 failures, exactly the rows predicted to
fail, with the four anti-regression rows already green.

collectBundleEntries now skips a hardcoded, gitignore-independent set from the DIGEST:
basenames __pycache__, .pytest_cache, .DS_Store, and any .pyc/.pyo file. Matching is
byte-exact on the raw Buffer name (the walk never utf8-decodes) and case-sensitive, so the
digest does not vary with how a name happens to be spelled on a case-insensitive volume.

Three properties were preserved deliberately, each pinned by a test:

  - The filter runs AFTER the lstat symlink/non-regular fail-closed rejection. Filtering
    first would have turned the exclusion into a way to smuggle a symlink past the check;
    a symlink named x.pyc still throws.
  - Excluded entries still count toward BUNDLE_MAX_FILES and BUNDLE_MAX_TOTAL_BYTES. The
    caps guard the WALK; the digest answers a different question, and exclusion must not
    become an unbounded-bytes hole.
  - An excluded DIRECTORY is neither emitted as a TAG_DIR marker nor recursed into. The
    directory marker was the sharper half of this bug: an empty __pycache__ flipped the
    hash before any .pyc existed, so a *.pyc-only filter would have left it live.

The issue proposed either a gitignore-aware walk or a list including node_modules. Both
are rejected. A consent binding must not delegate its scope to a .gitignore the bundle
author does not control — one line there would drop arbitrary executable content out of
the hash. And node_modules holds code that is required at runtime; excluding it would stop
consent binding executable content, turning a usability bug into a supply-chain hole. What
makes __pycache__ different is that CPython validates each .pyc against its sibling
source, which remains hashed, so a real code change still invalidates consent.

Docs: CONTEXT.md's 'EVERY regular file AND directory' claim is corrected in place.
ADR-2363's residual-gap section said the walk had 'no exclusions' — per
docs/adr/README.md ('ADRs are append-only') that is corrected by a dated amendment rather
than an in-place edit. Its D4 argument is unaffected: skill bodies are .md and stay bound.

Fixes #3631

* fix(3631): narrow the digest exclusion after two isolated security reviews

The first cut of this fix passed the full suite and was still wrong. Both orthogonal
reviews rejected it, and the second one found a hole that has nothing to do with Python.

HIGH — an excluded DIRECTORY was 'continue'd before recursion, so its whole subtree was
permanently outside the digest. Declared hook script paths allow '_', '.' and '/' with no
directory or extension rule, so hooks:[{script:'__pycache__/run.js'}] installed, executed
via node, and its bytes could be rewritten forever without moving the hash. Ship benign
v1, collect consent, then own the machine. No Python involved.

FALSE RATIONALE — the justification I wrote into the code, CONTEXT.md, the ADR amendment
and the changeset claimed CPython validates a cached .pyc against its sibling source, so
the source staying hashed kept consent honest. That is not true, and I proved it by
execution rather than argument: default timestamp invalidation compares only the source's
mtime and size, both settable by anyone who can write the bundle. A forged pyc ran while
the .py was byte-identical.

Also wrong: '*.pyc' matched anywhere, but a legacy sourceless scripts/x.pyc IS importable,
so excluding it was a live vector.

Narrowed to what is actually defensible:
  - a DIRECTORY named __pycache__/.pytest_cache has only its TAG_DIR marker suppressed;
    the walk still recurses and hashes every non-excluded child.
  - .pyc/.pyo are excluded ONLY when the parent basename is exactly __pycache__.
  - a regular FILE named __pycache__, and a DIRECTORY named x.pyc, stay bound.
  - declared hook paths containing a __pycache__/.pytest_cache segment or a .pyc/.pyo
    basename are now rejected in both validator copies — a file named .pyc can contain
    perfectly valid JavaScript, so the exclusion must not be reachable from a declared
    surface.

Accepted residual risk, stated plainly in ADR-2363 and CONTEXT.md instead of explained
away: a forged __pycache__/mod.pyc matching an unmodified, still-hashed mod.py executes
without moving the digest. Before this change that write was detected. It is accepted to
stop routine bytecode caching from silently deactivating capabilities, and it is bounded —
the attacker needs post-consent write access, everything outside __pycache__/*.pyc stays
hashed, and no declared surface can point into the excluded space.

Known limitation, not papered over: .pytest_cache CONTENTS still move the digest. Only the
directory marker is suppressed. Excluding that subtree would reopen the HIGH finding.

Refs #3631

* fix(3631): drop the .DS_Store exclusion and pin what the caps actually bind

Second round of isolated review findings. The hardening closed the two original holes —
both re-reviews confirmed that by execution — but it introduced a new one of the same
shape, and left three claims unbacked.

HIGH, self-inflicted: .DS_Store was excluded from the digest at any depth, but the hook
path validator was hardened only for __pycache__/.pytest_cache/.pyc/.pyo. So
script:'hooks/.DS_Store' was ACCEPTED, runnableHookCommand emits the bare quoted path for
a non-.js name (the branch .sh hooks already use), and capability-source copies it with
its mode bit intact. Ship it +x with a benign shebang, take consent, then rewrite it
forever — the digest never moves. Fixed by DELETING the .DS_Store exclusion rather than
teaching the validator about it: .DS_Store has nothing to do with this issue's Python
bytecode symptom, and an excluded filename is a permanently unhashed name. The narrower
the exclusion, the smaller the hole.

The residual-risk bound in ADR-2363 and CONTEXT.md claimed declared surfaces cannot reach
excluded space. That is false and is now stated correctly: node resolves an unregistered
extension through the default .js handler, so a hashed, consent-covered hooks/run.js that
requires '../__pycache__/mod.pyc' reaches it in one hop. The validator guard raises the
bar for DECLARED surfaces; it does not contain the risk. The two bounds that are real —
post-consent write access required, everything outside __pycache__/*.pyc still hashed —
are kept.

The BUNDLE_MAX_FILES boundary test had gone vacuous: it padded with root-level *.pyc,
which the hardening made non-excluded, so it no longer proved anything about excluded
entries while the ADR claimed the caps were test-pinned. It now pads __pycache__/f{i}.pyc,
with the arithmetic re-derived by execution (capability.json + the still-counted
__pycache__ dir + N). BUNDLE_MAX_TOTAL_BYTES had zero coverage at all and is now pinned by
a sparse 32 MiB __pycache__/big.pyc that must still trip the size cap — the test that
proves exclusion did not become an unbounded-bytes hole.

Added the parity assertion CLAUDE.md's Generative Fix Divergence rule requires for the two
isSafeHookScriptPath copies, and proved it can fail: mutating one BUILT copy to drop .pyo
made the parity check report the divergence. Also pinned semantics that were correct but
untested and would have survived mutation — __pycache__/sub/x.pyc stays hashed (the parent
resets to sub, which is the recursion threading itself), .pytest_cache/y.pyc stays hashed,
and .pyo in both directions, which was a free surviving mutant.

Changeset rewritten: it still described the rejected wholesale-exclusion semantics.

Refs #3631

* chore(3631): backfill changeset PR number (#3650)

---------

Co-authored-by: sim <sim@local>

docs/adr/2363-capability-instruction-surface-trust.md

@@ -155,3 +155,81 @@ The README's first ratification trap is *"shipped code is necessary, not suffici
 **Full consent parity with hooks** — require the same ceremony for a skill contribution as for a hook. Rejected as consent fatigue. [`capability-trust-model.md`](../explanation/capability-trust-model.md) already rejects a per-run egress prompt on exactly these grounds: inflating every contribution to hook-level ceremony trains users to click through, degrading the prompt that matters.
 
 **Docs correction with no ADR** — rejected: `CONTRIBUTING.md` requires an ADR for an architectural decision, and a docs edit with no recorded decision reproduces the unrecorded posture this ADR exists to end.
+
+## Amendment (2026-08-18): `bundleContentHash` is no longer exclusion-free
+
+The residual-gap section above states that `bundleContentHash` "walks every entry under the
+bundle with no exclusions." As of #3631 that is no longer literally true, so the sentence is
+corrected here rather than edited in place.
+
+The trigger was a usability defect, not a security one — running a Python-backed capability's
+own test suite wrote `__pycache__` inside the bundle (an *empty* `__pycache__` directory was
+enough, because the walk emits a typed DIR marker per directory), which changed the recomputed
+hash and silently deactivated the capability. The FIRST fix shipped for this (basename-excluded
+`__pycache__`/`.pytest_cache`/`.DS_Store`, plus any `.pyc`/`.pyo` file anywhere, with an excluded
+DIRECTORY skipped from recursion entirely) was itself found UNSAFE by two orthogonal reviews and
+was corrected before merge to next. That draft is not described further here; this section
+describes the shipped exclusion. `.DS_Store` was part of that first draft and was later removed
+from the exclusion set entirely (not merely narrowed) — see the amendment below.
+
+`bundleContentHash` now excludes from the DIGEST ONLY:
+- A `__pycache__` or `.pytest_cache` DIRECTORY's own marker (its bare existence no longer moves
+  the hash) — but the directory is ALWAYS recursed into; every non-excluded child underneath is
+  still hashed. (Skipping recursion was the unsafe draft's hole: an excluded directory became an
+  unbounded, permanently-unhashed region a manifest hook `script` could point into —
+  `__pycache__/run.js` — ship benign, get consent, then rewrite freely afterward.)
+- A `.pyc`/`.pyo` FILE, but ONLY when its immediate parent directory's basename is exactly
+  `__pycache__`. A `.pyc`/`.pyo` anywhere else (bundle root, `scripts/`, a directory literally
+  named `cache.pyc`, etc.) stays hashed, because a sourceless legacy `.pyc` there is genuinely
+  importable/executable. (The unsafe draft matched the suffix anywhere in the tree.)
+
+`.DS_Store` is deliberately NOT excluded (the first draft excluded it; that exclusion was removed
+entirely, not narrowed). An excluded filename is a permanently unhashed name that a declared hook
+`script` could still be pointed at — e.g. `hooks/.DS_Store` — and `isSafeHookScriptPath` (below)
+was hardened only for the `__pycache__`/`.pytest_cache`/`.pyc`/`.pyo` shapes, not for `.DS_Store`.
+It was also unrelated to #3631's reported symptom (Python bytecode caching from a test run), so it
+was not worth carrying as a permanently unhashed name. `.DS_Store` now stays bound like any other
+file.
+
+`isSafeHookScriptPath` (`src/capability-lifecycle.cts` and its mirror
+`gsd-core/bin/lib/capability-validator.cjs`) rejects any DECLARED script path containing a
+`__pycache__`/`.pytest_cache` path segment, or whose basename ends `.pyc`/`.pyo` — a file named
+e.g. `x.pyc` can contain perfectly valid JavaScript and would be executed by `node` regardless of
+extension. This raises the bar for a manifest-declared hook `script`, but it is NOT a containment
+bound on the excluded region: it only ever inspects the declared `script` path string itself, not
+what that script `require`s/`import`s at runtime. A hashed, consent-covered `hooks/run.js`
+containing `require('../__pycache__/mod.pyc')` reaches the excluded region in one hop — Node loads
+an unregistered extension through its default `.js` handler — and the validator never sees that
+reference. Once loaded that way, the referenced `.pyc` is free to be rewritten post-consent with
+the digest unmoved. See the corrected bound below.
+
+**D4's argument is unaffected.** The claim this ADR rests on is that a single changed byte in a
+skill body deactivates a project-scoped capability until re-consent. Skill bodies are `.md` files
+and are not in the exclusion set, so that still holds exactly as written.
+
+**ACCEPTED RESIDUAL RISK — stated plainly, not glossed over.** An earlier version of this section
+claimed CPython "validates [a cached `.pyc`] against its sibling source" before trusting it. That
+claim is FALSE and was disproven by execution: CPython's default (timestamp-based) invalidation
+compares the cached `.pyc` header's stored mtime and size against the CURRENT source file's mtime
+and size — it does NOT check source content. Both mtime and size are ordinary file metadata an
+attacker who can already write to the bundle can forge. A forged `__pycache__/mod.cpython-3XX.pyc`
+whose header mtime/size were copied from an unmodified, still-hashed `mod.py` executes without
+moving this digest. Before this change, any write under `__pycache__` (even an empty directory)
+was detected; after it, a `__pycache__/*.pyc` matching that narrow shape is not. This is accepted
+deliberately — it stops routine bytecode caching from silently deactivating capabilities, which is
+the usability defect this exclusion exists to fix — and it is bounded by: (1) the attacker must
+already have POST-CONSENT write access to the bundle (this is not a remote-exploit surface); (2)
+everything outside `__pycache__/*.pyc` — including sourceless legacy `.pyc`/`.pyo` files anywhere
+else in the bundle — remains hashed. NOT a bound: the excluded region IS reachable by indirection
+from any hashed, consent-covered script — a `require`/`import` of a `__pycache__/*.pyc` path is one
+hop, not only CPython's own bytecode loading path described above — so `isSafeHookScriptPath`
+raises the bar for a DECLARED hook `script` surface but does not contain the risk. KNOWN
+LIMITATION: `.pytest_cache`'s CONTENTS still change the digest as ordinary hashed files — only its
+directory marker is suppressed, so this residual risk does not extend to `.pytest_cache`.
+
+Two properties were preserved deliberately and are pinned by tests: the exclusion is applied
+AFTER the symlink/non-regular fail-closed rejection (so a symlink named `x.pyc` still throws
+rather than being silently skipped), and excluded entries still count toward the walk's
+size/count caps. `node_modules`, `dist`, and `build` were considered and deliberately NOT
+excluded: their contents are required/executed at runtime, so dropping them from the digest
+would stop consent binding executable content.