Files
msd-core/docs/adr/3574-install-materialization-primitives.md
Tom Boucher 3ab0007164 enh(#2875): materialization primitives — durable user-artifact staging and descriptor-authoritative agents (#3600)
* fix(#2875): stage user artifacts durably across install wipes (#1874-F19)

preserveUserArtifacts held user files only in an in-memory Map across the
wipe, so any process death between preserve and restore lost them outright.

Seven call sites, not the four the issue records. Three of them never called
the helper at all - they open-coded the same read/wipe/write - so searching
for callers under-counted by construction; the extra sites were found by
sweeping for the pattern instead.

The worst is the mainline install path, where the crash window spans the
entire gsd-core tree copy rather than a single rmSync.

Adds src/user-artifact-staging.cts: durable on-disk staging with a record
written after the copies land as the commit point, plus recovery of orphaned
batches on the next run - without recovery the staged bytes survive but the
user's file is still gone, which would pass its own test while delivering
nothing.

Routes copyPreservingSymlink through installFs() so staging cannot bypass the
install fs seam, and reunites its symlink-safety docblock with the function it
documents.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(#2875): amend ADR-3574 with four claims disproved by implementation

Implementing Phase 6 disproved four statements the ADR rests on. The central
decision - no single materializer - is unaffected and stands.

Corrected: decision 3 was already satisfied, so nothing was extracted; the
agents-bypass runtime set omitted claude, kilo and opencode, and closing it
needed three new pieces of descriptor contract rather than proceeding on its
own terms; three of the four blockers the layout comment names were already
stale; and F19 is seven call sites, not four.

Records the generalizable lesson: the defect is the pattern of holding user
data in memory across a wipe, not the helper, so searching for callers of the
helper under-counts by construction.

Also resolves the ADR's open question on USER_OWNED_ARTIFACTS membership, and
notes that copyPreservingSymlink needed routing through the install fs seam
before it could be reused.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(#2875): close dangling-symlink blind spot and harden staging recovery

An adversarial review found the F19 staging work shipped red and unsafe.

Root cause, shared by two arbitrary-write findings: hasExistingSymlinkBetween
missed dangling symlinks in both its root check and its per-segment walk,
because it probed with existsSync, which is false for a link whose target does
not exist. Fixing only the new module would have reused a guard that was
itself blind. This guard protects the whole install tree.

Recovery no longer throws: it degrades per entry and per file, so one bad
batch cannot block the others. Previously an unrecoverable entry propagated
out of the first statement of install and uninstall, before the cleanup that
would have removed it - wedging the installer permanently.

Partial fs adapters now throw on any omitted method instead of silently
reaching the real filesystem, closing the trap that let a test poison list
pass while real IO happened.

Staged names must be flat, recovery refuses a dangling destination symlink,
and a batch whose recovery genuinely failed is no longer swept - it was
discarding the only durable copy of the file it had just failed to restore.

Replaces three tests that could not fail, including the one labelled negative
proof.

Known limitation, documented not closed: concurrent installs sharing a staging
key can still lose a batch. A real fix needs a cross-process lock.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* enh(#2875): make the descriptor authoritative for the agents kind

Deletes the inline agent-staging loop in bin/install.js and the
_DESCRIPTOR_AGENTS_RUNTIMES set, so every runtime materializes agents from
its capability descriptor instead of an inline hostBehaviors dispatch.

Closing it needed three pieces of contract the descriptor pipeline never had,
all reducible to one missing input - per-agent resolution context: a
frontmatter-extensions step for claude's effort and disallowedTools, per-agent
model-override resolution for kilo and opencode, and a named branding
converter for hermes, whose rewrite data was already declared.

Seven runtimes were on the loop, not the six the design recorded - kimi-code
was found by a golden fixture, not by analysis. claude-local and kimi-code
both silently lost their agents mid-change; the fixtures caught both and the
cause was fixed rather than the fixtures regenerated.

A parity harness gates the migration: both pipelines over identical inputs,
byte-identical output including filenames, per runtime. It is demonstrated
red before being trusted. Surface and install paths converge for all seven,
which also fixes surface previously writing no agents for these runtimes.

Codex's config.toml strip stays put - it mutates host config, which no
descriptor kind models.

Also routes install-model-override-resolver and install-effort-resolver
through the install fs seam. Both leaked real filesystem IO from the install
call tree; the stricter adapter is what exposed them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(#2875): record the agents-descriptor migration and correct the ADR count

The _DESCRIPTOR_AGENTS_RUNTIMES allow-list no longer exists, so the host
integration guide told readers to join a set that is gone. Replaces that with
what is now true - declare an agents entry and it installs, on the surface
path as well as install - and points anyone needing a per-agent transform at
the three extension points rather than at a new inline branch.

Corrects the ADR amendment: seven runtimes were on the inline loop, not six.
kimi-code was found by a golden fixture going red, not by reading. That is the
third short count this phase, all from enumerating by symbol or set membership
when the thing that matters is a behavior.

Adds the Changed changeset for the surface-path convergence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(#2875): amend ADR-2866 - claude global always wrote agents on disk

The claude row's global=[skills] described what capability.json declared, not
what the installer wrote. bin/install.js's inline agent-staging loop was never
scope-gated and never consulted the descriptor, so a claude --global install
has always written agents/gsd-*.md.

Phase 6 closes the gap by deleting that loop and declaring agents on claude's
descriptor at global scope. On-disk bytes are unchanged - the golden fixtures
did not move, which is the evidence that the descriptor, not the installer,
was incomplete.

#2218 is unaffected: agents are not trigger-bearing, so the wider row does not
introduce a new shadowing case.

Records the warning that an incomplete descriptor is invisible while a second
code path silently does its work, and only surfaces when the two are forced
into agreement.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(#2875): close review findings across staging, agents and the parity harness

Two independent reviews of this branch found defects the local gates missed.

Security: a dangling symlink at a migration destination allowed writing
outside configDir - the same class this change claimed to close, missed at the
terminal write of the flow being added. The staging-root resolver threw as the
first statement of install and uninstall, so a hostile symlink bricked both,
and symlinked-configDir users lost uninstall as well as install; it now
degrades instead of aborting. Recovery gained a source-side symlink check and
now refuses a relative destDir, which resolved against cwd. Converter dispatch
gained a runtime allowlist - lint-time validation stopped mattering once this
branch promoted that dispatch from the surface path to real installs.

Correctness: claude --local --minimal exited 1 because the minimal profile
legitimately yields zero agents and the new path treated that as a failure.
cline --local silently lost its agents - its descriptor declared none while
the deleted loop wrote them unconditionally. The agents prune was widened to
any gsd-* entry and destroyed user files it never owned.

The parity harness, on which the migration's safety argument rested, drove a
synthetic registry and never byte-compared the shipped descriptors; two of its
trap rows could not fail. It now drives the real registry across 13
runtime-scope rows including kimi-code and cline-local, and its red-proof is
demonstrated by corrupting a live capability.json. Three goldens that had
encoded the cline regression as expected behavior were corrected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(#2875): close findings from both mandated review engines

/security-review found the staging source-side walk honouring
GSD_ALLOW_SYMLINKED_DEST, an opt-in documented as relaxing only the write
destination. A symlinked files/ component dereferenced because
copyPreservingSymlink lstats the leaf only, so an intermediate link is
followed. The source walk no longer honours the opt-in; the destination check
still does.

/code-review spec axis found this branch had reintroduced its own bug:
migrateLegacyDevPreferencesToSkill's new symlink refusal threw unguarded after
the legacy dir was wiped and before the staged batch was restored, so a
planted symlink bricked uninstall permanently and orphaned the batch. Refusal
kept, abort removed.

kimi-code local silently lost its agents, the same class as the cline bug, and
the parity harness recorded that exclusion as intentional - the third test in
this branch to pin a regression as correct.

--minimal now creates an empty agents/ dir that never existed. Behaviour
restored rather than softening the changeset, so its byte-identical claim
stays true.

Standards axis: try/finally removed from twelve test bodies, fast-check
properties added for parseOwnerPid, boundary coverage at the grace window and
the ancestor-probe depth, a parity assertion for the staging-root helper
duplicated across two files, and the 8-deep config walk deduplicated.

Records 60-review.json with every finding and disposition from five passes,
including the smells left unfixed and why.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(#2875): prune stale agents unconditionally in minimal mode

The previous round stopped an empty agents/ directory being created when the
resolved profile yields no agents. That was implemented by skipping the agents
kind entirely, which also skipped its stale-agent prune - so a full to minimal
downgrade left stale gsd-* agents behind.

The deleted inline loop pruned unconditionally and only skipped writing. Those
are three separate conditions, not one: prune always, write only when there is
something to write, create the directory only when writing.

Both call sites now run _removeGsdEntries before the empty-staged early exit.
The symlink-escape guard moved with it, since the prune also touches dest.
Codex .toml agents and the config.toml stanzas are cleaned again, and
user-owned agents are still preserved.

The agents/ directory is left in place after a prune empties it, matching
every sibling kind - none of them remove the destination directory itself.

Golden fixtures confirmed byte-identical: the prune is a no-op on a fresh
install, so fixture generation is unaffected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(#2875): document interrupted-install recovery for user-owned files

The durable-staging fix is invisible to the user it protects. Someone whose
install died mid-flight has no way to know USER-PROFILE.md was staged before
the delete, that the next run restores it, or that recovery happens at the
start of that run rather than in the background.

Written as the task the user has - finish the interrupted command - rather
than as a description of the mechanism, and states what it will not do:
overwrite a file already present, or touch staging belonging to another
install still running.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* chore(#2875): backfill changeset pr number

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(#2875): assert the J8 model override without building a regex

CodeQL flagged incomplete string escaping: the assertion interpolated the
override value into a RegExp while escaping only forward slashes, which is
meaningless in a constructor, leaving real metacharacters unescaped.

The failure direction was the dangerous one - a metacharacter would have made
the match more permissive, so the row would pass when it should fail. That
matters here because J8 exists precisely because an earlier revision was a
tautology; the rewrite reintroduced a different way for the same assertion to
stop discriminating.

Replaced with a line-wise exact match, so no regex is constructed at all.
Swept the other test files this branch adds; no sibling instances.

lint:ci passed on the original - lint-no-adhoc-regex-escape matches a full
metachar-escape copy, so a single slash replace slipped under it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: sim <sim@local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-17 17:25:53 -04:00

17 KiB

ADR-3574: Install materialization shares primitives, not one writer

Context

Epic #2866 Phase 6 was scoped on the premise that "materialize this layout" is implemented three times and skipped once, and that the remedy is to extract the preserve → prune → stage → copy → restore choreography into one module with the three sites becoming callers.

That premise was measured against the tree on 2026-08-16, after Phase 5 (#2874) landed. It does not hold. The three sites overlap in shape and diverge in mechanism:

step installRuntimeArtifacts
src/install-engine.cts:770-958
applySurface
src/surface.cts:359-452
agent loop
bin/install.js:11120+
preserve snapshot-based, skills kind only (_snapshotDir) none none
prune _removeGsdEntries, prefix-scoped wipe pruneSkillDirs — allow-list, never wipes stale gsd-* unlink
stage copies straight into dest temp dir first, then syncs inline transform, no staging dir
restore _restoreDir the snapshot none none

The divergence is deliberate on at least one side. applySurface's prune is allow-list precisely so that it structurally cannot delete a user's files — its own doc comment ties that to the #2973/#3664 user-directory-preserving fix. installRuntimeArtifacts instead wipes a prefix-scoped set and restores a snapshot of the one user-owned directory it knows about.

A single writer must pick one of these. Forcing applySurface onto snapshot-restore would replace a design that cannot lose user files with one that deletes them and puts them back — trading a structural guarantee for a procedural one. Forcing installRuntimeArtifacts onto temp-staging adds a staging hop it does not need.

Two further premises of the original scoping are stale:

  • The agents bypass is shrinking, not static. _DESCRIPTOR_AGENTS_RUNTIMES (bin/install.js:11152) already routes ten runtimes — cursor, windsurf, augment, trae, codebuddy, copilot, antigravity, qwen, kimi, zcode — through the descriptor. The inline _hostBehaviors() dispatch survives only for codex, cline, hermes and generic runtimes.
  • The duplication comment is stale in the opposite direction. It lives at src/runtime-artifact-layout.cts:277-297 (not the range #2875 cites) and reads: "That duplication is deliberate until the second layout.kinds consumer — applySurface … — mirrors the legacy agent pipeline." That condition has partly been met, via agentCtx / stageAgentsForRuntimeWithConverter in applySurface.

Separately and independently, #1874-F19 is confirmed real: preserveUserArtifacts (src/install-engine.cts:168-179) builds an in-memory Map<string,string> via readFileSync, and restoreUserArtifacts (:187-195) writes it back. Nothing touches disk in between. Any process death between the intervening wipe and the restore loses the content outright, at four call sites (install-engine.cts:633, :705; bin/install.js:8658, :11056).

Decision

1. There will be no single materializer module

The three choreographies stay distinct. This ADR explicitly declines Phase 6's first acceptance criterion as written — "One module writes a Layout; the three former call sites delegate to it" — because satisfying it requires breaking one of two mechanisms that are each correct for their own caller.

Recording the refusal is the point: the next reader who notices three similar-looking loops should find this file rather than re-derive the extraction and rediscover the conflict.

2. What IS extracted: durable user-artifact staging (F19)

preserveUserArtifacts / restoreUserArtifacts move to a shared module and stage to a durable on-disk path before any wipe, reusing copyPreservingSymlink (src/installer-migrations.cts:166-177) — a pure two-argument function with no migration-specific state, already used by the backup-and-remove migration action in exactly this copy-strictly-before-delete order.

copyPreservingSymlink is the correct primitive for a second reason beyond durability: it never dereferences a symlink target. Its own doc comment records why — dereferencing could copy the bytes behind a link like ~/.ssh/id_rsa into the backup tree. A hand-rolled copyFileSync here would reintroduce that.

The journal / backupRoot / runId scaffolding around it is migration-specific and is not extracted. Only the primitive is shared.

3. What IS extracted: the retired-kind prune

pruneRetiredRuntimeArtifacts is already called by both installRuntimeArtifacts and applySurface with the same intent. That is genuine shared behavior rather than parallel evolution, and it is the one step where a single owner costs nothing.

4. The agents bypass is closed on its own terms

Removing the inline _hostBehaviors() agent dispatch so the descriptor is authoritative for every runtime is independent of the prune question and proceeds regardless. It is the part of Phase 6 whose evidence survived scrutiny intact, and ten runtimes have already made the trip.

5. Placement and content ownership are unchanged

Per ADR-3660, placement knowledge outside runtime-artifact-layout is drift; the extracted primitives consume Layout, never re-derive it. Per ADR-1508, "Layout owns placement; this module owns content", and the conversion module imports nothing upward. The primitives sit downstream of both and introduce no upward dependency.

Per ADR-58, these primitives are on the adapter side of the pure-policy/thin-adapter split — they execute IO. Phase 5 routed that IO through an injectable seam and established that the write-confinement decisions (hasExistingSymlinkBetween, assertDestWithinConfigHome) stay outside the adapter, so a fake cannot certify an install the real filesystem would refuse. The extraction must not relocate those decisions.

What this ADR does not decide

  • Whether the three choreographies ever unify. The applySurface descriptor-agents migration is partly landed; when it completes, the shapes may converge enough that the question is worth reopening on evidence. Revisit then, not before — and not by re-deriving the extraction this file declines.
  • The prune model itself. Whether allow-list or prefix-scoped-wipe is the better default across the installer is a real question and a separate one. Nothing here endorses either as canonical.
  • USER_OWNED_ARTIFACTS' membership. #2875 names USER-PROFILE.md as at risk; that could not be confirmed in this codebase state — dev-preferences.md is confirmed at three of four call sites. The implementing phase must enumerate the list rather than inherit the claim.

Consequences

  • Phase 6's acceptance criterion 1 is not met as written, deliberately. #2875 needs a scope update to match this decision before implementation. That is the cost of having measured the premise instead of executing it.
  • Three loops that look duplicated remain, now with a recorded reason. Future reviews should treat this file, not the loops, as the answer.
  • The agents bypass closes; the runtime-artifact-layout.cts:277-297 comment is rewritten rather than deleted, because its "deliberate until X" framing is stale in a way a plain deletion would not capture.
  • F19's durability fix lands here rather than in #1874, per maintainer direction on #2875. #1874's F5, F6 and F18 are untouched.
  • The regression test for F19 must inject the crash window by monkeypatching the fs method and restoring in a finally — never via chmod/permission tricks, which root bypasses, yielding a test that passes with zero coverage in root Docker and CI.

Alternatives considered

  1. One materializer, three callers delegate — Phase 6 as originally scoped. Rejected on the measured evidence above: it forces either applySurface off its non-wiping prune or installRuntimeArtifacts into unnecessary temp-staging.
  2. One materializer with a preservation-strategy parameter. Rejected. It preserves both behaviors but makes the module own two concepts and defer the choice to its callers — the exact widening ADR-2866 warns future reviews to resist, and a strategy flag is how a shared module becomes two modules wearing one name.
  3. Do nothing; leave F19 to #1874. Rejected. This phase rewrites that precise choreography, so landing the durability fix elsewhere means two conflicting passes over the same code.
  4. Delete the duplication comment as no longer true. Rejected. It is not simply false — it is stale in a specific, informative way, and its "deliberate until X" condition is now partly met. A rewrite carries that; a deletion loses it.

A note on the evidence

Blast-radius figures for this seam are not reliable and were not used to justify anything above. get_impact on installRuntimeArtifacts resolved to a same-named test helper (tests/adapter-declarative-equivalence.test.cjs:52) and reported zero affected — the same name-collision failure mode that produced a misleading clean radius during #3544. applySurface returned CRITICAL / 184+ from one tool and LOW / 0 from another, disambiguating to two different in-file matches of the same name. The decision above rests on read code, not on those numbers.

References

  • Epic: #2866; this phase: #2875; this ADR: #3574
  • Durability finding: #1874-F19 (and its closed child #1878 — do not re-file)
  • Placement seam: ADR-3660 · content seam: ADR-1508 · policy/adapter split: ADR-58
  • The epic's own frame: ADR-2866, which mandated that this module owe its own ADR
  • User-directory preservation this ADR protects: #2973, #3664

Amendment (2026-08-17, #2875): four factual claims corrected by implementation

Implementing this ADR as Phase 6 disproved four of the statements it rests on. The central decision — §1, no single materializer — is unaffected and stands; the divergence table that justified it was measured correctly. What follows corrects the surrounding claims, because a reader who acts on them will be misled.

This is the same failure mode the ADR itself warns about in "A note on the evidence": conclusions reached by reading code without executing it. Three of the four corrections below are cases where inspection produced a confident, wrong answer.

1. §Decision 3 is void — the retired-kind prune already had a single owner

The ADR says the prune "is extracted" and is "already called by both installRuntimeArtifacts and applySurface". Measured: pruneRetiredRuntimeArtifacts already lives alone in src/retired-artifact-cleanup.cts, already exports a single function, already routes every fs call through installFs(), and has three callers — installRuntimeArtifacts, uninstallRuntimeArtifacts and applySurface.

There was nothing to extract. No refactor was invented to satisfy this decision. A future reader should treat §3 as already-satisfied, not as outstanding work.

2. The agents-bypass runtime set was wrong, and §Decision 4 was the hardest part, not the easiest

The ADR states the inline dispatch "survives only for codex, cline, hermes and generic runtimes", and calls closing it "the part of Phase 6 whose evidence survived scrutiny intact".

Both are wrong. _DESCRIPTOR_AGENTS_RUNTIMES (bin/install.js) held ten runtimes; every other runtime reached the inline loop — seven of them: claude (the flagship), cline, codex, hermes, kilo, opencode and kimi-code. (pi is excluded separately by its pluginOnlyInstall branch.)

Even this correction undercounted. It originally said six. kimi-code was found only when a golden install-tree fixture went red mid-implementation — not by any amount of reading. That is the third time this phase's enumeration was short (four call sites → seven; six runtimes → seven), and every miss shares one cause: counting by symbol or set membership when the thing that matters is a behavior. Fixtures and executed tests found what inspection did not.

The set and the loop are both gone as of this phase; the descriptor is authoritative for agents on every runtime, so there is no longer an allow-list to join.

Worse, closing the bypass could not be done "on its own terms". It required three new pieces of descriptor contract, because the descriptor pipeline had no per-agent resolution context:

gap consumer
a frontmatter-extensions step (effort, disallowedTools) claude
per-agent model-override resolution threaded to the converter kilo, opencode
a named branding converter (the data was already declared; the converter was not) hermes

Every one of those failed silently if migrated without the contract work — wrong bytes, nothing thrown. §Decision 4's framing as independent and low-risk should not be relied on.

3. Three of the four blockers in runtime-artifact-layout.cts were already stale

The ADR treats that comment's blocker list as current. Measured, only one was:

blocker status
Copilot's .agent.md filename rename stale — #2099 dropped the ternary; the suffix comes from hostBehaviors.agentFileExtension
cross-cutting path-prefix rewrite + attribution stale — stageAgentsForRuntimeWithConverter already does both when agentCtx is present
stale-file cleanup stale — _removeGsdEntries prunes more broadly than the loop's extension-gated check
config-reading steps real — and it was the entire remaining gap (see §2 above)

4. F19 is seven call sites, not four — and the helper was the wrong thing to search for

The ADR names four call sites, found by locating callers of preserveUserArtifacts. There are seven. Three of them never call the helper at all; they open-code the same readFileSync → wipe → writeFileSync.

The generalizable lesson: the defect is the pattern "user data held only in memory across a wipe", not the helper. Searching for callers of the helper under-counts by construction. The three extra sites were found by sweeping for the pattern — a read shortly before a wipe and a write shortly after.

The ADR also understates the severity. The worst site is the mainline install path, where the window spans the entire gsd-core tree rebuild inside copyWithPathReplacement, not a single rmSync. Any interruption of a normal install destroys the file.

5. Resolved: USER_OWNED_ARTIFACTS

"What this ADR does not decide" records its membership as unconfirmable. It is confirmed: src/install-engine.cts defines it as exactly ['USER-PROFILE.md'], with a docblock recording the invariant that a file is either manifest-tracked distribution or a preserved user artifact, never both (#2771). dev-preferences.md is preserved at other sites by explicit name. That open question is closed.

§Decision 2 directs reusing it, and that is still the right primitive for the reason given (it never dereferences a symlink). But it used raw fs for all five of its calls, while its new caller sits on the install path Phase 5 routed through an injectable seam. Verbatim reuse would have punched a hole through that seam — the partial-adapter trap install-fs-adapter.cts documents. It was routed through installFs() as part of the extraction; its existing migration caller is unaffected, since the ambient default resolves to real fs.

A caution for anyone extending this module: that same fall-through is a live hazard. A missing method on an injected adapter does not fail loudly — it silently reaches the real filesystem. Adding a new installFs() call to a routed path without extending every adapter is a real-IO bug that passes typecheck.