msd-core/docs/research/3875-ack-sweep-automation-failure.md

Why the ack-fragment sweep (#3875) is not "fully automatic": PR #3927 case study

Status: research note — grounded in the repo's own source, workflow YAML, ADRs, and live gh/GitHub API data for PR #3927, pulled 2026-08-27.

Summary

ack-fragment-sweep.yml (cron 30 */6 * * *) correctly detects and plans the deletion of spent tests/emitted-drift-acks/*.json fragments, and correctly opens a PR. It does not, and structurally cannot, get that PR merged unattended:

1. Its hardcoded PR title (ack-fragment-sweep.yml:266) never satisfies the repo's own title-validator regex (scripts/release-notes/conventional-title.cjs:28,84-93) — a self-inflicted, deterministic failure on every run.
2. Its freeform PR body fails the typed-template check, but that check only warns (not fails) because the sweep authenticates as a MEMBER account, not because the body is compliant (scripts/pr-template-policy.cjs:167-240).
3. Its diff — one or more tests/emitted-drift-acks/*.json deletions — matches no rule in scripts/ci-test-scope.cjs's RULES table, so classify() falls through the #408 empty-match fallback and selects the 'unit' suite sentinel for the unsharded, 15-minute-capped "targeted/fast-signal" matrix lane (scripts/ci-test-scope.cjs:520-563, .github/workflows/test.yml:155,166-172). That lane then runs effectively the whole 827-file corpus and is killed by its own job timeout — verified directly in the run log (below). This is not flaky; it will recur on every future sweep PR whose diff is exactly the kind the sweep workflow always produces.
4. There is no auto-merge path for this PR anywhere in the workflow set. It sat open 2h38m and was merged by a human maintainer (trek-e, MEMBER) via a merge that landed while validate-title and Required tests were still red — an override, not an automatic pass.

A. The model itself

A1. What an emitted-drift-ack fragment is, and its lifecycle

Read in full: scripts/lint-emitted-drift-ack.cjs (937 lines) and tests/emitted-drift-acks/README.md.

• Purpose (scripts/lint-emitted-drift-ack.cjs:1-31): the differential-attribution gate (tests/emitted-attribution.test.cjs, ADR-2719) requires every emitted-artifact byte or size delta between next and a PR's HEAD to be attributable to that PR's own diff. When it legitimately cannot be (a converter change, a deliberate size grow), the escape hatch is a committed acknowledgment fragment under tests/emitted-drift-acks/<name>.json, one file per PR (#2914), never the legacy single tests/emitted-drift-ack.json.
• Two key spaces (tests/emitted-drift-acks/README.md:20-23): an unattributable hash ripple is keyed on the emitted path; workflow/agent-file growth is keyed on the bare filename.
• Creation: the differential check's own failure output prints the exact fragment name/shape to paste (CONTRIBUTING.md:1074-1076, docs/TESTING-SUITES.md:139-145) — a human authors it as part of the PR that needs it.
• "Spent" (scripts/lint-emitted-drift-ack.cjs:304-322, function assertNoAllSpentFragments): every entry's prose in the fragment is compared, byte-for-byte after invisible-character/whitespace normalization (ackProse, line 261), against the copy of the same fragment at the pre-push tip of next (readFragmentAtRef, line 538). If every surviving entry matches its base-ref copy, the whole fragment is "all-spent" — its ripple is now baked into next, so it can no longer clear anything, yet it still owns its path keys (a second source declaring the same key is a hard duplicate-key error, lines 872-885).
• Red on next: the guard-no-ack-on-next job (.github/workflows/test.yml:865-920) runs on every push to next and calls --guard-next --base-ref <before> --defer-to-open-prs. It fails when (a) the legacy tests/emitted-drift-ack.json is present at all (assertAbsentOnNext, line 449), or (b) any fragment is fully spent and not held by an open PR that still touches it (#3842 deferral, line 363).
• Remedy: git rm the fragment. Since #3875, ack-fragment-sweep.yml automates exactly this git rm, computed via --sweep-plan (lines 815-833), which asks the guard for the identical set it reasoned about rather than re-deriving it.

A2. Governing ADRs/docs (verbatim quotes)

• ADR-2719 (docs/adr/2719-emitted-artifact-attribution.md:1-11): "Emitted-artifact attribution — replace the committed parity fixtures with a computed conservation law." Accepted 2026-07-27, supersedes ADR-2264 §2-4.
• CONTEXT.md:625 (RULESET.EMITTED_ATTRIBUTION): "...#2914 replaced the single shared ack file with per-PR fragments...exactly the shape .changeset/ already uses for the identical 'every PR rewrites one shared document' conflict problem...#3078 REVERSED that [premise] — fragments do not share a FILE but they DO share a PATH KEY SPACE...#3875 automated the REMEDY that alert asks for, because detection without an executable remedy is what actually failed: #3823 shipped the guard together with a static 45-fragment sweep computed at its own branch point, #3809's fragment merged to next while it was in flight, and the guard reddened on its own merge commit and stayed red for 24 consecutive pushes over two days."
• CONTRIBUTING.md:1086-1094: "every PR needing an acknowledgment used to rewrite tests/emitted-drift-ack.json's paths map wholesale — a single shared mutable file every such PR touches guarantees a merge conflict between any two of them (5 of 6 conflicting PRs in one open queue collided on this file and nothing else)."
• tests/emitted-drift-acks/README.md:37-41: "The sweep is automated because the manual remedy could not keep up. The guard evaluates at MERGE time; a hand-authored git rm is fixed at BRANCH time...#3823 lost exactly that race to #3809 on its own merge commit and left next red for 24 consecutive pushes."

A3. Every CI surface referencing the ack model

• .github/workflows/test.yml:865-920 — guard-no-ack-on-next job (push-to-next trigger; the "next-lane guard"). Also referenced at test.yml:842-864 (comment block explaining scope/limits) and test.yml:127 (the lint-tests step runs node scripts/lint-emitted-drift-ack.cjs as part of lint:ci, see CONTRIBUTING.md cross-ref and the raw log line captured in this session: lint-tests step at 2026-08-27T09:38:55Z runs the full chained lint including node scripts/lint-emitted-drift-ack.cjs).
• .github/workflows/ack-fragment-sweep.yml — the entire file (237 lines); scheduled sweep, sweep job, calls --guard-next --sweep-plan --defer-to-open-prs (line 90-92) and --guard-next --sweep-plan unheld variant (line 115-116).
• scripts/lint-emitted-drift-ack.cjs — the validator/guard/sweep-plan CLI itself (937 lines; exports validateAckText, assertAbsentOnNext, assertNoAllSpentFragments, runGuardNext, main, fetchOpenPrTouchedAckPaths, etc.).
• tests/helpers/emitted-diff.cjs and tests/helpers/emitted-runtime.cjs — the "ships"-side duplicate of the same constants/logic (ACK_INVISIBLE, MAX_ACK_FRAGMENTS, RESERVED_ACK_KEYS, readAckFileAtRef), held to parity by tests/emitted-attribution.test.cjs (per scripts/lint-emitted-drift-ack.cjs:19-23,58-63,77-81,94-99).
• No eslint-rules/* or .claude/hooks/* file references emitted-drift/ack fragments (grepped both directories; zero hits).
• docs/TESTING-SUITES.md:130-160 and CONTRIBUTING.md:1055-1124 — the human-facing "Editing shipped content" / TESTING-SUITES how-to sections.

A4. Design-history issues (via gh issue view)

• #2789 — root-caused why merging an ack reddened next: the ack set was the one input read absolutely (working tree only) in an otherwise base-relative differential machine, so a spent ack (already at base) still looked live to staleAcks. Fixed by scoping every ack to the diff that introduced it.
• #3078 — proved #2914's premise wrong. #2914 exempted the fragment directory from the next-lane guard on the theory that independently-named fragments "cannot conflict with any other PR." #3078 measured 45 fragments owning 403 paths on next and showed fragments share a path key space even without sharing a file, so a spent fragment left in place still walls off the next PR that grows the same path. Extended guard-no-ack-on-next to cover fragments (implemented in PR #3823, "Closes #3078," merge commit a84f75630).
• #3842 — the #3078 sweep, once it started deleting fragments, handed three outside-contributor open PRs (#3330, #3774, #3648) a modify/delete git conflict apiece, each on the exact fragment it was the PR's only conflicting path. Added the --defer-to-open-prs hold (fetchOpenPrTouchedAckPaths).
• #3823 (a PR, not a separate issue — gh issue view 3823 resolves to the PR body) — implemented the fragment half of guard-no-ack-on-next, closing #3078.
• #3875 — root-caused the 24-consecutive-push red streak: #3823 computed its sweep as a static list at branch time; #3809's fragment merged to next while #3823 was in flight and was therefore never in that list. Proposed the scheduled sweeper (ack-fragment-sweep.yml) that recomputes the plan from the guard itself, on a timer, "honouring the existing --defer-to-open-prs hold." This issue is exactly what PR #3927 is discharging.

A5. Blast radius of removing the model entirely

Files/jobs/tests that would need to change if emitted-drift-ack fragments were removed outright:

• scripts/lint-emitted-drift-ack.cjs — deleted (or gutted to a no-op).
• .github/workflows/test.yml:865-920 (guard-no-ack-on-next job) and its two checkout/fetch steps (test.yml:875-895) — deleted.
• .github/workflows/ack-fragment-sweep.yml — deleted entirely (237 lines).
• tests/emitted-drift-acks/ directory and its README.md — deleted.
• tests/helpers/emitted-diff.cjs, tests/helpers/emitted-runtime.cjs — the ack-reading halves (readAckFileAtRef, mergeAckSources, readAckSources, readAckSourcesAtRef, isSpent, staleAcks, spentAcks) removed; tests/emitted-attribution.test.cjs's differential check would then need a different escape hatch for legitimate unattributable ripples, or none at all (making every ripple a hard failure).
• tests/emitted-attribution.test.cjs — its ack-consuming assertions and the "#2914 migration pins" mentioned in tests/emitted-drift-acks/README.md:64-69 removed.
• tests/pr-template-policy.test.cjs, scripts/pr-template-policy.cjs — unaffected directly, but the sweep PR's body text (which cites the ack model) would need rewriting if the workflow itself survives in some other form.
• CONTEXT.md's RULESET.EMITTED_ATTRIBUTION entry (CONTEXT.md:625) and the ### Emitted Artifact Provenance section it cross-references — rewritten.
• CONTRIBUTING.md:1055-1124 ("Editing shipped content" ack sections) — rewritten.
• docs/TESTING-SUITES.md:130-160 (the "workflow or agent grew" how-to) — rewritten.
• docs/adr/2719-emitted-artifact-attribution.md — would need a superseding ADR documenting the new escape hatch (or its removal), per this repo's own convention that architectural reversals get a new ADR rather than a silent edit.
• Any lint/test referencing tests/emitted-drift-ack.json or tests/emitted-drift-acks/ by path (grep -rl across .github/workflows/, scripts/, tests/ returns test.yml, ack-fragment-sweep.yml, lint-emitted-drift-ack.cjs, docs/TESTING-SUITES.md, CONTRIBUTING.md, CONTEXT.md, and the tests/emitted-drift-acks/README.md itself — all of the above).
• Issues #2789, #2914, #3078, #3842, #3875, #3823 would all become historical/moot context requiring a note that the model they describe no longer exists.

B. Why #3927 hung — one root cause per red signal

B6. validate-title — the sweep's own hardcoded title cannot pass its own gate

The sweep hardcodes: --title "chore: sweep spent ack fragments from next (${SHORT_SHA})" (ack-fragment-sweep.yml:266), producing the literal title chore: sweep spent ack fragments from next (929e02cb) for PR #3927.

The gate (.github/workflows/pr-title-validator.yml) delegates to evaluatePrTitle() in scripts/release-notes/conventional-title.cjs:81-97, which requires a header matching HEADER_RE = /^([a-z]+)(\([^)]*\))?(!)?:/i (line 28) and a scope group containing #\d+ (ISSUE_REF_IN_SCOPE_RE, line 31; checked at line 92).

Against chore: sweep spent ack fragments from next (929e02cb): HEADER_RE matches with type="chore" and scope-group undefined (there is no (...) immediately after chore — the colon follows directly). Since !scope is true, evaluatePrTitle returns { valid: false, reason: 'missing-issue-ref' } (line 92-93) — confirmed as the actual CI failure (gh pr checks 3927: validate-title fail 8s).

Two independent defects in the hardcoded string, either one sufficient to fail: (1) the parenthetical (${SHORT_SHA}) sits at the end of the title, not immediately after the type, so it is never read as the "scope" group at all; (2) even if repositioned, a short git SHA (929e02cb) contains no #, so it would still fail ISSUE_REF_IN_SCOPE_RE.

A title that would pass: chore(#3875): sweep spent ack fragments from next (929e02cb) — matches the repo's own worked example in conventional-title.cjs:71 (enhance(#1549): add PR-title validator) and correctly attributes the sweep to the issue it closes toward (#3875), which the current PR body already does in prose but the title does not.

B7. Pull request template format — passed, but only because the account is trusted, not because the body complies

Contrary to the initial framing, this check passed (gh pr checks 3927: Pull request template format pass 12s). The mechanism, read in full from scripts/pr-template-policy.cjs:

• The sweep's PR body (ack-fragment-sweep.yml:243-261) is freeform prose — no ## Fix PR / ## Enhancement PR / ## Feature PR heading, none of the required headings in TEMPLATES (pr-template-policy.cjs:55-97).
• The changed-files carve-out does not apply: tests/emitted-drift-acks/*.json is not in TOOLING_PATH_ALLOWLIST (pr-template-policy.cjs:24-43), so allPathsAreTooling returns false.
• matchingTemplate() finds no heading match → template: null → falls to reason = 'PR body does not match the fix, enhancement, or feature template.', valid: false (pr-template-policy.cjs:217-223).
• The consequence is gated on author trust, not template compliance (pr-template-policy.cjs:226-229): action = trusted ? 'warn' : 'close'. TRUSTED_AUTHOR_ASSOCIATIONS includes MEMBER (line 6-11). The PR's actual author association is MEMBER (gh api repos/golem15/msd-core/pulls/3927 --jq '.author_association' → MEMBER), because the workflow authenticates gh pr create with MSD_BOT_PR_TOKEN — a personal-access token belonging to a real maintainer account (trek-e), not a GitHub App/bot identity. So action: warn: the msd-pr-template-policy bot comment fires (core.warning, pr-template-format.yml's "Warn trusted contributor" step), but the workflow only core.setFaileds on action == 'close' — never reached. The template body is objectively wrong; the check is objectively green.

B8. Required tests / test (ubuntu-latest, 24) — the fast-signal lane ran the WHOLE suite and hit its own 15-minute cap

gh run view 33059472019 --repo golem15/msd-core --log-failed and the per-job API (jobs/98474440239, jobs/98478252101) show:

• test (ubuntu-latest, 24) — the unsharded matrix entry (.github/workflows/test.yml:198-200: {os: ubuntu-latest, node-version: 24, scope: targeted}, no shard key, hence the bare job name per the name: template at test.yml:130) — started 09:38:38Z, its "Run scoped tests" step ran from 09:39:10Z to 09:53:51Z (14m41s), and the whole job was marked cancelled at 09:53:56Z, total 15m18s — exactly the job's timeout-minutes: 15 cap (test.yml:155).
• The raw log for that job shows run-tests: suite="all" files=827 at 09:39:11Z, and the job reached only chunk 13/14 before being killed (chunk 13/14 — 61 files at 09:52:37Z) — i.e. it was running virtually the entire test corpus (827 files) in one unsharded process, not a narrow "targeted/fast-signal" subset. The three actually-sharded ubuntu scope: full jobs (which exist precisely to spread this same corpus across 3 runners, per the #2952/#3057 comments at test.yml:141-164) all finished successfully in 6-9 minutes each.
• Root cause, traced through the scope classifier: scripts/ci-test-scope.cjs's RULES array (lines 71-324) has no entry matching tests/emitted-drift-acks/*.json. In classify() (lines 483-591), the changed path starts with tests/ so productOrPipelineChanged = true (line 502), but it is not a .test.cjs file so it is not added to targeted/windows (line 520 requires endsWith('.test.cjs')), and no RULES[].match fires for it. The #408 fallback then fires: if (codeChanged && targetedTests.length === 0) targetedTests.push('unit') (lines 561-563). scripts/ci-prepare-test-scope.cjs's resolveSelection() (lines 49-71) passes that non-empty ['unit'] list through verbatim to .ci-selected-tests.txt, and 'unit' is a SUITE_SENTINEL (ci-prepare-test-scope.cjs:22) that run-tests.cjs resolves live to (effectively) the whole non-integration/security/install/slow corpus — 827 files, matching the observed log.
• Required tests (.github/workflows/test.yml:702-703) then reads TEST_RESULT: cancelled (its own job log, captured verbatim above) and fails with ##[error]test matrix did not pass — a mechanical propagation of the canceled test job, not an independent finding.
• This is deterministic, not flaky. Every sweep PR's diff is, by construction, one or more deletions under tests/emitted-drift-acks/ — a path class the scope-classification table has never had a rule for. Any future run of ack-fragment-sweep.yml will hit the identical fallback and the identical 15-minute cliff, for the same reason PR #3094 (referenced in test.yml:151-153) hit it on the scope: windows lane before that lane was sharded — except the scope: targeted lane was never sharded because nothing was expected to widen it to "all."

B9. How #3927 was actually merged despite red required checks

gh api repos/golem15/msd-core/pulls/3927 --jq '.merged_by.login, .merge_commit_sha, .author_association' → trek-e, ad6abc896..., MEMBER. The PR was opened 09:38:15Z and merged 12:16:49Z (2h38m later) by the same account, trek-e (a human maintainer, is_bot: false per gh pr list ... --json mergedBy). The timeline (gh api .../issues/3927/timeline) shows only review_requested → two bot commented events → merged/closed/head_ref_deleted, all attributed to trek-e or github-actions[bot]; no re-run or re-triggered check event appears between open and merge. gh pr checks 3927 (queried post-merge) still reports validate-title fail and Required tests fail as the latest, final state of those checks — they were never turned green. The merge therefore landed with required checks still red: a maintainer override (admin merge / branch-protection bypass), not an automatic pass triggered by the automation.

B10. No auto-merge path exists for the sweep PR

ack-fragment-sweep.yml ends at gh pr create + gh pr edit --add-label automation --add-label no-changelog (lines 263-280) — no gh pr merge, no enableAutoMerge call, nothing. Grepping every workflow for automerge/auto-merge finds only auto-backmerge.yml (an unrelated main→next release-engineering workflow, gated on push: branches: [main], using -s ours merges — nothing about ack fragments) and a comment in release.yml:729 about release→main PRs. pr-mergeable-preflight.yml is a reusable merge-conflict early-exit gate shared by eight caller workflows; it applies no label and performs no merge (pr-mergeable-preflight.yml:1-40). There is no maintainer-facing auto-merge wired to ack-fragment-sweep.yml's PRs at all — every one requires a human to notice it, review it, and merge it by hand (or override), exactly as ack-fragment-sweep.yml:14-15's own top-of-file comment states: "a human still approves the deletion."

B11. MSD_BOT_PR_TOKEN fallback warning did NOT fire — ruled out as a cause

gh run view 33059445125 --repo golem15/msd-core --log (the sweep workflow's own run) shows the "Open the sweep PR" step's env block: HAS_BOT_TOKEN: 1. The step's guard is if [ -z "${HAS_BOT_TOKEN:-}" ]; then echo '::warning::MSD_BOT_PR_TOKEN is unset...'; fi (ack-fragment-sweep.yml:191-193) — with HAS_BOT_TOKEN=1 this branch is skipped, and indeed no such warning line appears anywhere in the run log. MSD_BOT_PR_TOKEN was configured and used. This is corroborated independently by B7/B9: the PR's author_association is MEMBER (not the unauthenticated-fallback shape) and the title/template required checks did run and report real verdicts (validate-title, Pull request template format both executed) — under the GITHUB_TOKEN fallback the workflow's own comment (ack-fragment-sweep.yml:189) states "no required checks will run on it," which is not what happened. The fallback-token hypothesis is not the cause of this hang. (Git commit authorship on the branch itself still reads github-actions[bot] per git config user.name at ack-fragment-sweep.yml:204-205 — a separate, cosmetic identity from the gh API actor, which is trek-e via the PAT.)

C. Track record — is #3927 representative or a one-off?

gh pr list --repo golem15/msd-core --search "head:chore/ack-sweep" --state all --limit 30 --json number,title,state,createdAt,mergedAt,mergedBy returns exactly one PR: #3927 itself (opened 09:38:15Z, merged 12:16:49Z, mergedBy: trek-e). There is no prior sweep PR to compare against — this is the automation's first (and so far only) production run that produced a non-empty plan and opened a PR. It cannot yet be called "systematically failing" by volume, but the failure mechanism identified in B8 is a property of the scope-classification table and the sweep's title-generation code, not of this specific run's data — so it will reproduce on the next sweep PR with high confidence, absent a fix to either ack-fragment-sweep.yml's title format or scripts/ci-test-scope.cjs's RULES table (or both).

Open questions

• Should scripts/ci-test-scope.cjs's RULES gain an explicit entry for tests/emitted-drift-acks/ (mapping to a narrow, cheap test list — e.g. tests/emitted-attribution.test.cjs, tests/lint-emitted-drift-ack.cjs's own test file if any) so a fragment-only diff no longer falls through the #408 "unit" fallback onto the unsharded lane? This is the most direct fix for B8, and does not require sharding the scope: targeted lane.
• Should ack-fragment-sweep.yml:266 be changed to --title "chore(#3875): sweep ${COUNT} spent ack fragment(s) from next (${SHORT_SHA})" (or similarly scoped) to satisfy conventional-title.cjs outright? This is a one-line fix for B6.
• Should the sweep PR body be reshaped to match one of the three typed templates (likely "Fix" or a new "chore" carve-out) so Pull request template format passes on its merits rather than on the author's trust level — closing the latent gap where an untrusted-token run of this same workflow (if MSD_BOT_PR_TOKEN ever lapses to a non-MEMBER identity) would hit action: close instead of warn?
• Given B10/B11, should the workflow (or a separate follow-up automation) add a bounded auto-merge step once its own required checks are green, so a correctly-titled, correctly-scoped sweep PR does not still require a human to notice and merge it by hand every six hours?