golem15/msd-core
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: PID-liveness gate for the two core-path file locks (audit M1+M2)

Browse Source 
The STATE.md write lock (acquireStateLock) and the .planning/ workspace lock
(withPlanningLock) stole contended locks on mtime age alone with no
process.kill(pid,0) liveness check, and mis-ordered stale-vs-wait so a
live-but-slow holder could be robbed mid-critical-section.

M1 (lost update / STATE.md corruption): a live writer whose critical section
ran past the stale threshold aged out and a waiter unlinked its lock and
acquired -> two writers in STATE.md's read-modify-write window. mtime is a
leaky proxy for "holder is alive"; it leaks under exactly the slow-holder
condition the lock guards against.

M2 (uncaught EEXIST): withPlanningLock's timeout fallback unconditionally
unlinked whatever lock existed (even a live holder's) and re-acquired OUTSIDE
any try -- a concurrent re-create raced a raw EEXIST out of the helper.

Fix backports capability-lock.cts's liveness gate (process.kill(pid,0) via a
_setLockProbes/_resetLockProbes test seam):
- acquireStateLock: steal when holder pid is DEAD (any age) OR age exceeds a
  deadman ceiling (60000ms, ABOVE maxWaitMs=30000) so a verified-live holder is
  never stolen within budget; garbage/legacy bodies stay recoverable.
- withPlanningLock: same gate in the EEXIST path (dead stolen promptly, live
  waited on); removed the unconditional force-steal -> clear timeout throw,
  which also closes M2 (no re-acquire outside try).

Uncontended path unchanged byte-for-behaviour; realClock + real process.kill
remain the defaults. Pid-reuse residual fails safe (waits/times out, never
corrupts) and recovers at the deadman ceiling.

Tests: TDD red->green via the clock + new pid-liveness probe seams (no
wall-clock; #453 deleted the race tests). 8 new behavioural tests across
tests/clock-seam.test.cjs and tests/planning-workspace.test.cjs; the prior
withPlanningLock timeout test rewritten to pin the no-force-steal contract.

Claude-Session: https://claude.ai/code/session_01R88n7Q54bAaVHFkDbbH1yz
This commit is contained in:
Dave
2026-06-21 09:35:19 -04:00
parent eb81faaeab
commit fc9bd70aff
4 changed files with 381 additions and 63 deletions
Download Patch File Download Diff File Expand all files Collapse all files

83
src/planning-workspace.cts
View File

@@ -37,6 +37,53 @@ process.on('exit', () => {
}
});
// ---------------------------------------------------------------------------
// Lock liveness probe (test seam) — audit M1
//
// mtime is a leaky proxy for "the holder is alive". The prior withPlanningLock
// timeout fallback unconditionally unlinked WHATEVER lock existed — even a fresh,
// live holder's — and re-acquired it, force-stealing a live writer's critical
// section. We backport capability-lock.cts's pid-liveness gate: a dead holder is
// stolen promptly inside the polite loop; a live holder is waited on. The
// indirection lets unit tests inject a deterministic isPidAlive without real pids.
// ---------------------------------------------------------------------------
/** Is `pid` a live process? process.kill(pid, 0) succeeds for a live (signalable) process. */
function _realIsPidAlive(pid: number): boolean {
try {
process.kill(pid, 0);
return true; // signalable → alive
} catch (err) {
// EPERM = process exists but we cannot signal it (still ALIVE). ESRCH = gone.
return (err as NodeJS.ErrnoException).code === 'EPERM';
}
}
const _planningLockProbes: { isPidAlive: (pid: number) => boolean } = { isPidAlive: _realIsPidAlive };
function _planningLockIsPidAlive(pid: number): boolean {
return _planningLockProbes.isPidAlive(pid);
}
/**
* Is the holder recorded in the .lock body VERIFIED-LIVE? The body is JSON
* { pid, cwd, acquired }. Returns true ONLY when the body parses AND the recorded
* pid signals alive. A garbage / pid-less / unreadable body (or a dead pid) is NOT
* verified-live, so the lock stays stealable — corrupt locks never block forever,
* and a live holder is never force-stolen.
*/
function _planningHolderVerifiedLive(lockPath: string): boolean {
let parsed: unknown;
try {
parsed = JSON.parse(fs.readFileSync(lockPath, 'utf-8'));
} catch {
return false; // unreadable / unparseable body → cannot verify → not verified-live
}
const pid = (parsed as { pid?: unknown } | null)?.pid;
if (typeof pid !== 'number' || !Number.isInteger(pid) || pid <= 0) return false;
return _planningLockIsPidAlive(pid);
}
// Transient errno codes that indicate a temporary filesystem condition under
// concurrent O_EXCL races — Docker overlay-fs (ENOENT/EINVAL/EIO), NFS
// (ESTALE), and OS-level interrupt/retry signals (EAGAIN/EINTR). These are
@@ -160,16 +207,18 @@ function withPlanningLock<T>(cwd: string, fn: () => T, clock?: Clock): T {
continue;
}
if (nodeErr.code === 'EEXIST') {
// Lock exists — check if stale (>30s old)
// Liveness-gated steal (audit M1). Steal the lock PROMPTLY only when its
// recorded holder is NOT verified-live (crashed/dead pid or garbage body).
// A verified-live holder is waited on — never force-stolen — because nuking
// a slow-but-live writer's lock corrupts the .planning/ critical section.
try {
const stat = fs.statSync(lockPath);
if (clock.now() - stat.mtimeMs > 30000) {
if (!_planningHolderVerifiedLive(lockPath)) {
fs.unlinkSync(lockPath);
continue; // retry
continue; // dead/garbage holder — retry immediately to grab the freed lock
}
} catch { continue; }
// Wait and retry (cross-platform, no shell dependency)
// Live holder — wait and retry (cross-platform, no shell dependency).
clock.sleep(100);
continue;
}
@@ -177,10 +226,18 @@ function withPlanningLock<T>(cwd: string, fn: () => T, clock?: Clock): T {
}
}
// Timeout — stale-lock recovery, then re-acquire atomically before entering critical section.
try { fs.unlinkSync(lockPath); } catch { /* ok */ }
acquireLock();
return runWithHeldLock();
// Timeout against a holder still present at budget exhaustion. The polite loop
// already stole any DEAD holder; reaching here means the holder is verified-live
// (or a pid-reuse alias we must not corrupt). Do NOT force-steal — the prior
// unconditional `unlinkSync(lockPath); acquireLock()` here (audit M1) robbed live
// writers, and its re-acquire sat OUTSIDE any try so a concurrent re-create raced
// a raw EEXIST out of the helper (audit M2). Surface a clear timeout error instead.
const timeoutErr = new Error(
'withPlanningLock: ' + lockPath + ' held by a live process for ' +
(clock.now() - start) + 'ms (exceeded ' + lockTimeout + 'ms budget)'
);
(timeoutErr as unknown as Record<string, unknown>).lockTimeout = true;
throw timeoutErr;
}
function createPlanningWorkspace(cwd: string, opts: WorkstreamAdapterOpts = {}): {
@@ -269,4 +326,12 @@ export = {
getActiveWorkstream,
setActiveWorkstream,
findContextMdIn,
// Test seam (audit M1): inject a deterministic isPidAlive so the liveness-gated
// steal decision is exercised without real pids. Mirrors capability-lock.cts.
_setLockProbes(probes: Partial<{ isPidAlive: (pid: number) => boolean }>): void {
if (typeof probes.isPidAlive === 'function') _planningLockProbes.isPidAlive = probes.isPidAlive;
},
_resetLockProbes(): void {
_planningLockProbes.isPidAlive = _realIsPidAlive;
},
};

78
src/state.cts
View File

@@ -152,6 +152,54 @@ process.on('exit', () => {
}
});
// ---------------------------------------------------------------------------
// Lock liveness probe (test seam) — audit M1
//
// mtime is a LEAKY proxy for "the holder is still alive": a live-but-slow writer
// whose critical section runs past staleThresholdMs ages out and a waiter would
// steal its lock → two writers in STATE.md's read-modify-write window → lost
// update / corruption (the recurring #500/#905/#1230 family). The real signal —
// process.kill(pid, 0) — is already used by capability-lock.cts. We backport it
// here. The indirection lets unit tests inject a deterministic isPidAlive without
// real pids (mirrors capability-lock's _lockProbes / _setLockProbes seam).
// ---------------------------------------------------------------------------
/** Is `pid` a live process? process.kill(pid, 0) succeeds for a live (signalable) process. */
function _realIsPidAlive(pid: number): boolean {
try {
process.kill(pid, 0);
return true; // signalable → alive
} catch (err) {
// EPERM = process exists but we cannot signal it (still ALIVE). ESRCH = gone.
return (err as NodeJS.ErrnoException).code === 'EPERM';
}
}
const _stateLockProbes: { isPidAlive: (pid: number) => boolean } = { isPidAlive: _realIsPidAlive };
function _stateLockIsPidAlive(pid: number): boolean {
return _stateLockProbes.isPidAlive(pid);
}
/**
* Is the holder recorded in the lock body VERIFIED-LIVE? The STATE.md lock body is
* a bare pid (written at acquire time). Returns true ONLY when the body parses to a
* positive integer pid AND that pid signals alive. A garbage / non-numeric / legacy
* body (or a dead pid) is NOT verified-live, so the lock stays stealable — corrupt
* locks never block forever, and a live holder is never stolen.
*/
function _stateHolderVerifiedLive(lockPath: string): boolean {
let body: string;
try {
body = fs.readFileSync(lockPath, 'utf-8');
} catch {
return false; // unreadable body → cannot verify → not verified-live (stealable under ceiling)
}
const pid = parseInt(body.trim(), 10);
if (!Number.isInteger(pid) || pid <= 0 || String(pid) !== body.trim()) return false;
return _stateLockIsPidAlive(pid);
}
// Hoisted to module scope — compiled once, not per call (#320). Stateless (/i, used with .match).
const byPhaseTablePattern = /(\|\s*Phase\s*\|\s*Plans\s*\|\s*Total\s*\|\s*Avg\/Plan\s*\|[ \t]*\n\|(?:[- :\t]+\|)+[ \t]*\n)((?:[ \t]*\|[^\n]*\n)*)(?=\n|$)/i;
@@ -1587,8 +1635,14 @@ function acquireStateLock(statePath: string, clock?: StateLockClock): string {
if (clock === undefined) clock = realClock;
const lockPath = statePath + '.lock';
const retryDelay = 200; // ms
const staleThresholdMs = 10000;
const maxWaitMs = 30000;
// Deadman ceiling (audit M1) — set ABOVE maxWaitMs so a holder that reads as
// VERIFIED-LIVE is NEVER stolen within the wait budget; only a crashed (dead
// pid) or unparseable-body lock is stolen, and a pid-reuse holder (reads alive
// but is unrelated) is recovered once age crosses this absolute ceiling rather
// than blocking forever. The prior mtime-only `staleThresholdMs = 10000` gate
// was BELOW maxWaitMs, so a live-but-slow holder >10 s was robbed mid-write.
const deadmanCeilingMs = 60000;
const startedAt = clock.now();
// Shared helper: check the time budget then back off with jitter before the
@@ -1625,12 +1679,18 @@ function acquireStateLock(statePath: string, clock?: StateLockClock): string {
continue;
}
if ((err as NodeJS.ErrnoException).code !== 'EEXIST') throw err; // propagate — silent bypass causes lost updates
// Only unlink a lock we did not place when it has crossed the staleness
// threshold (crashed holder). Nuking a fresh lock held by a slow-but-live
// writer causes lost updates (#3711 regression).
// Liveness-gated steal (audit M1). Only unlink a lock we did not place when
// either (a) its recorded holder is NOT verified-live — a crashed/dead pid or
// a garbage/legacy body — so it is stolen PROMPTLY regardless of age, or
// (b) its age has crossed the absolute deadman ceiling (set above maxWaitMs)
// — the pid-reuse backstop. A VERIFIED-LIVE holder under the ceiling is NEVER
// stolen, even if older than the old mtime-only threshold: nuking a slow-but-
// live writer's lock causes lost updates (#3711 / #500/#905/#1230 family).
try {
const stat = fs.statSync(lockPath);
if ((clock).now() - stat.mtimeMs > staleThresholdMs) {
const ageMs = clock.now() - stat.mtimeMs;
const holderLive = _stateHolderVerifiedLive(lockPath);
if (!holderLive || ageMs > deadmanCeilingMs) {
let removed = false;
try { fs.unlinkSync(lockPath); removed = true; } catch { /* swallow: bounded below */ }
if (removed) {
@@ -2891,4 +2951,12 @@ export = {
cmdStateMilestoneSwitch,
cmdSignalWaiting,
cmdSignalResume,
// Test seam (audit M1): inject a deterministic isPidAlive so the liveness-gated
// steal decision is exercised without real pids. Mirrors capability-lock.cts.
_setLockProbes(probes: Partial<{ isPidAlive: (pid: number) => boolean }>): void {
if (typeof probes.isPidAlive === 'function') _stateLockProbes.isPidAlive = probes.isPidAlive;
},
_resetLockProbes(): void {
_stateLockProbes.isPidAlive = _realIsPidAlive;
},
};