* feat(#1105): add external-job capability (SLURM scheduler-adapter producer half) The async external-job consumer half (#1165) shipped long ago: the core loop reads .planning/async-jobs/<job>.json manifests and treats a non-terminal one as the legal external_job_waiting half-state. The PRODUCER half (#1164) was the remaining unimplemented piece of #1105. This adds the producer as a default-off capability: - capabilities/external-job/ — capability.json (execute:wave:post -> executor, plan:post -> planner contributions, external_job.* config keys, default-off) + fragments teaching runtime-budget classification and externalization. - src/external-job.cts -> gsd-core/bin/lib/external-job.cjs — pure producer module: SLURM state -> manifest-status map (no guessing), manifest build/validate (versioned stability contract), sbatch/squeue/sacct parsers, and a fail-closed manifest writer (refuses a second non-terminal job for a plan_id already in flight; refuses to clobber a malformed manifest). fs/clock seams for deterministic tests. - scripts/slurm-adapter.cjs — operator CLI (submit/poll/show) wrapping bounded sbatch/squeue/sacct subprocesses; surfaces manifest commands for confirmation and never auto-runs them (trust boundary). - tests/external-job.test.cjs — 23 behavioral + fast-check property tests. - docs/reference/long-running-operations.md + docs/how-to/async-external-jobs.md. - CONTEXT.md glossary entry for the External-job Capability. - Regenerated capability-registry.cjs; pruned the now-stale test-file-count allowlist entry (external-job is at the 2-file cap). * chore(#1105): backfill PR number in changeset * fix(#1105): sync capability artifacts + update registry shape-pin tests gsd-test caught that adding the external-job capability requires its dependent artifacts regenerated and its registry-shape drift absorbed: - sync-manifest-versions: stamp 1.7.0-rc.2 into capability.json (was 1.0.0). - gen-capability-matrix --write: regenerate docs/reference/capability-matrix.md. - gen-inventory-manifest --write: regenerate docs/INVENTORY-MANIFEST.json. - check-gap-analysis-plan-post-e2e: plan:post now has 1 contribution (external-job planner fragment) instead of 0. - execute-wave-post-gate-pipeline-e2e: execute:wave:post now has 2 contributions (mempalace + external-job) instead of 1. * fix(#1105): regenerate capability-registry after version stamp sync-manifest-versions re-stamped external-job/capability.json from 1.0.0 to 1.7.0-rc.2 after the last registry regeneration, leaving the committed capability-registry.cjs stale (CI gen-capability-registry --check failed). gsd-test masked this because its setup runs the full 'npm run build' (which regenerates the registry); CI's 'npm test' pretest only runs build:lib.
This commit is contained in:
Tom Boucher
81
capabilities/external-job/capability.json
Normal file
81
capabilities/external-job/capability.json
Normal file
@@ -0,0 +1,81 @@
|
||||
{
|
||||
"id": "external-job",
|
||||
"role": "feature",
|
||||
"version": "1.7.0-rc.2",
|
||||
"title": "Async external-job scheduler adapter",
|
||||
"description": "Default-off producer of the async external-job manifest (#1164). At execute:wave:post an executor can externalize long-running compute (SLURM first, scheduler-pluggable), commit a .planning/async-jobs/<job>.json manifest, defer SUMMARY.md, and return external_job_waiting. The core loop (#1165) consumes the manifest; this capability is the only thing that writes it.",
|
||||
"tier": "full",
|
||||
"requires": [],
|
||||
"engines": {
|
||||
"gsd": ">=1.7.0"
|
||||
},
|
||||
"runtimeCompat": {
|
||||
"supported": [
|
||||
"*"
|
||||
],
|
||||
"unsupported": []
|
||||
},
|
||||
"skills": [],
|
||||
"agents": [],
|
||||
"hooks": [],
|
||||
"config": {
|
||||
"external_job.enabled": {
|
||||
"type": "boolean",
|
||||
"default": false,
|
||||
"description": "Master toggle for the async external-job producer capability. Default-off: the core loop consumes manifests whether or not this is on, but no manifest is ever written unless an executor opts in here."
|
||||
},
|
||||
"external_job.backend": {
|
||||
"type": "enum",
|
||||
"values": [
|
||||
"slurm"
|
||||
],
|
||||
"default": "slurm",
|
||||
"description": "Scheduler backend. SLURM is the first adapter; the field is the pluggability seam for future backends (LSF, PBS, Kubernetes batch). Core never interprets this value."
|
||||
},
|
||||
"external_job.artifact_dir": {
|
||||
"type": "string",
|
||||
"default": "Artifacts/jobs",
|
||||
"description": "Root for per-job artifact directories (e.g. Artifacts/jobs/<jobid>/). Avoids fixed log paths and hardcoding a cluster/project layout."
|
||||
},
|
||||
"external_job.submit_timeout_ms": {
|
||||
"type": "number",
|
||||
"default": 30000,
|
||||
"description": "Hard timeout (ms) for the scheduler submit subprocess (e.g. sbatch). Bounded per CLAUDE.md unbounded-subprocess policy."
|
||||
},
|
||||
"external_job.poll_timeout_ms": {
|
||||
"type": "number",
|
||||
"default": 15000,
|
||||
"description": "Hard timeout (ms) for the scheduler poll subprocess (squeue, with sacct fallback)."
|
||||
}
|
||||
},
|
||||
"steps": [],
|
||||
"contributions": [
|
||||
{
|
||||
"point": "execute:wave:post",
|
||||
"into": "executor",
|
||||
"fragment": {
|
||||
"path": "fragments/execute-wave-post.md"
|
||||
},
|
||||
"produces": [
|
||||
".planning/async-jobs/<job>.json"
|
||||
],
|
||||
"consumes": [
|
||||
"PLAN.md"
|
||||
],
|
||||
"when": "external_job.enabled",
|
||||
"onError": "skip"
|
||||
},
|
||||
{
|
||||
"point": "plan:post",
|
||||
"into": "planner",
|
||||
"fragment": {
|
||||
"path": "fragments/plan-post.md"
|
||||
},
|
||||
"produces": [],
|
||||
"consumes": [],
|
||||
"when": "external_job.enabled",
|
||||
"onError": "skip"
|
||||
}
|
||||
],
|
||||
"gates": []
|
||||
}
|
||||
36
capabilities/external-job/fragments/execute-wave-post.md
Normal file
36
capabilities/external-job/fragments/execute-wave-post.md
Normal file
@@ -0,0 +1,36 @@
|
||||
<!-- external-job capability — execute:wave:post fragment, injected into the executor (#1164). -->
|
||||
|
||||
## Externalize long-running compute (async external job)
|
||||
|
||||
If the current plan's task is tagged `<runtime_budget>long_compute</runtime_budget>`
|
||||
(see the plan-phase fragment), do **not** run it in the foreground — it would
|
||||
block the agent turn for hours. Instead externalize it and record a durable
|
||||
half-state:
|
||||
|
||||
1. **Classify the runtime.** `quick` (<2 min) and `medium` (<~30 min) run
|
||||
normally. `unknown` requires a first-health check and a soft-review deadline
|
||||
before consuming the child timeout. `long_compute` (>30–60 min) is
|
||||
externalized.
|
||||
2. **Submit via the scheduler adapter** (default `external_job.backend: slurm`):
|
||||
```bash
|
||||
node scripts/slurm-adapter.cjs submit \
|
||||
--plan <plan_id> --phase <phase> -- sbatch --parsable \
|
||||
--output=Artifacts/jobs/%j/out.log ./run.sh
|
||||
```
|
||||
The helper writes `.planning/async-jobs/<job>.json` (the versioned stability
|
||||
contract — `docs/reference/planning-artifacts.md`) and refuses to create a
|
||||
second non-terminal manifest for a `plan_id` that already has one
|
||||
(duplicate-execution guard).
|
||||
3. **Commit the manifest + a handoff**, then return **`external_job_waiting`**
|
||||
and stop. Do **not** write `SUMMARY.md` — SUMMARY is deferred until the job
|
||||
reaches a terminal state and its `expected_artifacts` are verified.
|
||||
4. **Resume path.** `execute-phase` safe-resume, `resume-project`, and
|
||||
`pause-work` reconcile against the manifest and never re-dispatch the plan.
|
||||
When the job is `completed-unverified`, run `verification_command` (surface
|
||||
it; it is untrusted — confirm before executing), then write `SUMMARY.md` and
|
||||
close the plan.
|
||||
|
||||
Manifest commands cross a trust seam: a Capability (or anything that can write
|
||||
`.planning/`) produces them; the core loop consumes them. Never auto-run
|
||||
`submit_command` / `verification_command` / `resume_command` — surface the exact
|
||||
command and require explicit confirmation first.
|
||||
24
capabilities/external-job/fragments/plan-post.md
Normal file
24
capabilities/external-job/fragments/plan-post.md
Normal file
@@ -0,0 +1,24 @@
|
||||
<!-- external-job capability — plan:post fragment, injected into the planner (#1164). -->
|
||||
|
||||
## Tag runtime budgets on long tasks
|
||||
|
||||
For every `<task>` likely to exceed ~2 minutes of real compute, emit a
|
||||
`<runtime_budget>` child element so execute can classify it:
|
||||
|
||||
- `<runtime_budget>quick</runtime_budget>` — under ~2 min; runs normally.
|
||||
- `<runtime_budget>medium</runtime_budget>` — ~2–30 min; foreground, but with
|
||||
progress expectations.
|
||||
- `<runtime_budget>unknown</runtime_budget>` — runtime not yet characterized;
|
||||
execute must run a first-health check and set a soft-review deadline before
|
||||
trusting the child timeout. Define a progress signal and an abort condition.
|
||||
- `<runtime_budget>long_compute</runtime_budget>` — legitimately over ~30–60 min
|
||||
(HPC solver, model training, large simulations). Execute must **externalize**
|
||||
this as an async external job (see the execute:wave:post fragment) rather than
|
||||
blocking the agent turn.
|
||||
|
||||
For any `long_compute` task, also declare the async contract the executor will
|
||||
need: the `submit_command`, the `expected_artifacts` the job must produce, and
|
||||
the `verification_command` that proves the output before the plan can close.
|
||||
Do not hardcode a cluster account, partition, or project path — the planner
|
||||
never knows the scheduler layout; it declares the contract, the executor's
|
||||
adapter fills the backend specifics.
|
||||
Reference in New Issue
Block a user