* feat(#1105): add external-job capability (SLURM scheduler-adapter producer half) The async external-job consumer half (#1165) shipped long ago: the core loop reads .planning/async-jobs/<job>.json manifests and treats a non-terminal one as the legal external_job_waiting half-state. The PRODUCER half (#1164) was the remaining unimplemented piece of #1105. This adds the producer as a default-off capability: - capabilities/external-job/ — capability.json (execute:wave:post -> executor, plan:post -> planner contributions, external_job.* config keys, default-off) + fragments teaching runtime-budget classification and externalization. - src/external-job.cts -> gsd-core/bin/lib/external-job.cjs — pure producer module: SLURM state -> manifest-status map (no guessing), manifest build/validate (versioned stability contract), sbatch/squeue/sacct parsers, and a fail-closed manifest writer (refuses a second non-terminal job for a plan_id already in flight; refuses to clobber a malformed manifest). fs/clock seams for deterministic tests. - scripts/slurm-adapter.cjs — operator CLI (submit/poll/show) wrapping bounded sbatch/squeue/sacct subprocesses; surfaces manifest commands for confirmation and never auto-runs them (trust boundary). - tests/external-job.test.cjs — 23 behavioral + fast-check property tests. - docs/reference/long-running-operations.md + docs/how-to/async-external-jobs.md. - CONTEXT.md glossary entry for the External-job Capability. - Regenerated capability-registry.cjs; pruned the now-stale test-file-count allowlist entry (external-job is at the 2-file cap). * chore(#1105): backfill PR number in changeset * fix(#1105): sync capability artifacts + update registry shape-pin tests gsd-test caught that adding the external-job capability requires its dependent artifacts regenerated and its registry-shape drift absorbed: - sync-manifest-versions: stamp 1.7.0-rc.2 into capability.json (was 1.0.0). - gen-capability-matrix --write: regenerate docs/reference/capability-matrix.md. - gen-inventory-manifest --write: regenerate docs/INVENTORY-MANIFEST.json. - check-gap-analysis-plan-post-e2e: plan:post now has 1 contribution (external-job planner fragment) instead of 0. - execute-wave-post-gate-pipeline-e2e: execute:wave:post now has 2 contributions (mempalace + external-job) instead of 1. * fix(#1105): regenerate capability-registry after version stamp sync-manifest-versions re-stamped external-job/capability.json from 1.0.0 to 1.7.0-rc.2 after the last registry regeneration, leaving the committed capability-registry.cjs stale (CI gen-capability-registry --check failed). gsd-test masked this because its setup runs the full 'npm run build' (which regenerates the registry); CI's 'npm test' pretest only runs build:lib.
This commit is contained in:
Tom Boucher
@@ -332,6 +332,7 @@
|
||||
"eval-command-router.cjs",
|
||||
"eval.cjs",
|
||||
"external-descriptor-trust.cjs",
|
||||
"external-job.cjs",
|
||||
"fallow-runner.cjs",
|
||||
"federated-config.cjs",
|
||||
"frontmatter.cjs",
|
||||
|
||||
115
docs/how-to/async-external-jobs.md
Normal file
115
docs/how-to/async-external-jobs.md
Normal file
@@ -0,0 +1,115 @@
|
||||
# Run a long-running job asynchronously with the SLURM adapter
|
||||
|
||||
> **How-To** (Diátaxis). When a GSD execute task is legitimately long-running
|
||||
> (HPC solver, model training, large simulation — over ~30–60 min), externalize
|
||||
> it instead of blocking the agent turn. This guide uses the default-off
|
||||
> `external-job` Capability's SLURM adapter (#1164 / #1105).
|
||||
|
||||
## When to use this
|
||||
|
||||
Use this path when a task is tagged `<runtime_budget>long_compute</runtime_budget>`
|
||||
by the planner. For `quick`, `medium`, and `unknown` budgets, run normally —
|
||||
see the [operation policy reference](../reference/long-running-operations.md).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- The `external-job` capability is enabled (`external_job.enabled: true` in
|
||||
`.planning/config.json`). It is **default-off**.
|
||||
- A SLURM cluster is reachable (`sbatch`, `squeue`, `sacct` on PATH).
|
||||
- You are inside a GSD project (a `.planning/` directory is present).
|
||||
|
||||
## 1. Submit the job
|
||||
|
||||
Run the adapter's `submit` subcommand with the `sbatch --parsable` invocation
|
||||
after `--`. Declare the artifacts the job must produce and the command that
|
||||
verifies them.
|
||||
|
||||
```bash
|
||||
node scripts/slurm-adapter.cjs submit \
|
||||
--plan 3.1 --phase 3 \
|
||||
--expected Artifacts/jobs/12345/result.h5,Artifacts/jobs/12345/metrics.json \
|
||||
--verify "python -m verify.py 12345" \
|
||||
--resume "/gsd:execute-phase 3" \
|
||||
-- sbatch --parsable --output=Artifacts/jobs/%j/out.log ./train.sh
|
||||
```
|
||||
|
||||
What happens:
|
||||
- `sbatch` runs with a **bounded** subprocess timeout
|
||||
(`GSD_SLURM_SUBMIT_TIMEOUT_MS`, default 30 s).
|
||||
- The `--parsable` output is parsed for the job id.
|
||||
- A versioned manifest is written to `.planning/async-jobs/<job_id>.json`.
|
||||
- The adapter **refuses** to create a second non-terminal manifest for a
|
||||
`plan_id` that already has one in flight (duplicate-execution guard).
|
||||
|
||||
The command prints the job id, the manifest path, and reminds you the state is
|
||||
`external_job_waiting` with `SUMMARY` deferred.
|
||||
|
||||
## 2. Commit the manifest and a handoff
|
||||
|
||||
The manifest is durable state — commit it:
|
||||
|
||||
```bash
|
||||
git add .planning/async-jobs/<job_id>.json
|
||||
git commit -m "chore: externalize plan 3.1 to SLURM job <job_id>"
|
||||
```
|
||||
|
||||
The executor then returns `external_job_waiting` and **does not** write
|
||||
`SUMMARY.md`. `execute-phase` safe-resume, `resume-project`, and `pause-work`
|
||||
all recognize this as a legal deferred state.
|
||||
|
||||
## 3. Poll the job
|
||||
|
||||
```bash
|
||||
node scripts/slurm-adapter.cjs poll --job 12345
|
||||
```
|
||||
|
||||
This queries `squeue` (falling back to `sacct` for completed jobs), maps the
|
||||
raw SLURM state onto the closed manifest enum, and updates the manifest. Output
|
||||
is a JSON line:
|
||||
|
||||
```json
|
||||
{"job_id":"12345","slurm_state":"COMPLETED","manifest_status":"completed-unverified","path":".planning/async-jobs/12345.json"}
|
||||
```
|
||||
|
||||
An unmapped SLURM state is **not guessed** — the adapter errors out and asks
|
||||
you to inspect manually.
|
||||
|
||||
## 4. Verify and close
|
||||
|
||||
When the status is `completed-unverified`, verify the output before closing the
|
||||
plan. **Manifest commands are untrusted** — surface and confirm them; never
|
||||
auto-run:
|
||||
|
||||
```bash
|
||||
node scripts/slurm-adapter.cjs show --job 12345
|
||||
```
|
||||
|
||||
`show` prints the status and lists `submit_command`, `verification_command`,
|
||||
and `resume_command` for explicit confirmation. After you run the verification
|
||||
command yourself and confirm the `expected_artifacts` exist, write `SUMMARY.md`
|
||||
and close the plan (`/gsd:execute-phase 3` reconciles and lifts the deferral).
|
||||
|
||||
## 5. Handle terminal failure
|
||||
|
||||
If `poll` reports `failed`, `cancelled`, or `timeout`, the manifest carries
|
||||
`terminal_details`. Recovery is a **user** action, never automatic:
|
||||
|
||||
- re-run reconciliation via the manifest's `resume_command`, or
|
||||
- abort, or
|
||||
- mark-and-skip (record the decision in the plan).
|
||||
|
||||
Resubmitting the compute is your call — the adapter never resubmits on its own.
|
||||
|
||||
## Notes
|
||||
|
||||
- **No fixed log paths.** Use per-job artifact dirs (`Artifacts/jobs/<jobid>/`);
|
||||
the `external_job.artifact_dir` config key is the root.
|
||||
- **No hardcoded cluster.** Account/partition/project layout stays in your
|
||||
`sbatch` invocation; the adapter and the manifest never assume it.
|
||||
- **Pluggable backend.** `external_job.backend` (default `slurm`) is the seam
|
||||
for future adapters; the manifest `backend` field is opaque to core.
|
||||
|
||||
## Related
|
||||
|
||||
- [Long-running operations reference](../reference/long-running-operations.md)
|
||||
- [Async-job manifest contract](../reference/planning-artifacts.md)
|
||||
@@ -44,7 +44,7 @@ Core package and are stamped with the package version at release (per
|
||||
ADR-1244 D6). They are not subject to the consent or integrity-pin flow applied
|
||||
to third-party capabilities.
|
||||
|
||||
### Feature capabilities (role: feature) — 17
|
||||
### Feature capabilities (role: feature) — 18
|
||||
|
||||
Feature capabilities extend what the loop does — contributing research,
|
||||
planning, execution, verification, or ship artefacts at the loop extension
|
||||
@@ -57,6 +57,7 @@ points.
|
||||
| `audit` | feature | full | `>=1.6.0` | — | — | first-party |
|
||||
| `code-review` | feature | full | `>=1.6.0` | `execute:post` | step | first-party |
|
||||
| `drift` | feature | full | `>=1.6.0` | `plan:pre`, `execute:wave:post` | gate | first-party |
|
||||
| `external-job` | feature | full | `>=1.7.0` | `plan:post`, `execute:wave:post` | contribution | first-party |
|
||||
| `gap-analysis` | feature | standard | `>=1.6.0` | `plan:post` | gate | first-party |
|
||||
| `graphify` | feature | full | `>=1.6.0` | — | — | first-party |
|
||||
| `intel` | feature | full | `>=1.6.0` | `plan:pre` | step | first-party |
|
||||
|
||||
102
docs/reference/long-running-operations.md
Normal file
102
docs/reference/long-running-operations.md
Normal file
@@ -0,0 +1,102 @@
|
||||
# Long-running operations and async external jobs
|
||||
|
||||
> **Reference** (Diátaxis). The operation-classification policy and the async
|
||||
> external-job contract that GSD executors use when a task legitimately exceeds
|
||||
> a child-agent timeout. The producer is the default-off `external-job`
|
||||
> Capability (#1164, part of #1105); the core loop consumes the manifest
|
||||
> (#1165 — `external_job_waiting`).
|
||||
|
||||
## 1. The problem
|
||||
|
||||
Heavy GSD phases (HPC solvers, model training, large simulations) can
|
||||
legitimately exceed short child-agent timeouts. Raising the timeout alone is
|
||||
not sufficient: it prevents legitimate subagents from being killed, but it
|
||||
also lets truly hung commands consume the whole agent budget. GSD must
|
||||
distinguish legitimate heavy work from suspicious hangs, keep safety nets
|
||||
finite, and avoid blocking an agent turn on hours-long compute.
|
||||
|
||||
## 2. Operation classification policy
|
||||
|
||||
Every executable task carries a runtime budget. Planners emit it via a
|
||||
`<runtime_budget>` element (taught by the `external-job` Capability's
|
||||
`plan:post` fragment); executors branch on it at `execute:wave:post`.
|
||||
|
||||
| Budget | Meaning | Execute behavior |
|
||||
|---|---|---|
|
||||
| `quick` | Under ~2 min | Run normally in the foreground. |
|
||||
| `medium` | ~2–30 min | Foreground, but with explicit progress expectations. |
|
||||
| `unknown` | Runtime not characterized | Run a **first-health check** and set a **soft-review deadline** before trusting the child timeout. Define a progress signal, an abort condition, and expected output. A truly hung command must surface before the child timeout is exhausted. |
|
||||
| `long_compute` | Over ~30–60 min | **Externalize** — submit an async external job, record durable state, and return `external_job_waiting`. Never block the agent turn. |
|
||||
|
||||
The classification is advisory metadata; the executor owns the decision at
|
||||
dispatch time. `unknown` is the safety-critical class: it is what catches a
|
||||
hung solver before it burns the budget, without making timeouts infinite.
|
||||
|
||||
## 3. The async external-job half-state
|
||||
|
||||
When an executor externalizes a `long_compute` task it enters a **legal
|
||||
deferred state**, not an illegal partial-plan state:
|
||||
|
||||
```
|
||||
input/code committed
|
||||
external job submitted
|
||||
.planning/async-jobs/<job>.json committed
|
||||
handoff committed
|
||||
SUMMARY.md deferred until verification
|
||||
```
|
||||
|
||||
`SUMMARY.md` is deferred until the job reaches a terminal state **and** its
|
||||
`expected_artifacts` are verified. Until then, the plan is
|
||||
`external_job_waiting`, and every resume/pause/dispatch path reconciles
|
||||
against the manifest — it never re-dispatches the plan (re-dispatching would
|
||||
duplicate the external job).
|
||||
|
||||
## 4. The manifest — a versioned stability contract
|
||||
|
||||
The manifest schema, status enum, trust boundary, matching rules, and the
|
||||
glob-safe matching probe are the **stability contract** documented in
|
||||
[`planning-artifacts.md`](./planning-artifacts.md#planningasync-jobsjobjson).
|
||||
The core loop depends only on the named fields and ignores any others; the
|
||||
`version` field is the evolution escape hatch. Producers MUST write the named
|
||||
fields and MAY add their own.
|
||||
|
||||
Status enum (closed, scheduler-agnostic — producers map backend states onto
|
||||
these):
|
||||
|
||||
| Status | Class | Resume action |
|
||||
|---|---|---|
|
||||
| `submitted`, `running` | non-terminal | Re-check; never re-dispatch. |
|
||||
| `completed-unverified` | finished, unverified | Verify `expected_artifacts` / run `verification_command`; on success write `SUMMARY.md` and close. |
|
||||
| `failed`, `cancelled`, `timeout` | terminal failure | Surface `terminal_details`; offer recovery (re-run reconciliation, abort, or mark-and-skip). Resubmitting compute is a user action, never automatic. |
|
||||
|
||||
## 5. Trust boundary
|
||||
|
||||
The manifest crosses a trust seam: a Capability (or anything that can write
|
||||
`.planning/`) produces it; the core loop consumes it. `submit_command`,
|
||||
`verification_command`, and `resume_command` are therefore **untrusted**. The
|
||||
core loop — and the `slurm-adapter` `show` subcommand — surface these commands
|
||||
for explicit operator confirmation; they are **never** auto-executed. Validate
|
||||
before trusting a manifest: recognized `version`, `plan_id` matches the plan
|
||||
under reconciliation, and `status` is one of the closed enum values. On a
|
||||
malformed manifest or multiple manifests for one `plan_id`, **fail closed**.
|
||||
|
||||
## 6. Scheduler pluggability
|
||||
|
||||
SLURM is the first backend. The design does not hardcode a cluster, account,
|
||||
partition, or project layout — per-job artifact directories
|
||||
(`external_job.artifact_dir`, default `Artifacts/jobs/<jobid>/`) avoid fixed
|
||||
log paths. The `backend` field on the manifest (opaque to core) and the
|
||||
`external_job.backend` config key are the pluggability seams for future
|
||||
backends (LSF, PBS, Kubernetes batch). The pure producer logic — SLURM
|
||||
state→manifest-status mapping, manifest build/validate, `sbatch`/`squeue`/
|
||||
`sacct` parsers, and the fail-closed writer — is backend-aware but lives
|
||||
behind a single module; a new backend adds a sibling state map and parser
|
||||
without touching core.
|
||||
|
||||
## 7. Related
|
||||
|
||||
- **How-To:** [`../how-to/async-external-jobs.md`](../how-to/async-external-jobs.md) — using the SLURM adapter.
|
||||
- **Contract:** [`planning-artifacts.md`](./planning-artifacts.md) — the manifest stability contract.
|
||||
- **Capability manifest:** [`../../capabilities/external-job/capability.json`](../../capabilities/external-job/capability.json).
|
||||
- **Pure module:** `gsd-core/src/external-job.cts` → `gsd-core/bin/lib/external-job.cjs`.
|
||||
- **Operator CLI:** `scripts/slurm-adapter.cjs`.
|
||||
Reference in New Issue
Block a user