ffd5370464
* fix(#2903): use the command form that actually works in reader-facing docs Docs told readers to type the colon form, which no runtime registers -- 18 of 19 runtimes use slash-hyphen and the 19th uses shell-var -- so anyone copying an example got an unrecognized command. Swept 178 occurrences across 53 files, locale mirrors included so they do not re-diverge from English. The colon form is a source-authoring token, not a user-facing one: install-time converters key on it to produce the hyphen form runtimes actually register. So the sweep is scoped, and three things are deliberately left alone: - ADRs, which are a historical record; editing their prose falsifies what was written at the time. - The legacy release-notes archive, pending a maintainer decision on whether it follows the same historical carve-out. Excluding it keeps a later reversal additive rather than a revert. - Source artifacts under commands, workflows and agents, where the colon form is load-bearing. Rewriting those would break the installed-skill guarantee across every runtime -- the single largest hazard here. The plugin namespace form is a real, separate token and survives untouched. Adds a lint enforcing exactly that boundary, since the correct form genuinely differs by directory and nothing previously caught the drift. Also fixes a hardcoded colon form in the capability-matrix generator. The sweep alone would have left the generated matrix disagreeing with the template that produces it, so the fix is at the source and the output regenerated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#2903): stop the sweep misquoting source frontmatter Adversarial review caught three lines where the sweep rewrote a citation of the literal YAML name: key from a source command file. That key genuinely is the colon form -- this change's own carve-out logic says source-authoring tokens keep it -- so the docs ended up misquoting the real files. One of the three is an acceptance-checklist assertion, which the sweep turned into a false statement. Restored the three citations to match their sources verbatim, surgically: where a line carried both a name: citation and a real reader-facing slash command, only the citation reverted and the command stayed corrected. The guard needed the same distinction, or it would have flagged the restoration and reddened the build: a gsd:<cmd> token preceded by name: is a citation of a source token and is now permitted. The exemption is deliberately narrow -- a bare gsd:<cmd> anywhere else still fails -- with a test pinning that narrowness. Also makes the detection case-insensitive. Review found /GSD:next slipped through silently; no such casing exists in the tree today, so this closes a latent gap rather than fixing a live one. Swept the whole tree for further corrupted citations: none beyond the three. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#2903): retire the stale-next invariant and sweep next like every other command Maintainer decision on a genuine conflict between two contracts. Invariant #3054 banned the literal /gsd-next from user-facing docs because it named a retired workflow-advance command. But commands/gsd/next.md is a live command -- the state-aware smart-entry launcher -- and this issue requires docs to use the hyphen form every runtime actually registers. Both could not hold for this one command, so docs had been sidestepping the ban by keeping the colon form, which is exactly the defect this issue exists to remove. FEATURES.md already recorded the reassignment: the hyphen form "is not the retired workflow-advance command; it is reserved for the state-aware smart-entry launcher. Workflow advancement remains under /gsd-progress --next." With that reassignment the invariant's premise is obsolete and the guard now contradicts the documented command form, so it is retired with a comment recording why rather than deleted silently. next is now swept like every other command, and the earlier exemption added to the new guard is removed so nothing is special-cased. Four citations of the literal name: frontmatter key stay in colon form, because the source file really does carry name: gsd:next and a doc quoting it must reproduce it verbatim. Two of those lines were reworded to say which side is the frontmatter key and which is the slash command, since they previously conflated the two. Verified the retired scan would now genuinely fail against this tree -- the conflict was real and resolved, not dodged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(#2903): backfill changeset pr number Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
116 lines
4.3 KiB
Markdown
116 lines
4.3 KiB
Markdown
# Run a long-running job asynchronously with the SLURM adapter
|
||
|
||
> **How-To** (Diátaxis). When a GSD execute task is legitimately long-running
|
||
> (HPC solver, model training, large simulation — over ~30–60 min), externalize
|
||
> it instead of blocking the agent turn. This guide uses the default-off
|
||
> `external-job` Capability's SLURM adapter (#1164 / #1105).
|
||
|
||
## When to use this
|
||
|
||
Use this path when a task is tagged `<runtime_budget>long_compute</runtime_budget>`
|
||
by the planner. For `quick`, `medium`, and `unknown` budgets, run normally —
|
||
see the [operation policy reference](../reference/long-running-operations.md).
|
||
|
||
## Prerequisites
|
||
|
||
- The `external-job` capability is enabled (`external_job.enabled: true` in
|
||
`.planning/config.json`). It is **default-off**.
|
||
- A SLURM cluster is reachable (`sbatch`, `squeue`, `sacct` on PATH).
|
||
- You are inside a GSD project (a `.planning/` directory is present).
|
||
|
||
## 1. Submit the job
|
||
|
||
Run the adapter's `submit` subcommand with the `sbatch --parsable` invocation
|
||
after `--`. Declare the artifacts the job must produce and the command that
|
||
verifies them.
|
||
|
||
```bash
|
||
node scripts/slurm-adapter.cjs submit \
|
||
--plan 3.1 --phase 3 \
|
||
--expected Artifacts/jobs/12345/result.h5,Artifacts/jobs/12345/metrics.json \
|
||
--verify "python -m verify.py 12345" \
|
||
--resume "/gsd-execute-phase 3" \
|
||
-- sbatch --parsable --output=Artifacts/jobs/%j/out.log ./train.sh
|
||
```
|
||
|
||
What happens:
|
||
- `sbatch` runs with a **bounded** subprocess timeout
|
||
(`GSD_SLURM_SUBMIT_TIMEOUT_MS`, default 30 s).
|
||
- The `--parsable` output is parsed for the job id.
|
||
- A versioned manifest is written to `.planning/async-jobs/<job_id>.json`.
|
||
- The adapter **refuses** to create a second non-terminal manifest for a
|
||
`plan_id` that already has one in flight (duplicate-execution guard).
|
||
|
||
The command prints the job id, the manifest path, and reminds you the state is
|
||
`external_job_waiting` with `SUMMARY` deferred.
|
||
|
||
## 2. Commit the manifest and a handoff
|
||
|
||
The manifest is durable state — commit it:
|
||
|
||
```bash
|
||
git add .planning/async-jobs/<job_id>.json
|
||
git commit -m "chore: externalize plan 3.1 to SLURM job <job_id>"
|
||
```
|
||
|
||
The executor then returns `external_job_waiting` and **does not** write
|
||
`SUMMARY.md`. `execute-phase` safe-resume, `resume-project`, and `pause-work`
|
||
all recognize this as a legal deferred state.
|
||
|
||
## 3. Poll the job
|
||
|
||
```bash
|
||
node scripts/slurm-adapter.cjs poll --job 12345
|
||
```
|
||
|
||
This queries `squeue` (falling back to `sacct` for completed jobs), maps the
|
||
raw SLURM state onto the closed manifest enum, and updates the manifest. Output
|
||
is a JSON line:
|
||
|
||
```json
|
||
{"job_id":"12345","slurm_state":"COMPLETED","manifest_status":"completed-unverified","path":".planning/async-jobs/12345.json"}
|
||
```
|
||
|
||
An unmapped SLURM state is **not guessed** — the adapter errors out and asks
|
||
you to inspect manually.
|
||
|
||
## 4. Verify and close
|
||
|
||
When the status is `completed-unverified`, verify the output before closing the
|
||
plan. **Manifest commands are untrusted** — surface and confirm them; never
|
||
auto-run:
|
||
|
||
```bash
|
||
node scripts/slurm-adapter.cjs show --job 12345
|
||
```
|
||
|
||
`show` prints the status and lists `submit_command`, `verification_command`,
|
||
and `resume_command` for explicit confirmation. After you run the verification
|
||
command yourself and confirm the `expected_artifacts` exist, write `SUMMARY.md`
|
||
and close the plan (`/gsd-execute-phase 3` reconciles and lifts the deferral).
|
||
|
||
## 5. Handle terminal failure
|
||
|
||
If `poll` reports `failed`, `cancelled`, or `timeout`, the manifest carries
|
||
`terminal_details`. Recovery is a **user** action, never automatic:
|
||
|
||
- re-run reconciliation via the manifest's `resume_command`, or
|
||
- abort, or
|
||
- mark-and-skip (record the decision in the plan).
|
||
|
||
Resubmitting the compute is your call — the adapter never resubmits on its own.
|
||
|
||
## Notes
|
||
|
||
- **No fixed log paths.** Use per-job artifact dirs (`Artifacts/jobs/<jobid>/`);
|
||
the `external_job.artifact_dir` config key is the root.
|
||
- **No hardcoded cluster.** Account/partition/project layout stays in your
|
||
`sbatch` invocation; the adapter and the manifest never assume it.
|
||
- **Pluggable backend.** `external_job.backend` (default `slurm`) is the seam
|
||
for future adapters; the manifest `backend` field is opaque to core.
|
||
|
||
## Related
|
||
|
||
- [Long-running operations reference](../reference/long-running-operations.md)
|
||
- [Async-job manifest contract](../reference/planning-artifacts.md)
|