* fix(#2615): document the effortSurface axis in the host-integration matrix #2481 added `effortSurface` as the ninth negotiated `hostIntegration` axis and wrote documentation-sourced values into 18 descriptors, but never touched `docs/reference/host-integration-capability-matrix.md`. The matrix that ADR-1239 designates the cited source of truth had zero occurrences of the axis: no entry in the axes legend, and no row in any of the per-runtime tables. `src/host-integration.cts` states "every value is documented or explicitly 'undocumented'" — for this axis that was false for every runtime. Adds the legend entry (the `argv` / `none` / `undocumented` vocabulary, plus why there is deliberately no config-file member) and an `effortSurface` row to all 19 per-runtime tables. Every citation is carried over from #2481's own commit message, where the values were sourced: - claude argv -- `claude --help` documents `--effort <level>` - opencode argv -- `opencode run --help` documents `--variant` - codex argv -- `model_reasoning_effort` is a config.toml key, not a dedicated flag, so the generic `-c key=value` override is the only argv route (still argv) - 15 hosts undocumented -- their docs state no reasoning setting kimi-code is the nineteenth section (added by #2603 after #2481) and is the one runtime with no declared value. Its row and a Documentation-gaps entry record why rather than inventing one: Kimi Code documents `/effort` (alias `/thinking`), but only as an INTERACTIVE slash command — `-m, --model` is the only model-adjacent argv. Neither vocabulary member is accurate (`none` would deny a mechanism the host has, `argv` would claim one it does not expose), so closing that gap needs a vocabulary decision, which is a negotiation change and not a documentation one. The absent value already degrades closed exactly as the sentinel does. The regression test derives its runtime list from the registry rather than hardcoding it, so a runtime added later fails until its matrix row exists — the ratchet whose absence let #2481 add an axis with nothing catching the missing docs. Closes #2615 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QT3ibz5qJuDuGqpTGRYVGf * docs(#2615): honest citations for the undocumented rows; fix the four stale 8-axis lists Two findings from the orthogonal review of the first commit. 1. The 15 `undocumented` rows shared byte-identical text — "searched the runtime's official docs (see Sources consulted above)" — which is weaker than this file's own convention ("no authoritative doc — searched: <url>") and, worse, implies a per-host targeted search that did not happen: each section's Sources-consulted list was gathered for OTHER axes and contains no CLI-reference or reasoning-effort source. The rows now say plainly what the finding is — an ABSENCE established by #2481's cross-host survey — and cite that survey rather than implying a URL was checked per host. 2. Four normative docs still described "the eight negotiated axes" and omitted effortSurface entirely. The worst of them is docs/how-to/add-or-update-a-host-integration.md — the maintainer's own guide for onboarding a host, whose Step 2 axis table would have a maintainer reproduce exactly the gap #2615 exists to close. Also fixed: docs/reference/host-integration-interface.md (which calls itself the normative reference and had no effortSurface row at all), docs/how-to/author-a-host-plugin.md, docs/registries/README.md ("**exactly** the eight … axes keys"), and CONTEXT.md's matching EoS-registry sentence. Deliberately NOT changed, because they are historical records rather than current contract: docs/whats-new-1.7.0.md and docs/FEATURES.md's 1.7.0 entry (effortSurface shipped in 1.8.0 via #2481 — rewriting them would falsify the release history), ADR-1239's pre-amendment body (already superseded by its own "Amendment (2026-07-21): effortSurface axis (#2481)"), and ADR-1016's "original eight axes", which refers to a different axis set entirely. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QT3ibz5qJuDuGqpTGRYVGf * chore(#2615): backfill changeset PR number (#2698) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
99
tests/fix-2615-effortsurface-matrix-parity.test.cjs
Normal file
99
tests/fix-2615-effortsurface-matrix-parity.test.cjs
Normal file
@@ -0,0 +1,99 @@
|
||||
// allow-test-rule: runtime-contract-is-the-product #2615 — the host-integration matrix
|
||||
// IS the cited source of truth for every descriptor axis (ADR-1239); asserting that a
|
||||
// shipped axis value appears there, and matches, is a contract assertion.
|
||||
|
||||
/**
|
||||
* Regression test for #2615 — `effortSurface` was a shipped `hostIntegration` axis
|
||||
* with NO presence in the matrix that is supposed to be its cited source of truth.
|
||||
*
|
||||
* #2481 added the axis and wrote docs-sourced values into 18 descriptors
|
||||
* (`claude`/`codex`/`opencode` -> `argv`, 15 others -> `undocumented`) but never
|
||||
* touched `docs/reference/host-integration-capability-matrix.md`: the axes legend
|
||||
* omitted it and not one per-runtime table carried a row. `src/host-integration.cts`
|
||||
* states "every value is documented or explicitly 'undocumented'" — for this axis
|
||||
* that was false for every runtime.
|
||||
*
|
||||
* This test is deliberately GENERIC rather than a hardcoded list: it derives the
|
||||
* runtimes from the registry, so a runtime added later fails here until its matrix
|
||||
* row exists. That is the ratchet the original gap needed — #2481 added an axis and
|
||||
* nothing caught the missing documentation.
|
||||
*/
|
||||
|
||||
'use strict';
|
||||
|
||||
process.env.GSD_TEST_MODE = '1';
|
||||
|
||||
const { describe, test } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const ROOT = path.join(__dirname, '..');
|
||||
const MATRIX = path.join(ROOT, 'docs', 'reference', 'host-integration-capability-matrix.md');
|
||||
const registry = require(path.join(ROOT, 'gsd-core', 'bin', 'lib', 'capability-registry.cjs'));
|
||||
|
||||
// Normalize CRLF so the row regexes hold on a Windows autocrlf checkout.
|
||||
const MATRIX_TEXT = fs.readFileSync(MATRIX, 'utf-8').replace(/\r\n/g, '\n');
|
||||
|
||||
/** Extract a `## <host>` section body, stopping at the next top-level host heading. */
|
||||
function section(host) {
|
||||
const start = MATRIX_TEXT.indexOf(`\n## ${host}\n`);
|
||||
if (start === -1) return null;
|
||||
const rest = MATRIX_TEXT.slice(start + 1);
|
||||
const end = rest.indexOf('\n## ');
|
||||
return end === -1 ? rest : rest.slice(0, end);
|
||||
}
|
||||
|
||||
/** Read the value cell of a `| <axis> | <value> | …` row. */
|
||||
function axisValue(body, axis) {
|
||||
const row = body.split(/\r?\n/).find((l) => l.startsWith(`| ${axis} |`));
|
||||
return row ? row.split('|')[2].trim() : null;
|
||||
}
|
||||
|
||||
const RUNTIMES = Object.keys(registry.runtimes).filter(
|
||||
(id) => registry.runtimes[id]?.runtime?.hostIntegration,
|
||||
);
|
||||
|
||||
describe('#2615: the matrix documents the effortSurface axis', () => {
|
||||
test('the axes legend defines effortSurface and its vocabulary', () => {
|
||||
const legendRow = MATRIX_TEXT.split(/\r?\n/).find((l) => l.startsWith('| `effortSurface` |'));
|
||||
assert.ok(legendRow, 'the axes legend must define effortSurface (#2615)');
|
||||
for (const member of ['`argv`', '`none`', '`undocumented`']) {
|
||||
assert.ok(legendRow.includes(member),
|
||||
`the legend must document the ${member} vocabulary member (#2615)`);
|
||||
}
|
||||
});
|
||||
|
||||
test('there is at least one runtime to check', () => {
|
||||
// Guards the loops below against silently asserting nothing.
|
||||
assert.ok(RUNTIMES.length >= 18, `expected the full runtime corpus, got ${RUNTIMES.length}`);
|
||||
});
|
||||
|
||||
for (const id of RUNTIMES) {
|
||||
describe(`runtime: ${id}`, () => {
|
||||
test('has a matrix section', () => {
|
||||
assert.ok(section(id), `${id}: every installed runtime needs a matrix section (ADR-1239)`);
|
||||
});
|
||||
|
||||
test('documents effortSurface, and the value matches the descriptor', () => {
|
||||
const body = section(id);
|
||||
assert.ok(body, `${id}: missing matrix section`);
|
||||
|
||||
const documented = axisValue(body, 'effortSurface');
|
||||
assert.ok(documented, `${id}: the matrix must carry an effortSurface row (#2615)`);
|
||||
|
||||
const declared = registry.runtimes[id].runtime.hostIntegration.effortSurface;
|
||||
if (declared === undefined) {
|
||||
// kimi-code declares no value: its mechanism (`/effort`) is interactive-only
|
||||
// and neither `argv` nor `none` describes it. The matrix must say so rather
|
||||
// than invent a value.
|
||||
assert.match(documented, /not declared/i,
|
||||
`${id}: an absent descriptor value must be documented as absent, not guessed (#2615)`);
|
||||
} else {
|
||||
assert.equal(documented, declared,
|
||||
`${id}: the matrix effortSurface value must match the shipped descriptor`);
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
});
|
||||
Reference in New Issue
Block a user