Tests
Purpose
What each category of test is there to prove, and which claims rest on it. The suite is run in full and again with every gated code disabled (TKIT_TIER_A_ONLY=1); at the last publish, 1177 tests passed in the first mode and 1177 in the second, at commit 6a13bde. Tests that need an optional code (TORAX, FreeGSNKE, ASCOT5, Aurora) or a downloaded file skip cleanly when it is absent.
The code now has a private hosted repository and the continuous-integration workflow runs on every change (OQ-3). Until it has passed, these counts come from runs made before each publish, not from CI.
Categories
| category | proves | examples | claims |
|---|---|---|---|
| unit | each component does what its interface says, in isolation | config parsing and scalar coercion; domain membership under jit/vmap; guard decisions; ensemble shapes; IMAS round trips; provenance validation; run tracking (local record, alerts, failed status, the Simvue mirror offline); smoke tests of ASCOT5 and Aurora |
C-024, C-031 |
| conservation | the discrete equations conserve what the physics conserves | single-step particle and energy balance; every step’s balance reconstructed from a full run’s trace; stored-energy identity over a run; TORAX’s per-step energy and particle balance on the ITER benchmark, with the edge flux computed independently | C-026, C-027, C-035, C-036, C-037, C-039 |
| physics | responses have the right sign and shape | more heating gives higher temperature; critical-gradient stiffness; Gaussian RF power, position and response; MAST-U equilibrium current, axis, completeness and resampling | C-017, C-018, C-020, C-021 |
| regression | known results stay known | 100 compiled toy steps; Newton converges quadratically and where Picard stalls; loop equals TORAX’s own driver; tracked command-line run end to end; reference reproduced; first-order time convergence; guard identities; TORAX conversion identities | C-001, C-003, C-005, C-006, C-007, C-009, C-010, C-014, C-015, C-040 |
| diff | derivatives exist and are right | gradients through 20 toy steps and three TORAX steps against finite differences; forward mode equals reverse mode | C-008, C-028 |
| tier_a | open code stands alone | every module labelled; every module imports with all gated packages blocked | – |
How a claim becomes tested
A claim is marked tested only if a named test asserts it, at a tolerance no looser than the one quoted. The test’s name is in the claim’s evidence column. Measured numbers that no test pins (timings, study results) are measured, and say which run produced them.
Honest limits
- Most tests were written by the same agent that wrote the code they test. The identity tests (wrapped equals unwrapped, loop equals TORAX’s own driver, recomputed equals reported) are the strongest, because they compare against something the agent did not write.
- The IMAS round trip is self-consistent by construction and missed K-007; see IMAS adapters; it is now fixed and covered by tests that compare against the source codes’ own values.
- No physicist has reviewed the physics tests yet (M1 gate).
Changelog
- 2026-09-30: first published version.
- 2026-09-30: TORAX balance tests in the conservation category (entry 8).
- 2026-10-01: run-tracking tests (entry 9).
Autonomous integration validation
The retained full normal and Tier-A-only suites cover every test file, with native per-file JUnit output (1095 tests pass with two intentional heavy-proof skips in each full normal and Tier-A-only suite; lint, typing and quick verification also passC-294). An independent audit checks that every source-manifest entry still matches the current checkout and that every expected test file has a corresponding record. Lint, formatting, typing and quick verification also complete successfully. These executions supersede the incomplete first validation attempt; they are separate from earlier incremental aggregates.
Quick verification deliberately omits expensive transport output-bound and local-robustness proofs. Passing suites do not close those scientific properties or award a human acceptance gate. Later source changes require affected checks and a new source binding; the recorded counts must not be silently applied to different code.
The later domain/balance and internal-branch diagnostic packages retain focused observer, analysis and corruption tests separately from those full-suite counts. Their independent auditors check canonical parent parity, retained rows, domain predicates and residual identities. They establish the reported diagnostic behavior (C-299, C-300, C-301), not nonlinear convergence or a causal endpoint error budget. No new full-suite execution is claimed for these diagnostic-only additions.