Results & Measurement Methodology

01Purpose and Scope

This page is the public record of the Lindwurm test programme: it defines how measurements are produced and keeps the register of every published result. It complements the Independent Evaluation & Audit Protocol, which governs third-party attestation; this page governs what the Developer publishes about its own and externally scored runs.

• In scope: the measurement methodology for the benchmarks of the test programme, the reporting rules, and the results register itself.
• Out of scope: any disclosure of the System's internals, and any claim beyond task performance on the listed benchmarks.

02Verification Tiers

Every entry in the register is labelled with the tier that produced it. The tiers form an ascending ladder of external verification:

A result may be upgraded over time — e.g. a tier-1 score later confirmed on a leaderboard or by audit; the register then records each verification as its own entry.

03Measurement Methodology (tier 1)

3.1 Benchmarks and versions

• Runs use the public benchmark suites of the test programme — ConceptARC, ARC-AGI, RAVEN / I-RAVEN / PGM, Bongard-LOGO, KANDY / Kandinsky patterns — in their published form.
• The exact dataset version (release tag or repository commit) is recorded for every run.

3.2 Run parameters

• Attempts: per each benchmark's official rules (e.g. two predictions per test input for ARC).
• Determinism: fixed random seeds where applicable; otherwise multiple runs are reported with variance.
• Compute budget: a fixed per-task time/compute budget, logged for every task.
• Isolation: no internet access or external services during scored runs.

3.3 Scoring

• Where a benchmark provides an official scorer, that scorer is used verbatim.
• Exact-match accuracy: a test output scores 1 if any allowed attempt matches the ground truth exactly (shape, values, positions), otherwise 0; the final score is the mean over all test outputs.
• Sample efficiency (demonstrations needed to reach a target accuracy) and the efficiency metrics of Section 3.6 are recorded alongside accuracy.

3.4 System identification

• Every published result is tied to a frozen build identified by a cryptographic hash (SHA-256); the register shows the hash prefix.
• Any change to the build after a run makes the result non-comparable and requires a new entry.

3.5 Logging

• Inputs, outputs, timestamps, seeds, and resource usage are logged and retained for every scored run, so any published number can be re-derived from the raw artifacts.

3.6 Efficiency metrics

Raw accuracy says little about how much it cost to obtain: a system that solves a task with a fraction of the computation is categorically more capable. Every scored run therefore records, per task, three efficiency figures alongside the score:

• Runtime per task (Sec / task): the wall-clock time spent solving one task, in seconds — reported as the mean over the evaluated set.
• Floating-point operations (FLOPs / task): the total number of floating-point operations the system executed to solve one task, reported as an order-of-magnitude figure (e.g. 10¹² FLOPs per task).
• Energy per task (J / task): the programme's headline efficiency indicator — the electrical energy, in joules, consumed to solve a single task, measured at the hardware level.

Where a metric was not captured for a run (e.g. early baseline runs), the register shows "—"; the metric is then reported from the next run of that benchmark onward.

04Benchmark-Specific Protocols

Section 3 fixes the common rules; this section fixes how each suite of the test programme is run and scored. Two conventions apply throughout. First, tasks are presented in one of two encodings, always stated in the entry: a textual encoding (grids and scenes serialised as structured text) or a visual encoding (rendered images). Second, wherever a benchmark was designed as multiple choice, the programme prefers the stricter generation setting: the engine produces the answer itself instead of selecting from candidates — eliminating choice-set biases and the guessing baseline. Any departure from a benchmark's official protocol is disclosed in the entry's notes.

4.1 ARC-AGI-1 and ARC-AGI-2

• Datasets: the official ARC-AGI releases from the ARC Prize Foundation. Both generations are first-class targets of the programme: ARC-AGI-1 (the original corpus of few-shot grid-transformation tasks) and ARC-AGI-2 (the harder successor, designed to resist memorisation-heavy and brute-force approaches with more compositional, multi-step rules). Each run records the exact release tag; the two generations are always separate register entries.
• Task format: coloured grids up to 30×30 cells (ten colour values); each task supplies a few demonstration input→output pairs and one or more test inputs.
• Required output: the complete output grid for every test input — dimensions and every cell must be produced, never chosen from candidates.
• Attempts and scoring: the official rules allow two predictions per test input (pass@2); register entries state the attempts actually used — a single-attempt run (pass@1, as in the current baseline entries) is stricter than the official setting. A test input scores 1 only on an exact grid match; task and set scores follow the official scorer.
• Splits and leakage policy: the public training set is the only set used during development. The public evaluation sets have published solutions, so self-run results on them are always tier 1 on a frozen build; hidden-set evaluations (the organiser's semi-private/private sets, tier 2) and auditor-injected novel tasks (tier 3) provide the leak-free confirmation.
• Recorded: accuracy per set, per-task demonstration counts (the sample-efficiency profile), and compute per task.

4.2 ConceptARC

• Dataset: the public ConceptARC corpus — ARC-format tasks organised into sixteen concept groups (e.g. above/below, inside/outside, same/different), each with systematic variations of one underlying concept. The repository commit is recorded per run.
• Presentations: the corpus is run in both programme encodings as separate entries — Main Corpus (Text-only) and Minimal Tasks (Visual) — same tasks, same scorer, different input modality; the pair of scores measures modality robustness.
• Required output and scoring: as for ARC-AGI — exact grid match, official attempt rules, attempts stated per entry.
• Purpose in the programme: concept-level generalisation. Accuracy is additionally recorded per concept group, which exposes whether a concept was genuinely acquired or a single variation merely matched.

4.3 The Raven family — RAVEN, I-RAVEN, PGM, and the classic matrices

• Datasets: RAVEN (3×3 matrix-completion tasks across seven figure configurations), I-RAVEN (the impartial revision whose unbiased answer sets remove RAVEN's known context-blind shortcut), and PGM (procedurally generated matrices with defined generalisation regimes). The dataset version and, for PGM, the regime (neutral, interpolation, extrapolation, …) are recorded per run.
• Generation setting — no option list: the programme's differentiating rule. The engine never sees the candidate panels: it generates the missing panel directly from the matrix, and the generated answer is compared against the ground-truth panel. This removes answer-set bias entirely (the very weakness I-RAVEN was created to fix) and turns an eight-way choice with a 12.5% guessing baseline into open-ended generation with an effectively zero guessing baseline.
• Classic instruments: the programme also runs the original human test forms — Raven's Standard (SPM) and Advanced (APM) Progressive Matrices — as image-native runs under the same no-option-list rule (these are the current baseline entries). As copyrighted psychometric instruments, only aggregate scores are published; no test items are reproduced.
• Scoring: exact match of the generated panel (structure and attributes); the evaluated split or item set is stated per entry.
• Recorded: accuracy, per-configuration / per-regime breakdown, compute per task.

4.4 Bongard-LOGO

• Dataset: the public Bongard-LOGO corpus — Bongard-style concept-induction problems constructed from LOGO action programs. The repository commit and the evaluation split (free-form, basic, combinatorial abstraction, novel abstraction) are recorded per run.
• Task format: each problem gives a small support set of positive and negative example images of a hidden visual concept; held-out query images must be classified as fitting the concept or not.
• Required output and scoring: a binary decision per query image; accuracy over queries (chance level 50%), reported per split.
• Purpose in the programme: few-shot concept induction from contrastive examples — the most direct external measure of the programme's sample-efficiency metric.

4.5 KANDY / Kandinsky patterns

• Datasets: the KANDY benchmark suite (curricula of compositional classification tasks over Kandinsky-figure scenes) and the Kandinsky Patterns test sets. Repository commits and the curriculum variant are recorded per run.
• Task format: geometric scenes of simple objects (shape, colour, size, position); each task defines membership in a pattern through a compositional rule; supervision is sparse and tasks arrive as a sequence.
• Required output and scoring: binary membership per scene; accuracy per task and aggregated over the curriculum.
• Continual measurement: because tasks arrive sequentially, the programme additionally records backward performance — accuracy on earlier tasks re-measured after later ones. This is the external check on the homeostatic claim: new learning must not erase old competence.
• Anti-memorisation: both suites ship generators with a machine-checkable "model of truth", so fresh, unseen instances can be generated for verification runs — including by an independent auditor at tier 3.

05Reporting Rules

• Every completed scored run is published on the benchmarks listed above — including weak results. No cherry-picking: development runs are marked as such, but a completed measurement run cannot be withheld.
• Entries are immutable. A published entry is never edited or removed; corrections are appended as new entries that reference the original.
• Tier labelling is mandatory. Self-reported numbers carry tier 1 until an external verification (tier 2 or 3) exists.
• Deviations are disclosed. Any departure from this methodology (subset runs, changed budgets, benchmark updates) is stated in the entry's notes.

06Results Register

Newest entries first. Scores are stated as defined in Section 3.3, the efficiency columns as defined in Section 3.6; the build column shows the SHA-256 prefix of the sealed build that produced the run. A "—" cell means the metric was not recorded for that run.