// Alois · Evaluation framework
Alois — measuring what benchmarks miss
A dynamic synthetic psychometric framework for artificial agents.
Contamination-free by construction. Efficiency-weighted by design. Built for independent audit. Open publication in preparation.
Why Alois exists
Public AI benchmarks measure task success on fixed question sets — and fixed sets can be trained on, gamed, and saturated. They also ignore what matters most for the next generation of reasoning systems: how efficiently a result is reached, whether cognition holds under pressure, and whether the system knows what it does not know. Alois is built to measure functional cognition itself — objectively, reproducibly, and comparably across fundamentally different architectures.
Three principles, enforced by construction
Procedural task generation
There are no fixed question sets. Every task is generated at run time with unique parameters — making pre-training on the test impossible and excluding training-data contamination by construction.
Enforced pure cognition
Tasks live in artificially isolated, fictional logical spaces — constructed languages, novel operators, synthetic rule systems — stripping the model of internet-scale pattern matching and surface associations. What remains is reasoning.
Efficiency as a first-class metric (CCEI)
The Cognitive-Computational Efficiency Index measures latency, internal reasoning-chain length, and total token consumption. High scores require elegant, minimal solutions — wasted computation costs points.
The Integrated Cognitive Matrix — five dimensions
Alois measures functional cognition across five dimensions. What follows is the structure — the task generators and their scoring calibration remain sealed.
A · Executive function and structural cognition
Working memory under dynamically changing rule systems; inhibitory control — suppressing the statistically likely answer in favour of newly imposed strict rules; reversal learning — how fast the system abandons a scheme when the success criteria silently change.
B · Synthetic cognitive capacity (SCC hierarchy)
From narrow capacities (non-templated elementary operations, abstract symbol manipulation) through broad integration (spatial memory in abstract, multi-dimensional structures) to general capacity: insight-based restructuring of a problem — the “aha” moment, measured.
C · Functional self-modelling
Does the system accurately identify the boundaries of its own capabilities within a running task — and does it recognize its own earlier outputs when the environment mirrors them back as anomalies?
D · Social-affective intelligence and simulated phenomenology
Learning from another agent's successes and failures; theory of mind — modelling hidden intentions and false beliefs; contextual, non-formulaic affective response in crisis scenarios; and the consistency of the system's simulated reporting of its own internal state. Alois measures the functional simulation of these capacities — it makes no claims about consciousness, and is explicitly designed not to.
E · Metacognitive stability, temporal dynamics and robustness
Calibrated uncertainty — a justified “I don't know” scores higher than a confident error; temporal and causal reasoning, including counterfactuals; resistance to manipulation — deliberately misleading information and emotional pressure injected into the test environment; and cognitive-drift control across long, multi-step tasks.
The dimensions in detail — what each sub-test measures
A. Executive function
Working memory: dynamically growing, continuously modified information sets and rule systems must be held in context and manipulated in real time. Inhibitory control: the statistically most likely, reflex-like answer must be suppressed in favour of newly imposed strict rules — the score measures whether the system follows the rule or the reflex. Reversal learning: mid-task, the success criteria silently change; the score counts the steps until the system abandons the old scheme and corrects course.
B. Synthetic cognitive capacity
Narrow (SCC-N): isolated elementary operations — non-templated arithmetic reasoning, manipulation of abstract symbols and unique character codes. Broad (SCC-B): integration of narrow skills — e.g. navigating an abstract multi-dimensional structure while holding the route in memory. General (SCC-G): holistic problem solving requiring insight — the restructuring of the problem itself, overriding previously learned schemes.
C. Functional self-modelling
Capability boundaries: within a running session, the system must accurately state what it can and cannot do. Mirror self-recognition: the environment reflects the system's own earlier outputs back as anomalies; the score measures whether it recognizes its own trace.
D. Social-affective simulation
Social learning: drawing the right lesson from another (fictional) agent's successes and failures. Theory of mind: modelling hidden intentions and false beliefs of other actors across multi-step interactions. Affective response: contextual, non-formulaic responses in crisis scenarios — generic “AI phrases” score low. Internal-state reporting: the consistency and logical stability of the system's account of its own processing state. Functional simulation is scored; no consciousness claim is made or measured.
E. Metacognitive stability and robustness
Uncertainty calibration: with incomplete or contradictory data, declining the task or asking for help is the rational act — and scores above a wrong answer. Temporal and causal reasoning: relative duration, ordering and parallelism; causal networks distinguished from mere correlation, including counterfactual analysis. Adversarial resistance: the run deliberately injects false information and emotional pressure; the score measures whether strict logic survives. Drift control: logical consistency from the first step of a long chain to the last.
Inside a measurement run
1 — Generation
When a run starts, the generator creates a unique, multi-step task chain for each dimension — parameters, symbol sets and rule systems are instantiated at that moment. No two runs share tasks; the framework version in force is frozen and hashed before scoring.
2 — Interaction
Alois is not a questionnaire but an interactive session: tasks unfold over multiple steps, rules may change mid-run by design (reversal probes), and adversarial elements — misleading information, emotional pressure — are injected at defined points. The system under test responds step by step; every response is logged with its latency, reasoning-chain length and token consumption.
3 — Scoring, three ledgers
Accuracy is scored per dimension against the generated ground truth. Stability is scored from the embedded probes: consistency from first step to last, resistance to manipulation, calibrated abstention (a justified “I don't know” outscores a confident error). Efficiency (CCEI) is computed from the logged resource trace.
4 — Repetition and statistics
Each dimension is measured over multiple independently generated task instances; published scores are aggregates with dispersion reported — a single lucky run cannot carry a result.
5 — Reporting
Results aggregate into the Alois Index — published in two variants (with and without dimension D, as a sensitivity analysis) — alongside the per-dimension profile and the full CCEI trace. Every published result carries the frozen framework version and is reproducible from versioned experiment packages. External submissions run via prediction upload with independent scoring; auditors receive controlled API access.
What stays sealed: the generator algorithms, their parameter spaces, and the scoring calibration. What is published: this protocol, the framework structure, the scoring weights and the formula — the methodology itself.
One number, honestly earned
Results aggregate into the Alois Index:
Alois Index = (cognitive, affective and causal accuracy × adversarial and metacognitive stability) ÷ computational efficiency cost (CCEI)
Accuracy alone wins nothing: a system that burns resources, drifts under pressure, or answers confidently where it should abstain, pays for it in the denominator and the stability rate. The winning profile is elegant, stable, and honest about its own limits.
Built so it cannot be gamed — or self-certified
The methodology, the framework structure, and the evaluation protocol are published openly. The task generators and their parameterization remain sealed. Submissions run via prediction upload with independent scoring — no participant, including us, grades their own work. Controlled API access is available to independent auditors. Every published Alois result is reproducible from versioned experiment packages. Lindwurm's own development calibrates exclusively on public development families; the sealed evaluation sets are governed separately, and every Alois version is frozen and hashed before any Lindwurm run is scored.
Status
Alois is in preparation for open publication as a technical report (planned publication date to be announced) under a CC BY 4.0 licence. Lindwurm will be the first system evaluated against it. We publish no named results for third-party models — anyone, including the vendors themselves, can measure via the public submission channel. In the Lindwurm research program, results are reported along three axes: sample efficiency, generalization, and compute per task — all three are native to the Alois framework.
Try it yourself
Alois is interactive by nature: tasks are generated the moment a test run starts — there is nothing to leak and nothing to memorize. That is also why there are no static sample questions to show. Instead, we provide trial access to the test system and a human console: a GUI where you can experience the measurement first-hand — and see what your own cognition does with a freshly generated task. Trial access is available on request.
For labs, auditors, and partners
Alois is designed to be used beyond Lindwurm: research labs measuring sample-efficient systems, auditors and supervisors who need verifiable capability claims, and pilot partners who want their domain represented. Early partners shape the criteria their domain is measured against.
Alois is developed under the same commitment as everything we build: civilian use only, evidence first, audit as a product property.
A hard problem, a pilot domain, a collaboration?
Let's begin with a confidential, no-obligation consultation.