Independent Evaluation & Audit Protocol

01Purpose and Scope

This protocol defines how an Independent Auditor may evaluate and verify the performance of the Lindwurm reasoning engine (the "System") on agreed public benchmarks, and issue a signed attestation of the results — without the Developer disclosing source code, model weights, or internal architecture.

Its purpose is to give funding bodies, reviewers, and investors objective, third-party-verified evidence of capability while fully protecting the Developer's intellectual property.

• In scope: measuring task performance on the benchmarks listed in Section 6, under controlled, leakage-resistant conditions.
• Out of scope: any disclosure of proprietary internals, and any certification of "consciousness" or "AGI" claims. The audit measures task performance only.

02Definitions

• Developer — the owner of the System and its intellectual property (Lile / Lindwurm program).
• Independent Auditor ("Auditor") — a qualified third party with no financial interest in the outcome, engaged to run the evaluation and issue the attestation.
• System — the Lindwurm reasoning engine as frozen for the audit (see Section 7.2).
• Benchmark — a publicly defined task suite used for evaluation (see Section 6).
• Held-out Test Set — benchmark tasks withheld from the Developer until after the System is frozen.
• Black-box Interface — an input/output boundary through which the System is exercised without exposing internals.
• Attestation — the Auditor's signed report of methodology and results (Section 10).

03Guiding Principles

• Independence: the Auditor controls the test data and the scoring; the Developer does not.
• Confidentiality: no source code, weights, or architecture are disclosed; the audit is strictly black-box.
• Integrity: explicit measures prevent data leakage and result manipulation (Section 9).
• Reproducibility within the audit: runs are logged and repeatable under the Auditor's supervision.
• Objectivity: scoring uses each benchmark's published, deterministic criteria (Section 8).
• Proportionate disclosure: the Auditor receives the minimum access required to score reliably.

04Parties and Independence

The Auditor should be an academic group, an accredited test laboratory, or a recognised machine-learning evaluation body with relevant expertise and no equity, revenue share, or advisory relationship tied to Lindwurm's success. The Auditor confirms this independence in writing. The engagement and its confidentiality are governed by the mutual non-disclosure agreement (NDA) referenced in Section 11.

Adversarial verification — including by competitors — is welcome through the public benchmarks and the test API. The signed attestation itself, however, must come from a party with no financial interest in the outcome in either direction. Any compensation, if agreed at all, is fixed and independent of the evaluation results.

05Disclosure Boundary

This boundary is the core of the arrangement: it enables objective scoring while protecting the Developer's IP.

06Evaluation Targets

Benchmarks and metrics are finalised with the Auditor before freezing. Indicative targets:

07Test Methodology

7.1 Data custody and anti-leakage

• The Auditor selects and holds the Held-out Test Set; the Developer must not access it before the System is frozen.
• For benchmarks with public answers, the Auditor may use a private re-split or newly generated tasks in the same format to prevent memorisation.
• The Developer attests, in writing, that the frozen System was not trained or tuned on the specific test tasks.

7.2 System freeze and sealing

• Before the test set is revealed, the Developer freezes the System build and provides a cryptographic hash (e.g., SHA-256) of the deployable artifact to the Auditor.
• The sealed build is used unchanged for the entire evaluation; any modification voids the run and requires re-sealing.

7.3 Execution modes (choose one)

1. Supervised run — the Auditor observes the System running on the Developer's hardware (in person or via screen-share), supplies the test inputs, and captures the outputs.
2. Sealed container — the Developer delivers the System as an encrypted/sealed container or binary that the Auditor runs in an isolated environment; internals remain non-inspectable by agreement.
3. Black-box API — the Developer hosts the System behind an API; the Auditor sends test inputs and records outputs, under agreed rate and format controls.

7.4 Run parameters

• Attempts: per benchmark rules (e.g., 2 predictions per test input for ARC).
• Compute/time budget: fixed and logged per task.
• Determinism: fixed random seeds where applicable; otherwise multiple runs are reported with variance.
• Isolation: no internet access or external API calls during scored runs (verified by the Auditor in modes 1–2).

7.5 Logging and chain of custody

• All inputs, outputs, timestamps, seeds, and resource usage are logged.
• The Auditor retains raw outputs and scoring artifacts; a chain-of-custody record is maintained.

08Scoring and Metric Definitions

• Exact-match accuracy: a task test output scores 1 if any allowed attempt matches the ground truth exactly (shape, values, and positions), otherwise 0. The final score is the mean over all test outputs.
• Sample efficiency: the number of demonstrations required to reach a target accuracy.
• Compute per task: wall-clock time, energy, or API cost per solved task.

Where a benchmark provides an official scorer, that scorer is used verbatim.

09Integrity and Anti-Gaming Controls

• Freeze-before-reveal (Section 7.2) prevents tuning to the test set.
• Auditor-controlled, unseen tasks prevent lookup tables and per-task hardcoding.
• No-internet during scored runs prevents external assistance.
• The Auditor may inject additional novel tasks to confirm genuine generalisation.
• Complete logs enable independent post-hoc review.

10Deliverables

• A signed Attestation Report stating: benchmarks and splits used, execution mode, run parameters, integrity measures, scores per benchmark with their definitions, and the sealed build hash.
• An access statement confirming what the Auditor did and did not receive — in particular, that source code was not disclosed.
• A publishable summary: the Developer may publish the scores, the methodology summary, and (with consent) the Auditor's identity. Everything concerning the System's internals remains confidential.

11Confidentiality, IP and Data Handling

• The engagement is governed by a mutual NDA; the Auditor treats all System materials as confidential.
• The Auditor may disclose only the agreed results and methodology.
• On completion, the Auditor returns or destroys any System artifacts and working copies of Developer data.
• All intellectual property in the System remains solely with the Developer; the audit conveys no licence.
• Publication rights and any embargo period are agreed in advance.

12Timeline and Fees

13Limitations and Disclaimers

• The audit certifies task performance on the specified benchmarks only. It does not certify general intelligence, consciousness, or fitness for any particular purpose.
• Results are valid for the frozen build and the specified test sets; any change requires a re-audit.
• Context self-reported by the Developer is identified as such in the report.