Contents
Where this sits
The focus is the Python reference, the first thing the CPU track builds (Phase 1). It is the authority for every exact stage: the browser core must match it line for line, and a CPU twin re-evaluates what the GPU track sends back.
Phase 0 is analysis, measurement spikes and a preregistration. No model code exists yet, on any track. This page says what each track will build, in which phase, and on what machine. Its sources are the read-only survey of four repositories and the literature (with its adversarial cross-check, whose corrections win), the design spec, the two Phase 0 pins (WordNet 3.0 and UD English EWT) and the existing systems’ own pages. Run times are planning figures, marked as estimates. The spikes S1–S7 were still running when it was written, and nothing here quotes them (see Explainer 8).
here runs in this container: 4 CPU cores, 15 GB RAM, no GPU, with PyPI and GitHub raw reachable. net needs a host that was blocked here in Phase 0; since the network was widened at the start of Phase 1, huggingface.co, the osf.io API, zenodo.org, arxiv.org and aclanthology.org are reachable, while osf.io’s file downloads, cdn.jsdelivr.net, download.pytorch.org, mattmahoney.net and the SimLex host are still blocked. gpu needs a GPU, on your hardware or in the cloud. exact means integers, rationals or bytes, identical in every engine. float means floating point, rounded at every step.
Numbers from the literature carry the survey’s grade. verified: read from the paper’s own page or its official repository. likely: from a search snippet of the primary source, which the survey could not open because arxiv.org and aclanthology.org were blocked here then. secondary: from someone else’s summary. estimate marks a planning cost, not a measurement.
- Track
- One place where code runs: the browser (JavaScript), the CPU (Python on this machine) or the GPU (Python on a machine with a graphics processor).
- Authority
- The implementation whose output decides a claim. For every exact stage it is the Python reference; the others must reproduce it. One exception: for compatibility with Owl Academy and BioChain-AI, their own JavaScript is the authority (claim L8).
- Trace line, parity
- A trace line is one recorded value, such as a token id, a rank list or a hash. Two implementations are at parity when every trace line matches byte for byte.
- float64
- 64-bit floating point. Each operation rounds, so two engines, or two orders of the same sum, can disagree in the last bits.
- Hyperboloid
- The sheet x0² − x1² − … − xd² = 1 with x0 > 0, which holds hyperbolic space in numbers (the Lorentz model). The Lorentz inner product of two points fixes their distance, so ranking by it ranks by distance.
- MAP
- Mean average precision. For each query node, rank every other node by distance and score how high its true neighbours come. MAP is the mean over queries; 1.0 means the true neighbours always rank first.
- FLOP
- One floating-point operation. Training cost is estimated as about 6·N·T FLOPs for N non-embedding parameters and T training tokens: about 2N per token for the forward pass and twice that for the backward pass (Kaplan et al. 2020, arXiv:2001.08361v1, §2.1) verified. The estimate leaves out the embedding parameters, the output layer’s compute and the attention cost that grows with context length.
Why three tracks
Two reasons. To compare what exactness buys against learned float models, at matched budgets. And to keep every result re-checkable in a browser, whoever ran it and wherever.
What exactness buys, and what it costs
This system bets on representation and certification, not on optimisation. Exact arithmetic is meant to give addresses that decode back without error, rank lists identical in Python, Node and Chromium, and metrics equal as exact fractions; claims L3 and L4 test this. Floats cannot promise it. Engram Mind’s Eye’s spike B found that float placement in H³ decodes cleanly to depth 12 (50/50) and fails from depth 14 (36/50 at 14, 0/50 at 18). A planning check found that float64 ranking inverts straight paths at depth 11–12; spike S1 measures this properly.
Exactness has its own price. The survey measured exact H³ points at depth 28 with numerators of about 138 bits and denominators of about 79. A planning check found the exact H² tree costs 1.5–2.4 times as many bits. Exactness moves Sala et al.’s precision problem into BigInt arithmetic. It does not remove it (Explainer 1).
Learned float models are the other side. A Lorentz table trained in float64, or a BERT that takes in the addresses, may beat the exact tree or lose to it. The only way to know is to measure both with the same tokenizer, data, seeds and metric code, at matched parameters, dimension, bits or evaluations. No single machine here can run all of that; the seven rules let three machines act as one experiment.
Hyperbolic geometry clearly helps on explicit hierarchies, such as WordNet at low dimension. On general NLP it helps modestly or not at all, and some of the reported gains are unchecked. HELM’s authors report gains of “up to 4%” for fully hyperbolic models, a figure the paper does not define: it matches only their largest single-benchmark gain, about 4 points on OpenbookQA at about 120M parameters, and at about a billion parameters the gains are +2.3 points on MMLU, +1.0 on ARC-Challenging and +1.0 on average, all near chance accuracy (Table 1 of arXiv:2505.24722v2) verified. The Hyperbolic Pre-Trained Language Model claims gains on GLUE, SuperGLUE and extractive QA tasks, but the survey could not see its numbers likely.
Hyperbolic Oracles’ own claim C1, “hyperbolic beats flat”, was refuted: 0 supported, 12 refuted and 1 inconclusive of 13 cells. On B4, hyper3 scored 0.325 against flat3’s 0.334. So every hyperbolic arm on every track has a matched Euclidean control, and the encoder claims planned for Phases 6 and 7 are expected to come out small or null.
Re-checkable by anyone
The browser cannot train a model. It can do something that matters more for trust: recompute. Every exact stage runs in plain JavaScript, on the cores of Hyperbolic Oracles and the orbital core, loaded unchanged. Engram Mind’s Eye showed the method: its browser core matches its Python reference on 849 of 849 parity lines, and the check runs in the page. The Lexicon goes one step further: tables and heads trained on the CPU or GPU are snapped to exact numbers and re-evaluated exactly in the browser. To trust a score, a reader need not trust the machine that trained the table.
The browser track
JavaScript only, with no CDN. Here it runs in Node 22.23.4 and in headless Chromium 153 (Chrome for Testing, driven by Playwright 1.63 from /opt/pw-browsers/chromium); elsewhere, in any browser. Its core, lexicon_core.js, was built in Phase 2: it reproduces every line of the Phase 1 traces, and with it L1, L3(f) and L4(a) were decided, all supported; it reproduced the reference’s L4(b) counts, which the reference decides (Explainer 10). Phase 3 added its chain section (the shard grower, the records, the manifests, the exact weave and the hardened verifier, with ECDSA through WebCrypto), and with it and the Python port L8–L11 were decided, all supported (Explainer 11). Phase 4 added the exact ranking of snapped learned tables (SnapArm, evaluateSnapped): in Node it re-ranked the 50 measured L6 tables and the exact arm and equalled the reference in every exact value, 0 of 118,200 per-query APs different (reported; Explainer 12).
| Job | What it does | Kind | Claim |
|---|---|---|---|
| Normaliser N_B | BERT-uncased rules on pinned Unicode 15.1 tables, never the runtime’s own functions. It fails closed: a code point unassigned in 15.1 turns its word into [UNK] and the offset is logged (stage A). | exact | L1 |
| WordPiece | Inference from a pinned vocabulary JSON. WordPiece splits each word, left to right, into the longest pieces in the vocabulary; pieces after the first carry a ## prefix. V0 “lex0” (8,192 entries) decides L1; V1, bert-base-uncased (30,522), pinned in Phase 1, is a second cell (stage B). | exact | L1 |
| Address tables and id codec | A word of the norm-5 tree for every WordNet noun synset and vocabulary token, and a bijective id: 59 bits for words of up to 25 letters, 64 bits up to 27 (stage D). | exact | L3 |
| Exact placement | Exact rational points of those words in H³ (t = 3/4) and H² (t = 11/13) (stage E). | exact | L3 |
| Exact ranking and metrics | Candidates sorted by the exact Lorentz inner product; MAP and mean rank as exact fractions, ties counted pessimistically. Mammals live in the page; noun samples in Node (ranking). | exact | L4, L5 |
| Shards | The synchronous shard grower, and the hardened verifier with ECDSA signatures checked through WebCrypto (Explainer 4). | exact | L8, L10, L11 |
| Manifests and the weave | RFC 6962 manifests with their inclusion and consistency proofs, and the exact norm-5 weave (manifest, weave). | exact | L9 |
| The X(5, q) walk | Walking a word through the LPS graph modulo q (Explainer 2). | exact | L7(b) |
| Exact re-evaluation | Every snapped table the CPU or GPU track produces, re-ranked exactly. | exact | L4(c) |
| PolyStepH heads | Gradient-free tuning of heads of about 10–300 parameters, such as the hierarchy head’s weight, threshold and temperature. | float | later |
| Distance attention | Inference only, for tiny exported models of up to about 1M parameters: softmax weights of minus the distance, then a Lorentz centroid. | float | demos |
| The 2D and 3D pages | The exact H² tree in the Poincaré disk and the exact H³ tree in the ball (Phase 5). | view | — |
It never trains embeddings or encoders. transformers.js runs BERT models in a browser through ONNX Runtime, but it does not train verified. A BERT-Tiny ONNX latency baseline comes later, as a vendored, hash-pinned copy, because cdn.jsdelivr.net is blocked here (huggingface.co was too, until the network was widened in Phase 1).
Why PolyStepH stays small
PolyStepH is Hyperbolic Oracles’ gradient-free optimizer on products of hyperbolic and flat factors (Explainer 1). It needs no gradients, so it runs in plain JavaScript; the survey measured about 2,900 evaluations per second in Node. Its largest model so far had 48 parameters (B4, 2×105 evaluations), and there it did no better than flat. Gradient-free methods in the literature, such as MeZO verified, fine-tune pretrained models; none pretrains from scratch. So the browser tunes heads of tens to a few hundred parameters. A candidate claim for Phase 6, L15, would compare PolyStep with MeZO-style zeroth-order SGD on a fixed head at equal evaluations.
Node 22.23.4 ships Unicode 17.0 (ICU 78.3); Python 3.13.18’s unicodedata is 15.1.0. Code points assigned after 15.1 are known to Node and not to Python, so the two can disagree on their category, case or decomposition. The normaliser therefore reads tables exported from unicodedata 15.1.0 and pinned by SHA-256, and its JavaScript never calls String.prototype.normalize, toLowerCase or \p{} classes. The platform’s own functions are kept as a control that must fail L1. Spike S6 lists where the three runtimes disagree; this page does not quote it.
The CPU Python track
Python 3.13.18 on this machine: 4 cores, 15 GB RAM, 22 GB of free disk. It is the authority for every exact stage, and the place where float64 learning happens until a GPU is in play.
What it builds
- The reference (Phase 1 on): normaliser, WordPiece and the V0 trainer, the WordNet reader, addresses and id codec, the exact H² tree, exact ranking, the flat arms, the LPS overlay; in Phase 3 the shard grower, manifests, records, attestation and the hardened verifier (
chain.py,records.py,manifest.py,attest.py,verify.py). Phase 1 decided L2, L3 (Python side), L5 and L7, all four supported (Explainer 9). In Phase 2 it was the authority the browser core was compared with for L1, L3(f) and L4(a), and it decided L4(b) itself, all supported (Explainer 10). In Phase 3 it was the authority for L8(d), L9 and L10’s verdicts, which the browser core equalled, while Owl Academy’s own code was the authority for L8(a, b); L8–L11 are all supported (Explainer 11). See reference/README.md for the modules and the authority rule. - The static-embedding trainer (Phase 4): numpy float64, in the Lorentz model (Nickel & Kiela’s Riemannian SGD) and in flat space (plain SGD), at d ∈ {2, 3, 5, 10, 50}. A Riemannian step projects the gradient onto the hyperboloid’s tangent plane and walks back onto the sheet with the exponential map. Built in Phase 4 (
embed.py, its random stream a numba copy of the shared generator checked againsthpstep.prng; snapping and exact ranking inlearned.py), it decided L6 and L4(c), both supported, L6(c) against its registered prediction (baseline B1; Explainer 12). - Tokenizer metrics: fertility, bytes per token, Rényi efficiency and the
[UNK]rate, used as gates, not goals (Stage 0). - Brown clusters, later.
- Small MLMs of 1–10M parameters (Phase 6, after the Stage-1 gate). A masked language model (MLM) is trained the way BERT is: hide some tokens and predict them.
What it has
| Package | Status here | Used for |
|---|---|---|
| numpy 2.5.3 | present | the exact helpers, the LPS overlay, the static-embedding trainer |
| cryptography 50.0.1 | present | ECDSA P-256 verification for L10 |
| PyYAML | present | — |
| pytest | from PyPI in build sessions | the reference’s tests |
| tokenization-scorer, tokenizers | from PyPI in build sessions | tokenizer metrics; agreement with the Hugging Face tokenizer, reported only |
| torch, geoopt | only if decision D9’s cross-check is wanted | PyPI has torch 2.15.0, but its Linux wheel brings the CUDA libraries, and download.pytorch.org is blocked |
That is why decision D9 recommends a numpy float64 trainer as the authority: it is deterministic and needs nothing else. If torch installs, geoopt’s Riemannian optimizers become a reported cross-check.
Data it can reach
- WordNet 3.0, from nltk_data on raw.githubusercontent.com (10,775,600 bytes), pinned in Phase 0: 82,115 noun synsets, 13,767 verbs, 18,156 adjectives and 3,621 adverbs; 743,241 hypernym-closure edges among nouns; the mammal closure, 1,182 nodes and 6,542 edges. C0, one line per noun synset (its lemmas and gloss), is 148 shards of at most 60,000 bytes.
- UD English EWT at commit
4a4d77f(GitHub, CC BY-SA 4.0): 12,544 / 2,001 / 2,077 sentences in train, dev and test. Its text is not committed. - BLiMP (GitHub, CC BY): 67 paradigms of 1,000 sentence pairs each.
Reachable since the network was widened at the start of Phase 1: Hugging Face datasets and models (BabyLM among them), the osf.io API and zenodo.org. Still blocked: osf.io’s file downloads, which redirect to a blocked object store, mattmahoney.net (enwik8) and the SimLex host (chapter 7).
What it costs
| Job | Size | Planning figure |
|---|---|---|
| L6, one run | the mammal hierarchy, about 1,180 nodes, numpy float64 | about 1 minute estimate |
| L6, all runs | about 290, tuning included | about 1–5 h; roughly 1 h spread over 4 cores estimate |
| Stage 1 on nouns | 1,000 sampled queries per arm | minutes per arm estimate |
| L7, dense eigenvalues | 12,180 vertices (q = 29) | minutes; spike S5 times it estimate |
| Phase 6 MLM | N = 5M parameters, T ≈ 1.3×108 tokens: ≈ 3.9×1015 FLOPs | about 11–22 h on 4 cores at an effective 50–100 GFLOP/s estimate |
None of these was measured when this page was written. Phase 1 has since timed two on this container (“~” timings): the L5 arms on the 1,000 noun queries took 5.8–18.8 s each in Python (S3 per permutation), and L7 at q = 29 took 45.3 s, 43.9 s of it for the eigenvalues (Explainer 9). Phase 4 timed L6, four runs at a time: from 30 s a run (Euclidean, d = 2) to 350 s (Lorentz, d = 50), about 51 s for Lorentz at d = 3; registered tuning, 240 runs, took 6,089 s and the decisive run, 100 trained tables with the floors, the exact arm and the controls, 3,051 s, about 2.5 h in all, and the dry run of the same size another 8,700 s (Explainer 12). For Phase 6, a calibration spike measures real throughput first, and the decisive strict-small runs are better made on your machine.
The GPU Python track
Your own hardware or a cloud GPU, with Hugging Face and OSF reachable. It starts in Phase 7, and only after every Stage-1 claim has been decided and reviewed with you. Nothing on it is registered yet: each claim comes by a dated amendment before any code.
What it builds
| Run | What it is | Compared with |
|---|---|---|
| BERT-mini and BERT-small with the hierarchy factor | BERT-mini has 4 layers of width 256 (L4/H256); BERT-small has 4 of width 512 (L4/H512). The hierarchy factor feeds each token’s address into the encoder as an extra input (the encoder). | Four controls at matched parameters: shuffled addresses (the same tree, words permuted), content-hash addresses (hash-like words), no factor, and a byte-level encoder that reads bytes with no tokenizer |
| BabyLM strict | Pretraining on at most 100M words | The BabyLM 2024 baselines verified (Stage 3) |
| GLUE | The pinned BabyLM evaluation-pipeline-2025 subset, plus GLUE dev | BERT-Tiny 64.2 to BERT-Medium 73.5, ELECTRA-Small 77.4 verified (B4) |
| MTEB(eng, v2) subset; SentEval | The cheap part of MTEB (similarity, classification, pair classification, a hierarchical clustering task; no retrieval), and SentEval’s TreeDepth and TopConst probes | — |
| HiT on MiniLM | Hierarchy Transformers verified: a Poincaré head on a small encoder, here all-MiniLM-L12-v2 (about 33M parameters; 33.4M on its Hugging Face model page), one of the three bases the paper re-trains (Sec. 4.3 of arXiv:2401.11374v4). The Hugging Face card of HiT’s L12 checkpoint names it; the repository README does not | A baseline for the Lexicon’s own hierarchy head (B3) |
| HyperCore arm | An exploratory, fully Lorentz encoder through HyperCore (MIT), which pins torch 2.6 with CUDA 12.4 | Exploratory only (L-a) |
Two anchors to plan by
ELECTRA-Small, 14M parameters
About 4 days on one V100 for a full run; decent after about 200k steps, roughly 10 hours. GLUE test 77.4. verified
BERT-small, 29M parameters
On 1.3×109 tokens: ≈ 1.0×1017 FLOPs, taking N as its 12.6M non-embedding parameters as Kaplan et al. do (≈ 2.3×1017 if all 29M parameters, embeddings included, are counted), roughly 0.5–2 h on one A100, worked out from the larger figure. estimate
A decision for you: where this track runs. The recommendation is your own GPU, or a rented single A100 or L4, from Phase 7 and only after the Stage-1 gate. Results come back as LBENCH/1 and LMODEL/1 receipts and are verified here, with a CPU twin re-evaluation (chapter 8).
The first encoder claim planned, L12 in Phase 6, pits the address factor against shuffled addresses and against no factor, at matched parameters and tokens, on BLiMP. It predicts an equivalence band of ±1 point: a null result. The GPU track repeats such comparisons at BERT-mini and BERT-small size, where published baselines exist. It is not where a hyperbolic win is expected.
Seven rules for comparable results
A number from the browser, one from the CPU and one from a GPU can stand side by side only if they share everything except the thing being compared. These seven rules fix the rest.
- One tokenizer. A single LVOCAB/1 record names it: the vocabulary’s SHA-256, the hash of the N_B specification, the hash of the Unicode 15.1 tables, the algorithm id and the special tokens. Every track loads the same record. L1 tests that Python, Node and Chromium agree byte for byte on the normalised text, the token strings, the ids, the byte offsets and the SHA-256 of the id stream.
- One data identity. Every corpus and split is named by its LCORPUS/1 root. That root is a Merkle root: one 32-byte hash at the top of a tree of hashes over every shard record (RFC 6962). Change any shard and the root changes, and a short proof shows that a given shard is included. Splits and MLM masks are drawn from one shared generator,
Rng: xoshiro128** seeded by splitmix32, the same generator inhyperbolic_core.jsand in Python’shpstep/prng.py. So every track masks identically, and anyone can replay the draws. Gated evaluation sets appear by hash only. - One seed scheme. Every random draw takes its seed from one table (below).
- One metric implementation. Python is the authority. For exact metrics, the JavaScript reproduces its traces line for line. Float metrics carry registered tolerances and a hash of every item’s output.
- Matched budgets. Gradient-free runs are matched on evaluations. Gradient runs record parameters, tokens seen, FLOPs (reported the ELUE way verified), hardware, library versions, thread count and wall-clock time. Exact representations are compared at matched dimension, and also at matched bits, in the sense of Sala et al.’s precision–dimension trade-off: their combinatorial embedding reaches WordNet MAP 0.989 in 2 dimensions, but needs about 500 bits verified (from the abstract, confirmed by the cross-check).
- Pinned scorers. Masked LMs are scored on BLiMP by PLL-word-l2r (Kauf & Ivanova 2023 likely) at a pinned babylm/evaluation-pipeline-2025 commit. A pseudo-log-likelihood (PLL) masks each token in turn and adds up the log-probabilities the model gives the hidden tokens. Plain PLL inflates the scores of words split into several pieces; the word-l2r variant also masks the rest of the word to the right. The GLUE fine-tuning recipe is fixed before any decisive run. The pin is needed because the numbers disagree: the 2024 pipeline gives LTG-BERT 60.6 on BLiMP strict-small verified, while figures near 80 for LTG-BERT and ELC-BERT (Table 1 of arXiv:2410.24159v2) verified come from 2023-era papers under a different pipeline. The 2024 README itself flags its strict-small LTG-BERT score as lower than expected, possibly from too few training epochs.
- Cross-track anchors. The browser exactly re-evaluates every table and head trained on the CPU or GPU. A fixed BLiMP subsample, 67 × 100 pairs drawn with Rng(1000+i), is scored on both CPU and GPU within a registered tolerance. A CPU twin re-evaluates each GPU checkpoint. Receipts are classed reproducible or attested (chapter 8).
The seed scheme
| Use | Seeds |
|---|---|
| Registered instances | 1000+i |
| Secondary draws | 2000+i |
| Dry runs, before any decisive run | the registered seed +3000 |
| Tuning | 9000–9023 |
| Spikes (data disjoint from every instance) | 90000+i |
| Bootstrap intervals | 10,000 resamples, seed 7 |
| Training | 0–4 |
hyperbolic_core.js.The box runs Hyperbolic Oracles’ own Rng, loaded unchanged. The Python reference uses the same generator, so the same seed gives the same 32-bit draws there. The masked sentence is a toy, not the masking rule, which Phase 6 registers; it shows only that a seed fixes the draws wherever they are made.
Where each stage runs
Some hosts are still blocked here, and there is no GPU. Stages 0 and 1 run here in full. Since the network was widened at the start of Phase 1, Stage 2 runs here too, apart from files on hosts still blocked, such as the SimLex host; Stage 3’s decisive runs take BabyLM from Hugging Face (osf.io’s file downloads stay blocked) and run here or on your machine; Stage 4 waits for a GPU.
| Stage and job | Browser | CPU | GPU | Phase |
|---|---|---|---|---|
| Stage 0 · L1, the front end with V0 “lex0” (8,192 entries) | here Node and Chromium | here authority | — | 1–2 |
| Stage 0 · L1’s second cell, V1 bert-base-uncased (30,522), pinned in Phase 1 from huggingface.co | here Node and Chromium | here authority | — | 1–2 |
| Stage 0 · L2, the engram/1 collision share on WordNet lemmas | here Node | here | — | 1 |
| Stage 0 · tokenizer descriptors | — | here | — | 1–2 |
| Stage 1 · L3–L5, addresses, exact points, exact ranking | here mammals in seconds; noun samples in Node | here 1,000 noun queries, minutes per arm | — | 1–2 |
| Stage 1 · L6, learned static embeddings; L4(c) | here exact re-evaluation of snapped tables | here numpy trainer | — | 4 |
| L7, the X(5, q) overlay | here the walk | here the graph and its eigenvalues | — | 1 |
| L8–L11, shards, manifests, attestation, verification cost | here Node, WebCrypto | here Python, cryptography | — | 3 |
| Stage 2 · HiT-protocol splits rebuilt from WordNet | — | here | — | 4, by amendment |
| Stage 2 · HyperLex, hypernymysuite, SimLex, WordSim, MEN | — | net most hosts | — | 4, by amendment |
| Stage 3 · dry runs on WordNet glosses and UD EWT | — | here | — | 6 |
| Stage 3 · 1–10M MLMs on BabyLM strict-small (10M words), scored on BLiMP | — | here BabyLM from Hugging Face, or your machine | — | 6 |
| Stage 3 · BabyLM strict (100M words) | — | — | gpu net | 7 |
| Stage 4 · GLUE, the MTEB subset, SentEval, HiT on MiniLM, HyperCore | — | — | gpu net | 7 |
| Checking GPU results: CPU twin, BLiMP subsample, exact re-ranking | here snapped tables and heads | here the twin | gpu the original run | 7 |
| Browser latency against a BERT-Tiny ONNX baseline | net once, for a vendored, hash-pinned copy | — | — | later |
L2 sits in Stage 0 for a reason. engram/1, the key inherited from Engram Mind’s Eye, keeps only the parity of each letter’s count (GF(2) rank 26). So LISTEN = SILENT, and numbers and words with every letter an even number of times land on the zero key. Engram’s own E9 found Hamming recall at chance after a one-letter substitution (Explainer 1). L2 measures how often keys collide on real lemmas, before anything treats the key as more than a checksum.
What the hosts blocked in Phase 0 hold
The first four rows are reachable since the network was widened at the start of Phase 1, osf.io through its API only: its file downloads redirect to a blocked object store, and BabyLM is also on Hugging Face. The last three rows are still blocked.
| Host | What it holds | What waits for it |
|---|---|---|
| huggingface.co | The V1 vocabulary, HiT and MiniLM, BERT-Tiny ONNX, the GLUE datasets | L1’s V1 cell (pinned in Phase 1); Stage 4; HiT |
| osf.io | The BabyLM corpora | Stage 3’s decisive runs, which now take BabyLM from Hugging Face |
| zenodo.org | HiT’s datasets | Comparison on HiT’s own splits (until then, splits are rebuilt from WordNet and labelled indicative) |
| arxiv.org, aclanthology.org | The primary papers | Turning likely numbers into verified ones |
| cdn.jsdelivr.net | Hosted builds of browser libraries | The transformers.js baseline, which comes as a vendored, hash-pinned copy instead |
| download.pytorch.org | CPU-only torch wheels | The optional geoopt cross-check |
| mattmahoney.net, the SimLex host | enwik8 and text8; SimLex-999 | Bits-per-byte checks, later; a Stage-2 relatedness control |
The spec recommended widening the network before Phase 4, to huggingface.co, osf.io, zenodo.org, arxiv.org and aclanthology.org. It was widened at the start of Phase 1: huggingface.co (with its large-file store), the osf.io API, zenodo.org, arxiv.org and aclanthology.org are reachable, which let V1 be pinned; osf.io’s file downloads still redirect to a blocked object store. Stages 0 and 1 run on WordNet, UD EWT and BLiMP from GitHub. The full list of decisions is on the hub and in the README.
Receipts: reproducible or attested
A result here is not a number in a log. It is a record that says exactly what was measured and how, so that someone who did not run it can check it.
LBENCH/1, the benchmark receipt
| Field | What it holds |
|---|---|
| claimId | The claim the result belongs to: one of L1–L11, or one added by amendment |
| subjectSha | The SHA-256 of what was evaluated: an address map, a table or a model |
| evalSetSha | The evaluation set’s hash; gated sets appear by hash only |
| scorer@version | For example PLL-word-l2r at the pinned pipeline commit |
| seeds | From the one seed scheme |
| per-item output hash | A hash over every item’s output, so two runs can be compared item by item |
| metricsSha | The hash of the metric results |
| hardware, libraries | The machine and the library versions that produced it |
| class | reproducible or attested |
| twinSha | The output hash of the twin run |
A twin, here, is a second run of the same evaluation in another engine or on another machine: the browser beside the Python reference, or the CPU beside a GPU. (It is not Hyperbolic Oracles’ twin system, two tri-channels in H³ on one clock.) Records are SHA-256 hashed and can be signed in BioChain-AI’s ATTEST/1 format, with ECDSA P-256 (Explainer 4).
reproducible
The twin’s output is bit-identical. Exact stages are built to land here: the browser recomputes them from the same inputs and must match the reference line for line.
attested
A float result from a single run. The receipt says who ran it, on what machine and with which libraries, and holds every item’s output hash. It does not prove the number is right, and it is labelled so.
How a GPU result comes home
- The run, on your GPU, writes an LBENCH/1 receipt and an LMODEL/1 record. LMODEL/1 is a per-file hash manifest of the model in the style of sigstore’s model-transparency, with the training seed, the LCORPUS root, the mask-schedule seed and checkpoint hashes by training step (a light form of proof of learning).
- The records are verified here: the hashes are recomputed from the files they name.
- A CPU twin re-evaluates the checkpoint. It scores the fixed BLiMP subsample within the registered tolerance.
- The browser re-ranks exactly any table or head that was snapped to exact numbers.
- The receipt is classed: reproducible if the twin’s output is bit-identical, attested if it is not.
The contamination receipt
A score means little if the test sentences were in the training data. So each corpus carries a contamination receipt: the SHA-256 of every 13-gram (run of 13 tokens) in the evaluation items, intersected with the training id streams. Only the counts are published, never the text.
A signature proves who committed a record. It does not prove that the record’s licence holds or that its data is correct. An attested number is one float run’s output. A record with no valid attestation is never called verified, and language shards keep the same split: their verdicts are reported separately as self-consistent and authentic.
The benchmark ladder sets out what each stage measures, with every baseline and its grade. The registered claims, and the spikes that must not be read as results, are in Explainer 8, in PREREGISTRATION.md and on the claims ledger.