Hyperbolic Lexicon · Explainer 7 · Phase 0

The Benchmark Ladder

Five rungs, from “does the text become the same ids in Python and in the browser?” to “does a hierarchy factor help a small BERT on GLUE?”. Each rung names its question, its data, its metrics in plain words, the published numbers to compare with and how far each can be trusted, the controls, and where it can run. Only the first two rungs run fully here. The rest wait for claims registered before any code, and some for a GPU or for hosts that are still blocked here.

The ladder at a glance Stage 0: the tokenizer Stage 1: WordNet Rules for every stage Licences
WordNet nouns 82,115 · closure edges 743,241 runs here: Stages 0–1 BLiMP, LTG-BERT strict-small: 60.6 (2024 pipeline) Euclidean control: every hyperbolic claim
Contents
  1. Where this sits
  2. The ladder at a glance
  3. Stage 0: determinism and the tokenizer
  4. Stage 1: the WordNet hierarchy
  5. Stage 2: lexical semantics
  6. Stage 3: small MLM pretraining
  7. Stage 4: sentence and encoder tasks
  8. Rules for every stage
  9. Licences
01

Where this sits

The ladder is the last layer of the system map. It scores what the exact placement, the learned static embeddings and, later, the encoder’s heads produce. Its own outside data is BabyLM, BLiMP and GLUE; WordNet and UD English EWT reach it through the earlier stages.

Each rung asks one question and is climbed only when the one below it holds: the same ids in every engine, then the geometry against a real hierarchy, then word meaning, grammar and sentence tasks. Stages 0 and 1 run fully in this container (4 CPU cores, 15 GB RAM, no GPU); the rest need hosts beyond GitHub (most of them reachable since the network was widened at the start of Phase 1) or a GPU (Explainer 6 has the three compute tracks).

This page is a design, not a report. No number on it is a Lexicon result; Phase 1’s results for Stages 0 and 1 are in Explainer 9, Phase 2’s in Explainer 10, Phase 3’s, on the records rather than a rung of the ladder, in Explainer 11, and Phase 4’s, the rest of Stage 1 (L6 and L4(c)), in Explainer 12. The WordNet and UD English EWT numbers are Phase 0 pins: counts and hashes measured from the pinned archives (wordnet_v1.meta.json, ewt_v1.meta.json). Numbers called “a planning check” or “an estimate” come from the design and wait for a spike or a run; where a run has since measured one, its value is given beside it. Spikes S1–S7 were still running when it was written, and nothing here quotes them (Explainer 8).

How to read the marks

Every published score carries the survey’s grade. verified: read from a repository or source code (for Sala et al., confirmed against the paper by the cross-check), or, since the network was widened, in the primary paper itself, whose table or section is named beside the number. likely: from a search summary of the primary paper, which could not be opened because arxiv.org and aclanthology.org were blocked here then. secondary: only from someone else’s summary or table. Where the survey’s adversarial cross-check corrected a survey, the correction is used.

Re-checked on 5 Nov 2026. Once arxiv.org, aclanthology.org and huggingface.co could be reached, the 34 outside numbers on these pages (of 59) whose primary source sits on one of those hosts, and that were not already verified, were each read in that source. Each verdict was then attacked by a second, independent download and reading, and all 33 verdicts stood. 28 were confirmed: where the pages grade them, the mark now reads verified, with the table or section named. 5 were corrected: Poincaré’s HyperLex 0.512 (a noun-only score), HiT’s 0.916 / 0.899 (HiT on all-mpnet-base-v2, not on MiniLM), the 6·N·T rule (N counts only non-embedding parameters), HELM’s “up to 4%” (OpenbookQA at about 120M parameters, not MMLU and ARC at 1B), and L6’s training settings, which the registration credits to Nickel & Kiela but which mix the paper with the authors’ later scripts; that one goes to an erratum amendment, since registered text is never edited. One, WordSim-353’s 203 / 252 split, was not found in the paper it is credited to (Agirre et al. 2009) and keeps its mark. Still unreachable: 5 numbers, on hosts still blocked here: Occiglot’s ideal values (occiglot.github.io), Yu & De Sa’s 2% (proceedings.neurips.cc), Gu et al.’s mixed-curvature figures (openreview.net), the Hyperbolic PLM’s 1.3× (ieeexplore.ieee.org, dl.acm.org, doi.org) and the Unicode 17.0 total (unicode.org). The 7 numbers read from GitHub repositories and the 13 already verified were not part of this pass.

The Unicode tables, checked the same day. The pinned table unicode_15_1_v1.json matches the official UCD 15.1.0 files in every field. There are no differences, so nothing needs deciding between CPython and the UCD, and no erratum is needed: check_ucd_v1.py --check exits 0, and its new test passes with the rest of the fast suite (127 passed). The files come from unicode-org/unicodetools at commit b91447019d1b785cd5e82193936e3c9209852e9b, the head of main on 2026-11-05, read with git ls-remote because api.github.com refused the repository here; so the pin is the branch head, not the last commit to touch the 15.1.0 folder, and the files’ 2023 headers fit the release. SpecialCasing.txt (16,832 bytes, SHA-256 55a477ef…) and DerivedCoreProperties.txt (1,072,686 bytes, f55d0db6…) name 15.1.0 on their first line; UnicodeData.txt (1,914,200 bytes, 2fc713e6…) has no header, so DerivedAge.txt (131,154 bytes, 04e16379…) is pinned beside it to show it is 15.1.0. Full hashes: ucd_15_1_v1.meta.json.

Where a stage can run: here in this container, browser or CPU; net needs a host that was blocked here in Phase 0 (huggingface.co, osf.io, zenodo.org and others; since the network was widened at the start of Phase 1, those three are reachable, except osf.io’s file downloads); gpu needs a GPU, on the user’s hardware or in the cloud.

02

The ladder at a glance

Five stages, cheapest first. The bottom two are exact, run here, and decide seven of the eleven registered claims (L1–L7; the other four, L8–L11, concern the records of Explainer 4). The top three need hosts or hardware this container lacks, and the top two are where hyperbolic geometry has its weakest published record, so they come last, after a gate.

STAGE 4Sentence and encoder tasks: GLUE, an MTEB subset, SentEval probesgpu + net STAGE 3Small MLM pretraining: BabyLM, BLiMP, a (Super)GLUE subsetnet; gpu at 100M gate: L1–L11 decided and reviewed before any encoder code STAGE 2Lexical semantics: HyperLex, hypernymy suites, similarity controlscpu + net STAGE 1Hierarchy: WordNet noun closure and mammals, exact MAPhere · L3–L7 STAGE 0Determinism and the tokenizer: byte identity, engram/1 collisionshere · L1, L2
Green rungs run here and decide registered claims; rose rungs need hosts that were blocked in Phase 0, most of them reachable since Phase 1; violet needs a GPU. The dashed line is the roadmap’s Stage-1 gate: L1–L11 decided and reviewed with the project’s owner before any encoder code is written.
StageQuestionDataMetricsPublished baselinesRuns
0 decidedDoes text become the same bytes, pieces, ids and offsets in Python, Node and Chromium? How often does engram/1 merge distinct words?WordNet glosses (117,659); UD EWT (16,622 sentences); a Unicode stress set; 2,000 synthetic strings; 147,306 WordNet lemma typesbyte identity (all-or-nothing); collision share; as gates only: fertility, continued words, bytes per token, Rényi efficiency, MorphScorenone for English WordPiece were found, so baselines are computed locally on the same texthere
1 decidedDoes the exact tree carry WordNet’s hierarchy, better than flat trees, and how does it stand against learned embeddings?noun closure (82,115 nouns, 743,241 edges); mammals (1,182 nodes); verb forest for dry runsgraph-edge and closure MAP, mean rank (pessimistic ties), distortion, bits per node; later cone F1, HiT F1Sala 2-d 0.989 verified; Poincaré 200-d 0.87 and 5-d values verified (Nickel & Kiela, Table 1)here
2 not registeredDo the geometries track graded and directional hypernymy, and lose where they should on similarity and relatedness?HyperLex (2,616 pairs); BLESS, EVAL, LEDS, Shwartz, WBLESS, BiBLESS; SimLex-999, WordSim-353, MEN; SemEval-2018 Task 9Spearman; detection AP, direction accuracy; MAP, MRR, P@5HyperLex nouns: Poincaré 0.512 (2,163 noun pairs), LEAR 0.705; all pairs: LEAR 0.686 verified; Hearst patterns verifiedCPU net
3 Phase 6Does a hierarchy factor help a small masked LM learn grammar, at matched parameters and tokens?BabyLM strict-small (≤ 10M words) and strict (≤ 100M); BLiMP 67 × 1,000 pairs; UD EWT for probesBLiMP accuracy under PLL-word-l2r; (Super)GLUE subset; probe UUAS and depth Spearman2024 pipeline: LTG-BERT 60.6, BabyLlama 69.8 on strict-small verifiednet gpu at 100M
4 Phase 7At BERT-mini and BERT-small size, does it help sentence tasks, and at what cost?GLUE; the cheap MTEB(eng, v2) subset; SentEval TreeDepth and TopConstmacro GLUE; MTEB task scores; probe accuracy; parameters, FLOPs, browser latencyBERT-Tiny 64.2 to BERT-Medium 73.5, ELECTRA-Small 77.4 verifiedgpu net
What each stage decides

Stage 0 decides L1 (byte identity) and L2 (engram/1 collisions) in Phases 1–2. Stage 1 decides L3–L7 in Phases 1–4: exact addresses, exact ranking, the tree against curvature 0, learned Lorentz against learned Euclidean, and the X(5,q) overlay. Phase 1 decided L2, L3 (Python side), L5 and L7 (Explainer 9), Phase 2 L1, L3(f) and L4(a, b) (Explainer 10), and Phase 4 L6 and L4(c) (Explainer 12), all supported, L6(c) against its registered prediction. Stages 2–4 decide nothing yet. Their claims are registered by dated amendment before any code for them is written, as decision D11 requires; Stage 2’s, planned for Phase 4, were not registered there and remain to be. The registered claims are on the claims ledger and in Explainer 8; their full text is in PREREGISTRATION.md.

03

Stage 0: determinism and the tokenizer

Before any geometry, the text has to become the same numbers everywhere. If Python and the browser disagree on a single byte, nothing computed on one side can be checked on the other. So Stage 0 has two hard gates and a handful of descriptors that are reported but never optimised.

Normaliser
The step that cleans text before it is split: lower-casing, splitting off punctuation, and removing accents by decomposing letters (NFD) and dropping the combining marks. The Lexicon’s N_B follows BERT-uncased’s rules on Unicode 15.1 tables exported from Python and pinned by SHA-256 (Explainer 3).
WordPiece
BERT’s subword tokenizer. Each word is cut, from the left, into the longest pieces found in a fixed vocabulary. Pieces after the first carry a ## mark: cats → cat ##s. A word with no possible cut becomes [UNK], the unknown token.
Fail closed
On a code point that Unicode 15.1 does not assign, N_B emits [UNK] for the word and logs its offset, rather than guessing.
Id stream
The token ids of a text in order. Its SHA-256 is one 64-hex-digit fingerprint: equal fingerprints mean byte-identical streams.

Gate 1 · byte identity (claim L1)

The Python reference (the authority), Node 22 and headless Chromium must produce identical normalised bytes, token strings, token ids, byte offsets and id-stream SHA-256 on every instance. The JavaScript never calls String.prototype.normalize, toLowerCase or \p{} classes, because the engines here ship different Unicode versions: Node 22.23.4 has Unicode 17.0 (ICU 78.3), Python 3.13.18 has unicodedata 15.1.0. Tables are data, so the platform’s version cannot leak in. In Phase 2 no instance differed in any of L1’s four cells and every control failed: L1 is supported (Explainer 10).

  • Instances. Every WordNet 3.0 gloss: 117,659 strings, one per synset (82,115 nouns, 13,767 verbs, 18,156 adjectives, 3,621 adverbs). Every sentence of UD English EWT at commit 4a4d77f: 12,544 train, 2,001 dev, 2,077 test. The Unicode stress set spike S6 writes. 2,000 synthetic strings from a registered pool of scripts, marks, controls and emoji.
  • Vocabularies. V0 “lex0”, 8,192 entries trained on the glosses and frozen by hash, decides. V1, bert-base-uncased (30,522 entries, Apache-2.0), is the second cell: it was pinned from huggingface.co in Phase 1, at revision 86b5e09.
  • Rule. Exact and all-or-nothing, per vocabulary and per runtime. One differing instance refutes and is recorded as a counterexample.
  • Controls that must fail. A normaliser that uses the platform’s own functions must differ somewhere on the stress set. Shortest-match-first WordPiece must differ on the glosses. Turning accent stripping off must differ on the glosses or EWT. If any of these passes, the test cannot see what it claims to see, and the cell is void.

Gate 2 · engram/1 collisions (claim L2)

engram/1 is the 64-bit key the existing systems give a word: the XOR of one fixed grid per letter. It keeps only whether each letter occurs an odd or even number of times (rank 26 over GF(2)), so LISTEN = SILENT, and numbers and words such as noon land on the zero key (Explainer 1). L2 measures how often that bites on a real vocabulary: the share of WordNet’s 147,306 distinct lemma types that share a key with another type. Supported at ≥ 1%; the prediction was 2–10%, and Phase 1 found far more (Explainer 9). The control, a 64-bit SHA-256-prefix key on the same types, must show no collision at all, and Python and Node must produce the same key table. Whatever the share, engram/1 stays a compatibility checksum, never a token id (decision D4); L2 measures what using it as an id would cost.

Descriptors: reported, never optimised

DescriptorIn plain wordsSource and grade
FertilityTokens per word. 1.0 means no word is split; higher means more splitting.Rust et al. likely
Continued wordsThe share of words cut into two or more tokens. Occiglot treats fertility 1.0 with continuation 0.0 as the ideal.Rust et al. likely
Bytes per tokenUTF-8 bytes of text per token: how much text each id carries on average.standard; computed locally
Rényi efficiency, α = 2.5How evenly the tokens use the vocabulary. Take each piece’s share p of all tokens; the Rényi entropy is Hα = log2(Σ pα) / (1 − α). Divide by log2 V, its largest possible value for V pieces. 1 is perfectly even use. At α = 2.5 frequent pieces dominate, so a vocabulary where a few pieces do most of the work scores low.tokenization-scorer (MIT). Zouhar et al. (ACL 2023): correlated with BLEU in German–English translation at r = 0.78 (arXiv version) and 0.82 (ACL version), where compressed length did not verified
MorphScore v2Whether token boundaries fall on morpheme boundaries (un|happi|ness): precision and recall, frequency-weighted, on Universal Dependencies treebanks in 70 languages.catherinearnett/morphscore; no licence stated, so report only. Its 70-language study found morphological alignment explains little of downstream performance verified
[UNK] rate, fail-closed countHow often a word cannot be represented, and how often the normaliser refused an unassigned code point.ours
Gates, not goals

Three large studies found that these numbers predict downstream quality weakly. Ali et al. (NAACL Findings 2024), with 24 models at 2.6B parameters: fertility and parity do not reliably predict downstream performance. Schmidt et al. (EMNLP 2024), with 64 models from 350M to 2.4B: fewer tokens does not mean a better model, and pre-tokenization matters. Lotz et al. (ACL 2025): compression alone is unreliable. Ali et al. and Schmidt et al. verified (the abstract and Sec. 3.3 of arXiv:2310.08754v4; the abstract and Sec. 1 of arXiv:2402.18376v2); Lotz et al. likely.

So the descriptors catch a broken tokenizer (a flood of [UNK], a fertility that explodes on one script) and are never used to claim one tokenizer is better. No published fertility for BERT’s WordPiece on English was found, so every comparison is computed here, on the same text, with both vocabularies.

Worked example: a made-up splitWordsPiecesFertilityContinuedBytes per tokenRényi efficiency
the cat ##s sat on the mat671.171 of 62.570.93
the cats sat on the mat661.000 of 63.000.92
t ##h ##e c ##a ##t ##s s ##a ##t3103.333 of 31.000.93
the the the the the the the cat881.000 of 83.000.31

Arithmetic on invented splits, not a measurement of any tokenizer. Bytes count each word’s UTF-8 bytes without spaces; V is the number of distinct pieces in the line (a real run states which V it uses and pins the scorer’s version). Read the rows together. Whole words score the “ideal” fertility of 1.0, which a vocabulary of every word would also give. Letter-by-letter splitting never fails but makes each id carry one byte. One piece used seven times in eight drags Rényi efficiency down to 0.31, because at α = 2.5 frequent pieces dominate. None of these says which tokenizer helps a model.

Later on this rung. Bits per byte of the shards against gzip, xz and zstd, and the enwik8 and text8 benchmarks once mattmahoney.net is reachable net. TokSuite’s perturbation benchmark (14 models that differ only in tokenizer; the abstract and Sec. 3.3 of arXiv:2512.20757v2, ICML 2026) is a later robustness probe verified.

04

Stage 1: the WordNet hierarchy

This is where hyperbolic geometry has its strongest published record, and where this stack can decide something now: does the exact, training-free LPS tree carry a real linguistic hierarchy? Everything runs here, exactly, on WordNet 3.0.

The data, pinned

WordNet 3.0 comes from nltk_data on GitHub (wordnet.zip, 10,775,600 bytes, SHA-256 cbda5ea6…), fetched once into a cache outside git and re-checked by make_wordnet_v1.cjs --check. A synset is one sense shared by a set of words; a hypernym is its more general sense (a dog is a kind of canine). Following hypernym and instance-hypernym links upward from every noun gives the closure: every (noun, ancestor) pair.

Pinned countValueWhy it matters
Noun synsets82,115Exactly Nickel & Kiela’s count (Sec. 4.1 of arXiv:1705.08039v2: “82,115 nouns and 743,241 hypernymy relations”) verified, so their protocol can be followed. The paper names no WordNet version; the 3.0 pin and the instance-hypernym links are the Lexicon’s own.
Closure edges (hypernym and instance hypernym)743,241Also exactly their count: the positives of closure reconstruction.
Spanning roots1entity.n.01; the spanning parent is each synset’s first hypernym or instance-hypernym pointer (@ or @i).
Maximum spanning depth19Deep enough that float64 ranking is at risk (L4).
Maximum children of one node659Below its root the tree has 5 children per node, so this node’s children need ⌈log5 659⌉ = 5 letters each.
Nodes with more than 5 / more than 25 children3,291 / 396Their children need 2 or more / 3 or more letters each; the letters beyond the first are “virtual letters”.
Nodes with exactly one child6,162Each still costs one letter.
Nouns with several parents2,213The spanning tree keeps one parent; closure reconstruction keeps all of them.
Mammal closure: nodes / edges1,182 / 6,542The set L6 trains and scores on.
Mammal spanning subtree1,176Every node a query in L4 and L5; seconds in the browser.
Verb synsets / spanning roots13,767 / 559A forest disjoint from the nouns: spikes and dry runs only.

Two protocols, never equated

Graph-edge (Sala-style) Closure (Nickel–Kiela) root parent query sibling child root (ancestor) parent query sibling child (descendant)
Green nodes count as the query’s true neighbours. A graph-edge score rewards keeping parent and children close; a closure score also rewards keeping distant ancestors closer than siblings, which a distance-based tree does not do.

The same embedding can score very differently under the two, so the Lexicon reports both and never compares a number from one with a number from the other. The design pairs Sala et al.’s numbers with graph-edge reconstruction and Nickel & Kiela’s with closure reconstruction.

The metrics

MAP
Mean average precision. For one query, rank every other node by distance. At each true neighbour, take the precision so far: true neighbours ranked at or above it, divided by its rank. Average those over the query’s true neighbours (AP), then average AP over all queries. 1 is perfect.
Mean rank
For each true neighbour, one plus the number of non-neighbours ranked ahead of it, averaged over all true neighbours of all queries. 1 is perfect.
Pessimistic ties
Exact distances can tie. Decisions place tied non-neighbours ahead of the true neighbour, the worst case, so a tie can never help a claim. Fractional ties, the average over every order of the tied block, are reported beside them.
Computed exactly
Ranks come from the exact Lorentz inner product of exact points, compared as integers, so MAP and mean rank are exact fractions, identical in Python and JavaScript (L4). Float64 ranks can invert on deep words; a planning check saw straight paths invert at depth 11–12.
Distortion
How far distances in the space stray from distances in the tree: the mean of |dH / (τ · dT) − 1|, where dT counts tree edges, dH is the hyperbolic distance and τ is one step’s length (ln 7 in the H³ tree). 0 is perfect.
Bits per node
What an address costs to store: letters × log2 5 ≈ 2.32 bits per letter. A learned table snapped to a 2−32 grid costs d × 33 bits per node: 66 at d = 2, 99 at d = 3, counting the snapped integers alone. Phase 4 measured bits per node as the exact point over one denominator (the lifted Lorentz point over 432 − |n|², the Euclidean n over 232): on average about 237 for learned Lorentz and 95 for learned Euclidean at d = 2, 304 and 127 at d = 3 (Explainer 12).
Cone F1 (later)
Ganea et al.’s protocol: train on the basic edges (the transitive reduction) plus 0, 10, 25 or 50% of the other closure edges, then test whether held-out pairs are recognised. F1 is the balance of precision and recall.
HiT F1 (later)
Hierarchy Transformers’ tests. Multi-hop: train on one-hop edges, test transitive ones. Mixed-hop: an incomplete hierarchy, possibly with unseen entities. Random and hard negatives; a threshold tuned on validation.
Worked example: one query, five candidatesABCDE
Exact distance12234
True neighbour?yesyesnonoyes

B and C tie. Pessimistic, C goes first: the neighbours sit at ranks 1, 3 and 5, so AP = (1/1 + 2/3 + 3/5) / 3 = 34/45 ≈ 0.756, and the mean rank is (1 + 2 + 3) / 3 = 2. Optimistic, B goes first: AP = (1 + 1 + 3/5) / 3 = 13/15 ≈ 0.867, mean rank 5/3. Fractional, the average over both orders: AP = 73/90 ≈ 0.811. A float evaluator that breaks ties by node index lands on one of the two, depending on how the nodes happen to be numbered; L4 keeps such an evaluator as a control that must disagree with the exact one on a query with exact ties.

Published baselines

WorkSettingNumberGrade
Sala, De Sa, Gu & Ré (ICML 2018): combinatorial, no trainingWordNet, 2 dimensionsMAP 0.989 at about 500 bits of precision, or 10 dimensions at 32 bitsverified from the abstract; confirmed by the cross-check
Nickel & Kiela (2017): Poincaré embeddingsnoun closure reconstruction, 200 dimensionsMAP 0.87 (mean rank 3.83), the figure Sala et al. quoteverified Table 1, arXiv:1705.08039v2
Same, Table 1noun closure reconstruction, 5 dimensionsMAP: Poincaré 0.823, Translational (a TransE-style score) 0.517, Euclidean 0.024verified Table 1, arXiv:1705.08039v2
lateral/poincare-embeddings, an open reimplementationa 1,000-node sampleMAP 0.5974, mean rank 286.23likely
Ganea et al. (2018): entailment cones10 dimensionsF1 85.9 with 10% non-basic edges, 94.4 with 50%; order embeddings lead at 0%verified Table 1, arXiv:1804.01882v3
HiT (He et al., NeurIPS 2024)WordNet multi-hop, random / hard negativesF1: HiT on all-mpnet-base-v2 0.916 / 0.899 (on all-MiniLM-L12-v2: 0.903 / 0.908); Poincaré (200-d) 0.864 / 0.830; fine-tuned all-mpnet-base-v2 0.615 / 0.687verified Table 2, arXiv:2401.11374v4
Bansal & Benton (2021)WordNet nouns, by dimensionEuclidean matches or beats Poincaré and Lorentz at 50 or more dimensions, in the authors’ own matched reproduction (reconstruction only)verified abstract and Table 1, arXiv:2109.07488v1
Exactness pays for precision; it does not escape it

Sala’s construction is the published family closest to the norm-5 tree, and it needs about 500 bits to reach 0.989 in two dimensions. One survey said our exact tree “avoids” that problem; the cross-check corrected it. Exact H³ points at depth 28 already carry 138-bit numerators and 79-bit denominators. Exact BigInt arithmetic pays the precision cost; it does not remove it. What exactness buys is certification: ranks, ties and MAP that are identical in every engine. Fixed branching also costs MAP: where a node has more than five children, “virtual” siblings sit as close as the parent, or closer. So the design predicts the noun graph-edge MAP will fall below Sala’s 0.989, and the closure MAP well below 0.87. Those are predictions, not results.

Controls and claims

ClaimQuestionIts controls
L3Does every noun synset get an exact, invertible, certified address and point in H³ and H²?Variants that drop or fold letters must break injectivity
L4Is exact ranking identical across engines, and float ranking not certifiable?Float32 must show an inversion, or the test is insensitive
L5Does the training-free tree beat curvature 0 at d = 2 and 3, on mammals and 1,000 noun queries?Flat trees F (unit edges) and F* (edges shrinking by 1/4 a level); shuffled (S3) and content-hash (K3) addresses must score below the tree, or the test is void
L6Do learned Lorentz embeddings beat learned Euclidean at d = 2 and 3, and the advantage vanish at d = 50?Euclidean at matched d (5 and 10 reported); shuffled labels must give MAP ≤ 0.05; training on random pairs must lose
L7Does the same tree, read mod q, give SHD-CCP RFC 03’s Ramanujan graphs (Explainer 2)?A wrong square root of −1, a 4-generator subset

Effects are paired differences with 95% bootstrap intervals; full rules are in Explainer 8, and the controls are explained in Explainer 5.

Cost. Mammals take seconds in the browser. Nouns run as 1,000 sampled queries in Python, minutes per arm; L6’s roughly 290 runs, tuning included, are estimated at 1–5 hours. All are estimates; Phase 4 measured L6’s: 240 tuning runs in 6,089 s and the decisive run, 100 trained tables with the controls, in 3,051 s, on 4 cores (Explainer 12). A cheap pre-check before any ranking: estimate how tree-like a set of distances is with Gromov’s δ, built from four points at a time, by sampling with the four-point machinery of Hyperbolic Oracles’ ln 2 guard. Later: cone F1, and HiT’s tests with splits rebuilt from WordNet by HiT’s protocol, compared with HiT’s published numbers only as indicative, because its own data sits on zenodo.org and huggingface.co net.

05

Stage 2: lexical semantics

From a curated tree to human judgements about words: is “dog” a kind of “animal”, and how much? Hypernymy is asymmetric and graded, the kind of relation hyperbolic embeddings are usually proposed for. Nothing here is registered yet: these claims were planned for Phase 4, which did not register them; they remain to be added by amendment, before any code for them.

Spearman correlation
Agreement between two rankings: the model’s scores and the humans’. 1 is the same order, 0 no relation.
Detection AP
Average precision of ranking true hypernym pairs above other pairs.
Direction accuracy
For true pairs, the share where the model picks which word is the more general one.
MRR and P@5
For hypernym discovery: the average of 1 / (rank of the first correct hypernym), and the share of the top five candidates that are correct.
SetWhat it holdsMetricPublished numbersNotes
HyperLex (Vulić et al., 2017)2,616 graded lexical-entailment pairs, nouns and verbsSpearmanPoincaré 0.512 on the noun part only (2,163 noun pairs; 5-d WordNet embeddings; Nickel & Kiela, Table 3) verified, so it sits beside LEAR’s nouns, not all pairs; gensim’s replication 0.48 likely; LEAR 0.686, nouns 0.705 (Table 2 of arXiv:1710.06371v2); humans about 0.85 (HyperLex, Table 4) verifiedlicence not checked
hypernymysuite: BLESS, LEDS, EVAL, Shwartzdetection sets; validation lines 1,453, 275, 736 and 5,256detection APHearst patterns AP 0.76 on BLESS (distributional 0.19) verified: the best pattern model (spmi) against the best distributional baseline, Table 2 of Roller et al. 2018, arXiv:1806.03191v1code CC-BY-NC 4.0, archived 31 Oct 2023; download_data.sh exits before downloading and must be adapted verified
hypernymysuite: WBLESS, BiBLESStwo further sets (WBLESS: 167 validation lines)accuracy (LEAR); detection AP (Hearst)LEAR 0.912 on WBLESS and 0.875 on BiBLESS (fastText input, Table 2) verified; Hearst patterns AP 0.96 on WBLESS (distributional 0.69; Table 2) verifiedseeded splits (seed 42)
SimLex-999666 noun, 222 verb, 111 adjective pairs; similarity, not relatednessSpearman—licence unknown; its host is blocked here
WordSim-353Agirre’s split: 203 similarity, 252 relatedness pairsSpearman—CC BY 4.0
MEN3,000 relatedness pairsSpearman—licence unverified
SemEval-2018 Task 9hypernym discovery from a corpus: 5 subtasks, 11 teams, 39 systemsMAP, MRR, P@5CRIM (run 1): Medical MAP 34.05, Music 40.97 verified (Table 1 of ACL S18-1116; Tables 7–8 of the task paper, S18-1115)data on CodaLab; licence not checked

Controls where flat should win. SimLex-999, WordSim-353 and MEN measure similarity and relatedness: “cup” and “coffee” are related, but neither is a kind of the other. The design expects a Euclidean space to win there. A hyperbolic gain on hypernymy means more when it sits next to a flat gain on relatedness, because it shows the geometry is doing a specific job. For context, Poincaré GloVe (ICLR 2019) beat Euclidean GloVe on similarity and set an unsupervised state of the art on WBLESS likely.

Where it runs. The computation is small and runs on the CPU, but most of these files sit on hosts that were blocked here in Phase 0 net; the SimLex host still is. What can be built here is built from WordNet: HiT-protocol multi-hop and mixed-hop splits, planned for Phase 4 and not yet built: they wait for Stage 2’s claims.

06

Stage 3: small MLM pretraining

The first stage with an encoder. A masked language model (MLM) learns by hiding some tokens and predicting them from the rest, as BERT does. The question is narrow: does feeding it the tree address as a hierarchy factor help it learn grammar from a child-sized corpus, at the same parameters and tokens?

The corpus: BabyLM

  • Tracks. Strict-small allows at most 10M words; strict at most 100M. From 2025 the total words a leaderboard checkpoint has seen are also capped, about 10 epochs: 100M for strict-small, 1B for strict verified (the word caps: Sec. 3 and Table 1 of the 2023 call, arXiv:2301.11796v1; the exposure caps: Sec. 4.2 of the 2025 call, arXiv:2502.10645v2). 2026 adds a multilingual track; which track to target is an open question.
  • Composition (2024 strict, Table 1 of the 2nd BabyLM CfP, arXiv:2404.06214v2; word counts approximate): CHILDES 29M, BNC dialogue portion 8M, Project Gutenberg children’s stories 26M, OpenSubtitles 20M, Simple English Wikipedia 15M, Switchboard 1M words verified. The CfP gives no strict-small breakdown; that strict-small is about a tenth of each comes from elsewhere secondary.
  • Access. The corpus is on osf.io and Hugging Face, both blocked here in Phase 0 net. Since the network was widened at the start of Phase 1, Hugging Face is reachable, while osf.io’s file downloads still redirect to a blocked object store, so BabyLM comes from Hugging Face. Dry runs use WordNet glosses and UD EWT here; decisive runs can be made here or on the user’s machine. As 60 KB shards, strict-small would be about 1,000 independent chains and strict about 10,000 (Explainer 4).
  • Cost. About 6 · N · T floating-point operations to train, with N the non-embedding parameters and T the training tokens (Kaplan et al. 2020, arXiv:2001.08361v1, §2.1) verified: N = 5M parameters on T ≈ 1.3 × 108 tokens is about 3.9 × 1015, roughly 11–22 hours on 4 cores. That is an estimate: the rule leaves out the embeddings, the output layer and the attention cost that grows with context, and how much of the 5M is embeddings is not fixed yet, so a calibration spike measures throughput first.

Scoring

BLiMP
67 paradigms × 1,000 minimal pairs: two sentences that differ in one place, one grammatical and one not. Accuracy is the share of pairs where the model prefers the grammatical sentence. Human agreement 96.4%; CC BY; on GitHub, reachable here; errata exist for its TACL Tables 3–4 verified.
Pseudo-log-likelihood
A masked model has no left-to-right probability, so a sentence is scored by masking each token in turn and adding up the log-probability of the true token (PLL).
PLL-word-l2r
Kauf & Ivanova (ACL 2023): plain PLL inflates scores for words split into several tokens, because the other pieces give the answer away. PLL-word-l2r also masks the rest of the word to the right. The choice changes BLiMP results likely, so it is pinned with a commit of babylm/evaluation-pipeline-2025 (Apache-2.0).
Structural probe
A linear map under which squared distances between a model’s word vectors track distances in the parse tree and squared norms track depth (Hewitt & Manning, 2019). The Poincaré probe (ICLR 2021) recovers syntax trees from BERT better than Euclidean probes. Scored by UUAS (the share of gold tree edges recovered, ignoring labels and direction) and the Spearman correlation of predicted with true depth, on UD EWT here.

Published baselines, and why the pipeline is pinned

TrackModelBLiMPBLiMP-suppEWoKGLUEMacro
strict-smallLTG-BERT (flagged undertrained)60.660.848.960.357.7
strict-smallBabyLlama69.859.550.763.360.8
strictLTG-BERT69.266.551.968.464.0
strictBabyLlama73.160.652.169.063.7

The baselines of babylm/evaluation-pipeline-2024 verified. The macro column is the plain mean of the four columns before it, which the numbers bear out to rounding. The GLUE column there is a subset: BoolQ, CoLA (Matthews correlation), MNLI, MNLI-mm, MRPC (F1), MultiRC, QNLI, QQP (F1), RTE, SST-2 and WSC.

Same architecture, 60.6 or about 80

The 2024 pipeline gives strict-small LTG-BERT a BLiMP of 60.6, and its README flags the score as lower than expected, possibly from too few epochs. 2023-era papers put LTG-BERT and ELC-BERT at about 80.5 on strict-small, under a different pipeline; GPT-BERT, the 2024 winner, is reported at 81.2 (about 30M parameters, 10M words) and 86.1 (119M parameters, 100M words) verified (all from Table 1 of Charpentier & Samuel, arXiv:2410.24159v2; the 2024 win from the BabyLM findings, arXiv:2412.05149, Sec. 5.2). The cross-check traced the clash to different pipeline years and scoring rules, on top of the undertraining the README flags. A gap of some 20 points between reports of one architecture is why every BLiMP number here names its pipeline commit and its scorer, and is compared only with numbers from the same ones.

Floors (2023 pipeline, majority label)

CoLA 0.0, SST-2 50.2, MRPC F1 82.0, QQP F1 53.1, MNLI 35.7, QNLI 35.4, RTE 53.1, BoolQ 50.5, MultiRC 59.9, WSC 53.2. RoBERTa-base on strict-small: CoLA 25.8, SST-2 87.0, MNLI 73.2.

babylm/evaluation-pipeline-2023 · MIT · verified

The 2025 suite

Zero-shot BLiMP, BLiMP-supplement, EWoK, entity tracking, derivational morphology and reading alignment, plus hidden tasks. Fine-tuning on (Super)GLUE with MNLI and QQP cut to 10k and CoLA, SST-2, MNLI-mm and QNLI removed: lr 3e-5, batch 32, 10 epochs, seed 42. Baselines GPT-BERT and GPT-2 Small.

babylm/evaluation-pipeline-2025 · Apache-2.0 · verified

EWoK (EWoK-core-1.0: 4,374 items in 11 domains) stays near chance, about 50, for every baseline, so it says little early. It is gated on Hugging Face, and its terms forbid public plain-text redistribution, so it is reported only and enters a chain by hash only verified.

Candidate claims, to be registered by amendment in Phase 6 before any code: L12, the address factor against shuffled addresses and no factor, on BLiMP, at matched parameters and tokens, with an equivalence band of ±1 point predicted (a null result); L13, Lorentz or Poincaré against Euclidean hierarchy heads on HiT-protocol F1; L14, the Poincaré probe against the Euclidean structural probe on UD EWT; L15, PolyStep against MeZO and ZO-SGD on a fixed head at equal evaluations. None is registered yet.

07

Stage 4: sentence and encoder tasks

The general-NLP rung, where hyperbolic geometry has helped modestly or not at all. It runs in Phase 7, on a GPU with access to Hugging Face, and only for claims registered by amendment first.

GLUE
Nine sentence-level tasks: acceptability, sentiment, paraphrase, similarity and inference. Tasks with two metrics (accuracy and F1, say) are averaged first.
Macro GLUE
The plain mean over tasks, each task counting once whatever its size.
MTEB
The Massive Text Embedding Benchmark: tasks that score a sentence embedding by similarity, classification, pair classification, clustering and retrieval.
SentEval probes
Small classifiers that read one property off a sentence vector. TreeDepth (depth of the parse tree) and TopConst (its top constituents) suit hierarchy claims.
Euclidean baselineShapeGLUE testGrade
BERT-Tiny2 layers, width 12864.2google-research/bert README, Apache-2.0; fine-tuned 4 epochs, lr 3e-4 to 3e-5 verified
BERT-Mini4 / 25665.8
BERT-Small4 / 51271.2
BERT-Medium8 / 51273.5
ELECTRA-Small12 / 256, 14M parameters77.4google-research/electra: about 4 days on one V100, decent after 200k steps (about 10 h) verified
  • MTEB(eng, v2) has exactly 41 tasks verified in the mteb source (Apache-2.0). The cheap subset: STS12–15, STS17, STS22.v2, STSBenchmark, SICK-R and BIOSSES; classification (Banking77 and others); pair classification (SprintDuplicateQuestions, TwitterSemEval2015, TwitterURLCorpus); and ArXivHierarchicalClustering, which is about hierarchy. Retrieval is skipped as too expensive.
  • SentEval TreeDepth and TopConst: 100k training and 10k test sentences each; the repository was archived on 6 Aug 2025 verified. STS-B has 8,628 pairs (5,749 / 1,500 / 1,379).
  • SuperGLUE is too hard for a tiny model, except the BoolQ, MultiRC and WSC slices BabyLM uses likely.
  • Efficiency. Parameters and FLOPs in the style of ELUE, which scores a model against the ElasticBERT curve (0 = on par; Sec. 3.3 and Table 3 of arXiv:2110.07038v2) verified. Browser latency and memory against a locally pinned BERT-Tiny ONNX run through transformers.js (Apache-2.0), which publishes no speed claims, so latency is measured here.
  • The runs. BERT-mini and BERT-small with the hierarchy factor, against shuffled-address, content-hash, no-factor and byte-level controls at matched parameters, on BabyLM strict. BERT-small (29M parameters, 12.6M of them outside the embeddings) on 1.3 × 109 tokens is about 1.0 × 1017 operations by Kaplan et al.’s rule (2.3 × 1017 if all 29M are counted), roughly 0.5–2 hours on one A100, worked out from the larger figure (estimate). Results come back as receipts that are re-checked here.
No hyperbolic encoder number is a baseline

The Hyperbolic Pre-Trained Language Model (TASLP 2024), a Lorentz BERT, reports gains across GLUE, SuperGLUE and QA, but its numbers could not be checked, and a secondary survey puts it at about 1.3× slower than BERT likely. HELM’s “up to 4%”, which the paper does not define, matches only its largest single-benchmark gain, about 4 points on OpenbookQA at about 120M parameters; at 1B its gains are +2.3 points on MMLU, +1.0 on ARC-Challenging and +1.0 on average, all near chance accuracy verified (the abstract and Table 1 of arXiv:2505.24722v2). Neither is used as a comparator: the first’s numbers are still unchecked, and the second’s gains are small and near chance.

08

Rules for every stage

The same rules hold on every rung and on every compute track, so that a number from the browser, the CPU and a GPU can be compared, and so that a null result is reported as plainly as a win. Decision D10 fixes what the tracks share: the tokenizer record, the manifests, the random generator, the metric code, the budgets and the receipts.

  1. A Euclidean control, always. Every hyperbolic claim is paired with a flat arm at matched dimension, matched bits and matched parameters, plus shuffled-address and content-hash controls. Bansal & Benton found Euclidean embeddings matching Poincaré at 50 or more dimensions verified, so a low-dimension win says nothing about high dimensions.
  2. Pinned scorers and pipelines. PLL-word-l2r at a pinned evaluation-pipeline-2025 commit; a GLUE fine-tuning recipe fixed before decisive runs; tokenization-scorer at a pinned version, with α = 2.5 stated. The 60.6-against-80 gap above is the reason.
  3. One tokenizer, one data identity, one seed scheme. A single LVOCAB/1 record names the vocabulary, the normaliser and the Unicode tables; L1 proves Python = Node = Chromium. Every corpus and split is named by its LCORPUS/1 root. Splits and MLM masks come from the shared generator, so every track masks identically. Instances use seeds 1000+i, secondary draws 2000+i, dry runs +3000, tuning 9000–9023, spikes 90000+i, training seeds 0–4; bootstraps take 10,000 resamples with seed 7.
  4. One metric implementation. Python is the authority. JavaScript reproduces exact metrics line for line; float metrics carry registered tolerances and per-item output hashes.
  5. Matched budgets, recorded. Gradient-free runs are matched on evaluations. Gradient runs record parameters, tokens seen, FLOPs, hardware, library versions, thread count and wall-clock time.
  6. Receipts. Each result is an LBENCH/1 record: claim id, the hash of what was scored, the evaluation set’s hash (gated sets by hash only), scorer and version, seeds, a hash per item’s output, the metrics’ hash, hardware and versions, and a class: reproducible (a CPU twin re-evaluation is bit-identical) or attested (float, single run). The browser re-evaluates every snapped table exactly, and a fixed BLiMP subsample of 67 × 100 pairs is scored on CPU and GPU within a registered tolerance (Explainer 4).
  7. A contamination receipt. SHA-256 sets of every 13-gram in the evaluation items, intersected with the training id streams; only the counts are published.
  8. Equivalence tests where a null is predicted. When the prediction is “no difference”, the claim says how small a difference counts as none, and is supported only if the whole interval sits inside that band. Engram’s E9b did this for recall after a one-letter substitution: 0.0002 against 0.0007 from a shuffled store, a difference of −0.0005 with interval [−0.0017, 0.0004], inside ±0.05 (Engram, Explainer 4). L12 would do the same with a ±1-point band on BLiMP.
  9. Grades travel with numbers. A literature number stays likely until its primary source is read, which needs arxiv.org and aclanthology.org; both are reachable now, and what was re-read once they were is dated under How to read the marks. No first-round claim compares against a likely or secondary number.
Honest about the limits

Hyperbolic geometry clearly helps on explicit hierarchies such as WordNet, and in low dimensions. On general NLP it helps modestly or not at all. This stack’s own record agrees. Hyperbolic Oracles’ C1, “on tree-structured problems the hyperbolic step beats the Euclidean step at equal evaluation budget”, was refuted in 12 of 13 cells, with none supported and one inconclusive; on B4, the inconclusive cell, hyper3 scored 0.325 against flat3’s 0.334 (B4, its preregistration). engram/1, the only word key the stack had, is letter-count parity: LISTEN = SILENT, numbers and even-count words on the zero key, and recall after a misspelling at chance. So encoder-level effects at Stages 3–4 are expected to be small or null, and where a null is predicted the claim is written as an equivalence test (L12). The bet is on Stages 0–1: representation, where low-dimensional trees favour curvature, and certification, where exact results are identical in every engine.

09

Licences

A dataset that cannot be redistributed cannot travel inside a public chain, and a licence nobody checked is treated as unknown. An audit of more than 1,800 fine-tuning datasets by the Data Provenance Initiative found licences omitted in over 70% and errors in over 50% verified, so each source is listed here with what is known about it.

ResourceLicenceHow the Lexicon uses it
WordNet 3.0WordNet licence: use, copy, modify and distribute for any purpose, without fee, if the copyright notice and disclaimer appear on all copies. The text is pinned beside the data (LICENSE-WORDNET.txt).Stages 0–2. Shards may carry its text with the notice kept.
UD English EWT @ 4a4d77fCC BY-SA 4.0L1 and the probes. Pinned by hashes; no text is committed.
BLiMPCC BYStage 3; on GitHub, reachable here.
Unicode tablesUnicode data licence, notice keptExported from Python’s unicodedata 15.1.0 for N_B.
facebookresearch/poincare-embeddingsCC-BY-NC 4.0; archived 21 Oct 2025Protocol reference only: how the closures are built and scored. Not redistributed.
facebookresearch/hypernymysuiteCC-BY-NC 4.0; archived 31 Oct 2023Protocol reference only: its source URLs and seeded splits. Not redistributed.
WordSim-353CC BY 4.0Stage 2 control.
tokenization-scorerMITStage 0 Rényi efficiency.
HiT code, mteb, babylm/evaluation-pipeline-2025, google-research/bert and electra, the bert-base-uncased vocabulary (V1), transformers.js, geooptApache-2.0Tools, baselines and the second vocabulary cell.
babylm/evaluation-pipeline-2023MITSource of the majority floors.
EWoK-core-1.0Gated on Hugging Face; no public plain-text redistribution; acknowledgement required if used in trainingReported only. Enters a chain by hash only.
SNOMED CTNeeds an affiliate licence (the survey’s note, from memory)Excluded, along with HiT’s SNOMED-based data.
BabyLM components: BNC, OpenSubtitles, CHILDES/TalkBank and othersUnknown until checked source by sourcePublic hashes and roots only; the text stays on the holder’s disk.
SimLex-999, MEN, MorphScore and its data, HiT’s Zenodo datasets, HyperLex, SemEval-2018 Task 9, individual GLUE tasks (QQP’s terms, for one), SentEval, HazyResearch/hyperbolicsUnknown: not found or not checkedReport only; nothing redistributed.
DOID, FoodOn (HiT transfer sets)CC0 and CC BY 4.0, from memory, not checkedLater, after a check.
Two tiers in the chain

Text travels inside a shard only when its licence allows redistribution: WordNet with its notice, UD EWT, BLiMP, public-domain Gutenberg and the team’s own text. Everything else publishes only hashes, roots, counts, SPDX licence identifiers, metrics and receipts, with salted commitments so that known text cannot be confirmed by guessing. That is attestation plus audit, not public recomputation. Explainer 4 has the records.