Hyperbolic Lexicon · Explainer 15 · Stage 2

The Lexical Claims, Decided

Stage 2 asked four questions of the exact trees and of learned embeddings, on word pairs that people rated or labelled. Amendment 13 registered the claims, their controls and their rules before any code existed. One run decided them. 9 of the 11 parts are supported and 2 are refuted: WordNet’s own graph tracks graded type-of better than the H³ tree (L12(d)), and curvature 0 does not match the exact trees on similarity and relatedness (L15(a)). Every control failed as required. Below, your browser re-reads the record and applies each registered rule again.

The decisions The record, re-read The controls What this does not show Claims ledger
decided: L12–L15 · 9 supported, 2 refuted L12(d): H3 − WN −0.160603 [−0.220516, −0.112235] L14: Lorentz ahead in 20 of 20 HiT comparisons one run, 4.63 h · 1,848 receipts
Contents
  1. Where this sits
  2. What was asked
  3. The decisions at a glance
  4. L12 · Graded type-of
  5. L13 · Detection and direction
  6. L14 · Learned Lorentz and Euclidean
  7. L15 · Similarity and relatedness
  8. The controls
  9. How it was decided
  10. What the dry sealed files showed
  11. The record, re-read
  12. What this does not show
01

Where this sits

The benchmark ladder’s Stage 2 is lexical semantics: what people judge about pairs of words. Stage 1 asked whether the exact trees are faithful copies of WordNet. Stage 2 asks whether that copy tracks human judgements, and where a flat construction or a corpus does better. This page’s focus node is Stage 2’s lexical claims.

The code is in reference/: the loaders and pins lexicon/stage2data.py, the sense rule senses.py, WordNet’s graph wngraph.py, the scores and their certified order scores2.py, the statistics and decision rules stats2.py, HiT’s rows hit.py, GloVe glove.py, and the runner run_claims_stage2.py. The browser’s re-evaluation is lexicon_stage2.js, beside the unedited lexicon_core.js, run by tests/parity-stage2.cjs.

Every number on this page is read from data/stage2_view_v1.js. make_stage2_view_v1.py writes it from the decisive record results/claims-stage2.json (summary claims-stage2.md), stage2-manifest.json, tuning-stage2.json, settings-stage2.json, the visible dry record results/dry/claims-stage2.json and the dry runs’ two sealed files, opened in step 8 and committed as gzip copies in results/dry/stage2-sealed/ (each checked against the SHA-256 that names it). Point at a number to see the file and the path it comes from. The page fills each number from that file when it loads, and the generator’s --check fails if the page as written differs. The rules are in PREREGISTRATION.md: Amendment 13 (the claims, items 1–21), 14 (the definitions after the dry run) and 15 (the frozen rates). Amendment 16, written with this page, records errata: the Node file’s name, one sentence of Amendment 14 on L14’s estimates, three readings the independent check made explicit, and a figure of Amendment 14 that the sealed file corrects. It changes no rule and no result. Where this page and those files differ, the files win.

How to read this page

supported and refuted mean what the registered rule says and nothing more. Each rule reads 95% bootstrap intervals of a difference between two arms. A prediction was registered with each claim. It is not part of the rule, and this page marks each one right, wrong or partly.

An interval is written as estimate [2.5%, 97.5%]. Decimals are rounded from the record’s values.

H3, H2
The exact trees: L3’s training-free points of every noun synset in H³ and H², with the origin o at the empty word’s point.
F3, F3*, F2, F2*
The same construction at curvature 0 (L5’s flat arms): the same words, placed in flat space. F collides on its lattice; F* shrinks each level.
WN
WordNet’s own hypernym graph, with no geometry.
G
The is-a score of a sense pair (s, t): r(s) − r(t) − d(s, t), with r the distance from o. It is at most 0, and exactly 0 when t lies on the path from o to s. A word pair takes the largest G over its senses (the maximum over senses).
Sym, Dep°
Sym is −d(s, t): it gives (x, y) and (y, x) one score. Dep° is the depth gap r(s) − r(t) of the senses G chose.
K3, S-full, W
Controls on H3’s points. K3: words from a content hash. S-fulli: the points shuffled among synsets. Wi: the senses shuffled among words of equal polysemy. Twenty of each; a control is read against its strongest permutation.
ρ
Spearman’s rank correlation with average ranks for ties, computed from exact integer sums.
Word-level interval
For a difference of ρ: it resamples words, not rows, because one word sits in many pairs. It is wider than a row bootstrap: in Amendment 14’s synthetic model (item 3) it was about 1.6–1.8 times the true spread. The row bootstrap is reported beside it and decides nothing.
AP, direction accuracy
AP: the average precision of the True rows when every row is ranked by G, with ties counted against the True row. Direction accuracy: the share of hypernym pairs with G(x, y) strictly above G(y, x).
RR
On HiT, the reciprocal rank of a child’s held-out ancestor among it and its 10 negatives, ties counted against it.
Cluster interval
For BLESS and WBLESS: it resamples the concepts the rows were built from.
02

What was asked

Four claims, registered by Amendment 13 on 2026-11-07 before any Stage-2 module existed. The Python reference decides all four.

  • L12, graded type-of. On HyperLex’s 2,163 noun pairs, rated 0–6 for “x is a type of y” and read by G: (a) H3 and H2 track the ratings better than F3, F3*, F2 and F2*; (b) G beats the depth gap Dep°, so the tree carries ancestry beyond generality; (c) G beats Sym, so it carries direction beyond similarity; (d) H3 tracks them as well as WordNet’s own graph, within ±0.02.
  • L13, detection and direction. On the hypernymy suite’s BLESS and WBLESS test folds, H3 and H2 beat curvature 0 in (a) BLESS detection AP, (b) WBLESS detection AP and (c) BLESS direction accuracy.
  • L14, learned geometry. Trained on WordNet’s 75,850 direct noun hypernym edges at d = 2 and 3, learned Lorentz embeddings beat learned Euclidean ones, both read by G: (a) on HiT’s multi-hop test, with random and with hard negatives; (b) on HyperLex’s nouns.
  • L15, similarity and relatedness. (a) Read by Sym, curvature 0 does as well as the exact trees on SimLex’s nouns and MEN’s noun pairs: the hyperbolic gain in ρ is at most 0.02. (b) GloVe’s 300-d cosine beats H3 on relatedness (MEN and WordSim’s relatedness pairs), and H3 beats GloVe on HyperLex.

Each claim has controls that must fail as required. If one does not, the parts it names are void, and a void part decides nothing. Apart from tuning’s scores on HiT’s validation split (chapter 6), no metric of any arm had been computed on any of these sets before the decisive run. One count was a partial preview: Amendment 13 counted, by HyperLex TYPE, which pairs are WordNet closure pairs before L12(d) was written. It says so (A13.20), and the record flags L12(d) as “registered after a partial preview (A13.20)”.

03

The decisions at a glance

The record’s verdicts, with the registered prediction for each part. Every control failed as required, so no part is void.

PartWhat it saysDecisionResultPredictedGraded
L12(a)H beats F on HyperLex, four cellssupported4 of 4 intervals above 0; lowest lower end +0.313005supportedright
L12(b)G beats Dep°, in H3 and H2supported2 of 2 above 0; lowest lower end +0.054986supportedright
L12(c)G beats Sym, in H3 and H2supported2 of 2 above 0; lowest lower end +0.308581supportedright
L12(d)H3 within ±0.02 of WordNet’s graphrefuted
flagged (A13.20)
H3 − WN −0.160603 [−0.220516, −0.112235]: below the bandrefuted, on the low sideright
its width: wrong
L13(a)H beats F, BLESS detection APsupported4 of 4 above 0; lowest lower end +0.624829supportedright
L13(b)H beats F, WBLESS detection APsupported4 of 4 above 0; lowest lower end +0.362911supportedright
L13(c)H beats F, BLESS directionsupported4 of 4 above 0; lowest lower end +0.304385supported; an F* cell the likeliest inconclusive oneright
F*: wrong
L14(a)Lorentz beats Euclidean on HiT test, d = 2 and 3, random and hardsupported20 of 20 seed intervals above 0; lowest lower end +0.174691supported, with a larger gap than (b)right
larger gap: partly
L14(b)Lorentz beats Euclidean on HyperLexsupported10 of 10 seed intervals above 0; lowest lower end +0.126876supportedright
L15(a)Curvature 0 does as well under Sym: gain at most 0.02refuted8 of 8 intervals have lo > +0.02; lowest lower end +0.034313refuted, at least on SimLexright
L15(b)GloVe beats H3 on relatedness; H3 beats GloVe on HyperLexsupported3 of 3 above 0; lowest lower end +0.023564supportedright

Every decision above was made by the Python reference. Amendment 13 item 1 also asks for a browser check before a page shows a Stage-2 claim as holding: the browser core must reproduce the decisive code files. It names lexicon_core.js; the integration overhaul’s decision D4 keeps that file unedited, so the code is in lexicon_stage2.js beside it. In step 8, lexicon_stage2.js re-rendered all 106 decision files of A13.14 (66 exact-arm files, 20 learned tables on HiT test and 20 on HyperLex; 12,685,235 lines) from the pinned word lists, WordNet and the committed snapped tables, and every file’s text matched the manifest’s SHA-256. So these decisions are shown as holding. Those 106 files are the group A13.14 names: the exact-arm files that any of the 141 decided comparisons reads, and the measured tables’ files on HiT test and HyperLex. Files that only controls read were not re-rendered in Node: L14’s edge-shuffled tables (control 3 compares them in 60 of the 141 comparisons), its tuning tables and relabelled points (controls 1 and 2), and GloVe’s files for the controls 3 of L13 and L15. The independent check reproduced those with its own Python code (chapter 12; Amendment 16 item 1).

04

L12 · Graded type-of on HyperLex

HyperLex asks people how much x is a type of y. The exact trees track those ratings far better than their flat twins, and their score G carries both ancestry and direction. But WordNet’s own graph tracks them better still.

Each arm scores the 2,163 noun pairs (they hold 1,843 distinct words), and each score is correlated with the mean rating.

Arm (G, maximum over senses)ρNote
H30.624924the exact tree in H³
H20.620716the exact tree in H²
F30.173168curvature 0
F3*0.216920
F20.054840
F2*0.211128
WN0.785527WordNet’s graph, no geometry
K30.097928control: no hierarchy
H3 by Dep° · by Sym0.450602 · 0.225535for (b) and (c)
H2 by Dep° · by Sym0.452892 · 0.210319
PartDifferenceWord-level interval (decides)Row bootstrap (reported)
(a) supportedρ(H3) − ρ(F3)+0.451756 [+0.366321, +0.537456][+0.412466, +0.490905]
ρ(H3) − ρ(F3*)+0.408004 [+0.317121, +0.502786][+0.371912, +0.443928]
ρ(H2) − ρ(F2)+0.565876 [+0.468303, +0.652605][+0.522615, +0.608590]
ρ(H2) − ρ(F2*)+0.409588 [+0.313005, +0.506924][+0.372121, +0.446802]
(b) supportedH3: ρ(G) − ρ(Dep°)+0.174322 [+0.061545, +0.292732][+0.125265, +0.222235]
H2: ρ(G) − ρ(Dep°)+0.167824 [+0.054986, +0.287257][+0.118842, +0.215620]
(c) supportedH3: ρ(G) − ρ(Sym)+0.399389 [+0.308581, +0.487686][+0.365521, +0.433175]
H2: ρ(G) − ρ(Sym)+0.410397 [+0.320309, +0.497834][+0.376480, +0.444836]
(d) refutedρ(H3) − ρ(WN)−0.160603 [−0.220516, −0.112235][−0.182059, −0.140491]

The predictions

  • right ρ(H3) between 0.45 and 0.70: it is 0.624924.
  • right Every F arm at least 0.10 below ρ(H3): the smallest gap is 0.408. In F a bent path’s deficit grows with its legs and the lattice collides, so ancestors at one depth mix with non-ancestors at another.
  • right right right (a), (b) and (c) supported.
  • right (d) refuted on the low side. The direction was right.
  • wrong The size was not. The word-level half-width was predicted at 0.015–0.026; it is 0.054 (the row bootstrap’s is 0.021). The estimate, −0.160603 [−0.220516, −0.112235], is also far larger than the 0.02–0.04 the prediction put on the 46 hyp pairs whose closure link is not a spanning one; it said the bends would add to that, without a size.

Why (d) is refuted, and what that rests on

The interval lies entirely below the band [−0.02, +0.02], which is what refutes (d): its upper end, −0.112235, is 0.092235 below the band’s low end. Its lower end, −0.220516, passes that end by 0.200516 (the record’s failing excess). A chain of reported arms shows where the gap opens. Each step keeps G and the maximum over senses and changes one thing:

Step (reported)ρWhat changes
WN0.785527WordNet’s graph, with its multiple parents
WN-U0.715703aside: the same graph, read without direction
ST0.738586the spanning tree: one parent each (0.047 below WN)
letter tree0.733715the address words’ letter tree (0.005 below ST)
H30.624924the words placed in H³ (0.109 below the letter tree)

Most of the gap opens in the last step. That is where the embedding bends a path, and a spanning ancestor then loses a little G. Amendment 13 named this loss but did not size it.

The tie reading behind (d)

WN’s G takes only 13 distinct values on these pairs; H3’s takes 1,939. ρ is decided with average ranks for ties (A13.7). Breaking each arm’s ties against the ratings gives the tie-pessimistic ρ, and breaking them with the ratings the tie-optimistic one. WN’s range is 0.495551 to 0.933593; H3’s is 0.620763 to 0.628662. Under the pessimistic reading H3 would lead WN. A13.7 chose average ranks in advance, with this case named as the reason: a pessimistic reading would decide (d) by the arms’ tie counts. So this is not a deviation, but (d) does rest on it.

G’s known limits show too. Two words that share a synset score G = 0, the largest value. In H3 those pairs form the top class on their own, above every ancestor pair, whose G the embedding’s bends push below 0; in WN, where an ancestor also scores 0, they tie. Either way synonyms rank with or above the type-of pairs. Within the hyp strata, G runs against the ratings: H3’s ρ in hyp-1 to hyp-4 is −0.066406, −0.046884, −0.221226 and −0.188723 (reported), because every ancestor sits near 0.

05

L13 · Detection and direction on BLESS and WBLESS

The exact trees find hypernym pairs and their direction well. Their flat twins sit near chance, and F2 and F2* fall below it in places: F2’s BLESS AP is 0.091665 against a chance level of 0.096563, and F2* picks the right direction on 0.438023 of the pairs. So H − F here mostly measures H against a near-chance baseline, and the gaps describe F as much as H.

BLESS’s test fold has 12,365 covered rows, 1,194 of them True, in 200 concept clusters. WBLESS has 1,493 rows, 764 True, in 192 clusters. Direction uses BLESS’s 1,194 True rows.

Arm (G)BLESS APWBLESS APBLESS direction (rows right)
H30.7892380.9701570.979899 (1,170)
H20.8010700.9706110.983250 (1,174)
F30.1055010.5694750.628141 (750)
F3*0.1365830.5574910.603015 (720)
F20.0916650.5158040.518425 (619)
F2*0.1215920.5368720.438023 (523)
share of True rows (chance AP)0.0965630.511721—
Cell(a) BLESS AP(b) WBLESS AP(c) BLESS direction
H3 − F3+0.683738 [+0.654393, +0.713618]+0.400682 [+0.362911, +0.432217]+0.351759 [+0.304385, +0.399163]
H3 − F3*+0.652656 [+0.624829, +0.681090]+0.412666 [+0.385307, +0.439142]+0.376884 [+0.343593, +0.410442]
H2 − F2+0.709405 [+0.681430, +0.737730]+0.454806 [+0.426190, +0.480855]+0.464824 [+0.414858, +0.514643]
H2 − F2*+0.679478 [+0.652498, +0.706789]+0.433739 [+0.408648, +0.457732]+0.545226 [+0.513709, +0.576344]
Decisionsupportedsupportedsupported

The predictions

  • right right right (a), (b) and (c) supported.
  • wrong “F* is the likeliest inconclusive cell of (c)”. Both F* cells are well above 0 (lowest lower end +0.343593). F2* gets the direction right on 523 of the 1,194 pairs, below one half, and F2 on 619. The reason the prediction gave, that a prefix’s radius is usually below its extension’s, does not show here.
The flat arms are close to chance

On BLESS the F arms’ AP lies at 0.091665 to 0.136583, beside the chance level 0.096563; on WBLESS at 0.515804 to 0.569475, beside 0.511721. So the decided gaps compare H with a near-chance baseline. The controls (chapter 8) compare H with arms that lose the hierarchy, and those gaps are just as large. For orientation only, as A13 registered it: Roller et al.’s spmi scores .76 / .96 / .96 for BLESS AP, WBLESS AP and BLESS direction, and HypeCones .81 / .98 / .94; H3 scores 0.789238 / 0.970157 / 0.979899. These are not comparable: their folds were never released, uncovered rows are dropped here, and the trees are built from WordNet, which also built most of the positives.

06

L14 · Learned Lorentz against learned Euclidean

Both geometries were trained on WordNet’s direct noun edges only, then asked about ancestors two or more steps up. Lorentz wins every comparison, at both dimensions, by wide margins.

The rows of HiT’s test (29,383) each hold a child, an indirect ancestor held out of training, and 10 random and 10 hard negatives. HiT draws the hard negatives from the child’s siblings. (b) reads the same tables on the 2,153 HyperLex rows whose words lie in the training graph. Five seeds (0–4) per geometry and dimension, rate 10 everywhere (Amendment 15).

CellLorentzEuclideanLorentz − EuclideanDecision
(a) d = 2, random · mean RR0.9565900.660182+0.296408supported
(a) d = 2, hard · mean RR0.8044740.336069+0.468405supported
(a) d = 3, random · mean RR0.9557190.749144+0.206574supported
(a) d = 3, hard · mean RR0.7265630.335124+0.391439supported
(b) d = 2 · mean ρ on HyperLex0.4820370.157682+0.324355supported
(b) d = 3 · mean ρ on HyperLex0.4940370.238121+0.255916supported

Means over the five seeds. A cell is supported when all five seeds are present and each seed’s interval lies above 0. All 20 HiT intervals and all 10 HyperLex intervals do:

(a) Seedd = 2, randomd = 2, hardd = 3, randomd = 3, hard
0+0.285316 [+0.281157, +0.289430]+0.450051 [+0.445046, +0.455000]+0.205747 [+0.201975, +0.209611]+0.391719 [+0.386640, +0.396831]
1+0.316292 [+0.312273, +0.320375]+0.486315 [+0.481581, +0.491007]+0.215865 [+0.212072, +0.219685]+0.409492 [+0.404350, +0.414578]
2+0.279870 [+0.275727, +0.283902]+0.446015 [+0.441095, +0.451009]+0.178339 [+0.174691, +0.182072]+0.381627 [+0.376357, +0.387048]
3+0.293680 [+0.289605, +0.297714]+0.473132 [+0.468279, +0.477967]+0.210631 [+0.206748, +0.214585]+0.386223 [+0.380943, +0.391429]
4+0.306881 [+0.302928, +0.310840]+0.486514 [+0.481788, +0.491259]+0.222290 [+0.218271, +0.226285]+0.388133 [+0.383054, +0.393225]
(b) Seedd = 2d = 3
0+0.316821 [+0.212232, +0.420832]+0.220881 [+0.126876, +0.309236]
1+0.285241 [+0.186487, +0.386932]+0.254431 [+0.174591, +0.331099]
2+0.366185 [+0.257133, +0.480721]+0.292658 [+0.200027, +0.376545]
3+0.313935 [+0.203011, +0.425013]+0.256050 [+0.146138, +0.368295]
4+0.339592 [+0.239928, +0.434629]+0.255562 [+0.174471, +0.333623]

The predictions

  • right right (a) and (b) supported. Amendment 13 held this with low confidence and named inconclusive as a likely alternative. The outcome is cleaner than predicted.
  • partly “A larger gap in (a) than in (b)”. It holds in 2 of the four HiT cells: with hard negatives (+0.468405 and +0.391439, against (b)’s +0.324355 and +0.255916), not with random ones (+0.296408 and +0.206574). The two gaps are in different units, RR and ρ, so the comparison is loose either way.

Against tuning’s preview

Tuning scored HiT’s validation rows, at other seeds, so it previewed (a) before the measured runs (Amendment 15 item 3). The test agrees with it in every respect it showed: Lorentz ahead everywhere, more so with hard negatives, more so at d = 2. (b) had no preview.

CellValidation gap (tuning, rate 10)Test gap (decisive)
d = 2, random+0.321695+0.296408
d = 2, hard+0.472652+0.468405
d = 3, random+0.166207+0.206574
d = 3, hard+0.366662+0.391439
Two things to know

Against hard negatives, Euclidean barely separates the ancestor. Training never sees the negatives; the same tables are read in both settings. Its mean RR with hard negatives is 0.318469 to 0.365971 per table. At d = 3, seed 0, its true table is even slightly worse than its edge-shuffled partner (chapter 8, control 3). Amendment 13 registered control 3 so that this case does not void the part.

(b) is not held out. HyperLex drew many of its pairs as WordNet edges: 418 of the 2,153 rows are training edges for some sense pairing (reported, as A13 asked). So (b) measures how training on the edges carries over to graded judgements, not generalisation to unseen pairs.

07

L15 · Similarity and relatedness

Phase 0 expected a flat distributional space to win here. With the same WordNet information in both arms, it does not: the exact trees beat curvature 0 by much more than 0.02, so (a) is refuted. Against a corpus space, each side wins where it should.

Arm (Sym, maximum over senses)SimLex nouns (666)MEN noun pairs (2,005)
H30.5891850.394799
H20.5826090.400265
F30.3373410.156337
F3*0.3634840.301520
F20.1912050.122020
F2*0.3606260.294601
(a) refutedSimLex: interval (decides)row bootstrapMEN: interval (decides)row bootstrap
ρ(H3) − ρ(F3)+0.251843 [+0.146815, +0.356256][+0.194029, +0.310083]+0.238462 [+0.168336, +0.309502][+0.202081, +0.276354]
ρ(H3) − ρ(F3*)+0.225700 [+0.119025, +0.333397][+0.167211, +0.286681]+0.093279 [+0.034313, +0.155472][+0.065183, +0.121356]
ρ(H2) − ρ(F2)+0.391403 [+0.261742, +0.519386][+0.319619, +0.463000]+0.278245 [+0.192758, +0.362548][+0.234619, +0.321670]
ρ(H2) − ρ(F2*)+0.221983 [+0.114264, +0.329887][+0.162969, +0.283409]+0.105665 [+0.043908, +0.170315][+0.076366, +0.134678]

(a) would be supported if every interval had hi ≤ +0.02 and is refuted if any has lo > +0.02. All 8 do, so the verdict does not rest on one cell. The tightest is MEN H3 − F3*, with lower end +0.034313. The word-level intervals are the wide ones; the row bootstrap’s, reported, are narrower still.

Why. The flat lattice collides: F3 has 42 distinct values on SimLex’s pairs against H3’s 420, and 97 on MEN’s against 1,730. Pairs at different WordNet distances get the same flat distance. F* avoids the collisions (573 and 1,890 values) but compresses depth; it comes closer to H on MEN.

(b) supportedρ(GloVe)ρ(H3, Sym)Difference, word-level interval
MEN within GloVe (2,005): GloVe − H30.7605170.394799+0.365718 [+0.276143, +0.455861]
WordSim relatedness (245): GloVe − H30.580236-0.004288+0.584524 [+0.284892, +0.858283]
HyperLex within GloVe (2,162): H3 − GloVe0.1164950.226030+0.109535 [+0.023564, +0.193084]

The predictions

  • right “(a) refuted, at least on SimLex.” It is refuted on MEN as well.
  • right “On MEN the gain is smaller”: each MEN estimate is below its SimLex twin.
  • wrong “MEN’s word reuse widens its intervals.” 0 of the four MEN intervals are wider than SimLex’s: their widths are 0.121 to 0.170 against SimLex’s 0.209 to 0.258. MEN has three times the rows, which outweighs its reuse.
  • right (b) supported. HyperLex’s leg, predicted less certain, is the narrowest: lower end +0.023564. On WordSim’s relatedness pairs H3 by Sym carries almost nothing (-0.004288): WordNet holds taxonomy, a corpus holds association.

Read with L12(a): curvature makes the tree a more faithful copy of WordNet for similarity and relatedness too, not only for type-of. One caution from sets that decide nothing: on WordSim’s small sets the gain over F* is not visible. For example H3 − F3* is −0.013690 [−0.159305, +0.123223] on WordSim combined and −0.041615 [−0.240357, +0.153700] on its relatedness pairs.

08

The controls, and what they rule out

Each control breaks one thing a claim depends on and must then show the weakness. All of them did, so no part is void. L14’s closure check, a positive control, passed.

ControlWhat must happenMeasured
L12 1 · hierarchy removedH3 − K3 and H3 − S-full* above 0K3 +0.526997 [+0.396770, +0.644590]; S-full-18 +0.531326 [+0.432922, +0.623247]
L12 2 · senses scrambledH3 − W* above 0W-10 +0.576757 [+0.473004, +0.672506]
L12 3 · roles reversedH3 read as G(x, y) minus H3 read as G(y, x), above 0+0.722705 [+0.606055, +0.827362]
L12 4 · ratings shuffled|mean ρ| of 20 shuffles ≤ 0.024045 for each of seven armslargest 0.014040 (F3)
L13 1 · hierarchy removed, senses scrambledH3 − K3, − S-full*, − W* above 0 in (a), (b) and (c)9 of 9 above 0; lowest lower end +0.309210
L13 2 · labels shuffledmean AP of 20 shuffles ≤ share of True + 1/50, exactlyacross the six arms, BLESS 0.095542–0.098119 against 0.116563; WBLESS 0.508088–0.520061 against 0.531721; 12 of 12 hold
L13 3 · a symmetric score on directionaccuracy exactly 0: every pair ties, and a tie is wrongSym on H3 0; GloVe 0
L14 1 · labels shuffled within each rowmean RR ≤ H11/11 + 1/50 = 0.294534 for every trained table88 of 88 hold; largest 0.278323 (shuffled-euclid-d3-lr10-s0)
L14 2 · points relabelledmacro mean RR ≤ the same band, for every trained table88 of 88 hold; largest 0.287305 (shuffled-lorentz-d3-lr10-s3)
L14 3 · edge-shuffled trainingtrue − shuffled above 0 in at least one geometry, per d, metric and seed30 of 30 pass; 1 cell “true ≈ shuffled”
L14 4 · ratings shuffled|mean ρ| ≤ 0.024101 per measured tablelargest 0.010623 of 20
L14 closure check (must pass)closure of the training edges ranks every test row firstfailing rows: random 0, hard 0
L15 1 · hierarchy and senses visibleH3 − K3, − S-full*, − W* above 0 on SimLex and MEN6 of 6 above 0; lowest lower end +0.239137
L15 2 · ratings shuffled|mean ρ| within 5/√(20(n − 1)) for every arm and set100 of 100 hold; closest MEN S-full-13, 0.012653 against 0.024975
L15 3 · GloVe vectors shuffled|mean ρ| within the same bandMEN −0.005602, WordSim −0.006289, HyperLex 0.002299

Two controls that worked as designed

L14 control 2 needed its macro mean. Under a relabelling, every row whose positive is one synset shares that synset’s new point, and a few general synsets hold large shares of the rows. Amendment 13 therefore averaged first over each positive synset. The per-row mean, reported, exceeds the band in 21 of the 88 tables (5 of them measured), up to 0.392249. The registered macro mean stays inside the band in every table. Under a per-row reading, L14 would have been void.

L14 control 3 needed “at least one cell”. At d = 3, seed 0, hard negatives, the true Euclidean table scores −0.006931 [−0.011760, −0.002228] against its edge-shuffled partner: “true ≈ shuffled”. The Lorentz cell there is +0.494018 [+0.489724, +0.498405]. Amendment 13 wrote this rule so that a losing arm that learns nothing does not void the part, which would shield the prediction from refutation. Under Phase 4’s every-cell reading (A10.9), (a) would have been void.

The strongest permutations. Each S-full and W control is read against the one of its 20 permutations with the largest score. For L12 those are S-full-18 (ρ 0.093598) and W-10 (ρ 0.048168); for L15 on SimLex S-full-12 and W-10, on MEN S-full-15 and W-18.

09

How it was decided

Definitions first, then code, a review, a dry run with its human-rated values sealed, the definitions amendment, tuning, the amendment that froze its rates, and one decisive run on the code as committed (A13.15). Times are UTC, from git and the records.

  1. b3a5003 · 2026-11-07 07:13 · Amendment 13, committed alone. The four claims, their arms, scores, statistics, controls, predictions and order of work, before any Stage-2 module existed. Before it, surveys downloaded and counted the candidate sets but scored nothing; one count was a partial preview of L12(d), named in A13.20.
  2. 613721d, e91e02d, 841b90c · 09:41–13:38 · The code and its review. Built on synthetic data, WordNet’s verbs and unregistered seeds. An adversarial review with independent implementations of the certified order, the statistics, the data and the runner found no defect in the decisions; its minor findings were fixed. The decision wiring was pinned to Amendment 13’s own sentences; the run checks its 141 decided comparisons and 321 control values against them.
  3. 05d467d · 14:40 · The dry run on WordNet’s verbs, with the dry seeds. Every value on a human-rated or human-labelled set went into a sealed file, unopened. The visible record showed three controls that did not fail as required (L14 controls 1 and 2, L15 control 1), and dry tuning chose the grid’s top rate in every cell.
  4. 513e079 · 17:16 · Amendment 14, the definitions after the dry run. It changed no definition. It stated the runner’s readings, froze the instance manifest, and diagnosed the three controls as chance and size at the dry sizes. It accepted the registered risk of a void rather than loosen a band after seeing a dry outcome.
  5. 9550eea, 5acbbb7 · 17:50–18:47 · One code commit and the dry rerun. The commit scores every S-fulli and Wi in L12’s reported grid and checks HiT validation’s counts; no decided comparison changed. The rerun reproduced the first dry run’s every table, file, count, visible value and control outcome, and wrote a new sealed file. The first was moved aside unopened. A test-only fix (93a5a8e) followed.
  6. b2ba411 · 18:48 · The timing projection: a projected 9.93 h in all (the dry run’s hour included) against an 11 h limit, so tuning could start.
  7. 60345b4 · 2026-11-08 00:43 · Registered tuning, at b2ba411: 48 runs (four cells, six rates, two seeds each), scored on HiT’s 29,383 validation rows. It took 3 starts. Start 1 (18:48:40) was stopped on purpose within seconds, before any job finished, because it ran under a time limit that would have killed it. Start 2 (18:49:01) ran until the container was suspended for inactivity; the restore ended its processes, and the jobs in flight were lost. Start 3 (22:22:12) reused the finished jobs, checked by SHA-256 (19 tuning runs and the replay), and ran the rest. A run’s bits depend only on its seed and the code, so the restarts changed no value. 24 of the 48 runs converged; none diverged; the replay with hpstep.prng.Rng ended equal.
  8. 6b6ec09 · 01:21 · Amendment 15, the rates frozen in settings-stage2.json: 10 in every cell, the grid’s top. The grid is never extended, so where the scores would peak is unknown. The closest choice was Euclidean d = 2, by 0.033031. The convergence gate decided both Euclidean cells: their slowest rate scored higher but had not left its starting plateau.
  9. The decisive run, started 2026-11-08 01:21:12 UTC at 6b6ec09, in one start. It refused unless only the four allowed paths had changed since tuning. It checked every count, rebuilt the instance manifest equal, replayed one measured run (equal), retrained the two pre-flight jobs (equal to tuning’s), computed L14’s relabelling for the first time, and wrote 1,635 code files (172,892,141 lines). 40 of its 42 training runs converged; the two that did not are the pre-flight retrains of rate 0.03, as in tuning. 4.63 h.
  10. 5d32f47 · 05:59 · The record, as it came out: claims-stage2.json and .md, 1,848 LBENCH/1 receipts, the manifest of every code file’s SHA-256, the console, and the 20 measured snapped tables (26.2 MB). The decision code files total 68.8 MB, above A13.14’s 50 MB rule, so they are committed by hash only.
  11. Step 8, this write-up. An independent check recomputed every decided value from the code files and the data with its own code and found the record equal. The Node re-evaluation reproduced the decision files (chapter 3). The sealed files were opened and committed (chapter 10). Amendment 16 records the errata.
What the independent check made explicit (Amendment 16)

None changes a value or a decision. The first corrects a sentence of Amendment 14; the other two are conventions the registration leaves open, now stated.

  • L14’s estimates. Amendment 14 item 1 says an interval’s estimate is the exact difference evaluated at 60 digits and rounded once to binary64. That holds for the ρ, AP and accuracy differences. For the intervals over HiT’s rows (L14(a) and the RR cells of control 3), the record reports rank.ci_mean’s own estimate instead: the float64 mean of the per-row differences, each RR rounded first, as A13.7 registered (“rank.ci_mean, unchanged”). The two differ in the last bit or two. Amendment 16 corrects Amendment 14’s sentence.
  • L13’s AP intervals. A resampled AP sums its terms in float64. The record sums them with numpy’s pairwise sum; a correctly rounded sum moves a few interval ends by a bit or two, none at six decimals. A13.7 says only “in float64”.
  • The tie bounds of ρ (reported) break every tie of the arm, including rows tied in both the arm and the ratings.

Tuning at the grid’s top

The score of a rate is the mean of its two runs’ RR with random and hard negatives. Only rates whose runs both converged are eligible. In every cell the eligible means still rise at 10:

Cell0.030.10.31310Frozen
Lorentz d = 2———0.5874140.7066170.87412310
Lorentz d = 3——0.5892690.6699000.7329690.84524010
Euclidean d = 2————0.4439190.47694910
Euclidean d = 3———0.5060040.5102260.57880510

A dash marks a rate whose runs did not converge in 1,000 epochs. Bold: the frozen rate.

10

What the dry sealed files showed

The dry run used WordNet’s verbs, so that nothing decided was seen. Its values on human-rated sets stayed sealed until the decisive run had finished (A13.13). Step 8 opened both sealed files and committed them as gzip copies (the rerun’s, the first run’s), so the dry values here come from committed files.

Each file has the SHA-256 that names it: the rerun’s is named in the visible dry record, the first run’s in Amendment 14. Their claims are equal. The first holds 620 sealed LBENCH/1 receipts and the rerun 1,180, the difference being the new grid cells. The dry sets were small stand-ins: HyperLex’s 452 verb pairs (also the detection stand-in for BLESS and WBLESS), SimLex’s 222 verb pairs, MEN’s 29 verb–verb pairs (22 words), and SimLex’s verbs within GloVe for WordSim.

PartDry, before controlsDryRegistered
L12(a)supportedsupportedsupported
L12(b)supportedsupportedsupported
L12(c)inconclusiveinconclusivesupported
L12(d)inconclusiveinconclusiverefuted
L13(a)supportedsupportedsupported
L13(b)supportedsupportedsupported
L13(c)supportedsupportedsupported
L14(a)supportedvoidsupported
L14(b)supportedvoidsupported
L15(a)refutedvoidrefuted
L15(b)inconclusiveinconclusivesupported

L14(a)’s dry values, on the verb graph and its split, and its controls there (1, 2, the closure check and control 3’s RR cells) are WordNet data and were visible all along; controls 1 and 2 voided it (87 and 79 of 88 tables held). L14(b), on HyperLex’s verbs, was sealed with its ratings-based controls, like every other dry value in this table. Before the voids, three dry parts differ from the registered ones. On the verbs, G gains little over Sym (L12(c): H3 +0.070389 [−0.018617, +0.171168]). WordNet is level with H3 on the verbs (L12(d): −0.008904 [−0.069285, +0.047977]) and far ahead on the nouns. L15(b) was inconclusive on two legs. The WordSim stand-in (SimLex’s verbs within GloVe) is a similarity set, on which H3 leads GloVe in estimate only: GloVe − H3 is −0.235418 [−0.512251, +0.052860], an interval that crosses 0. And the 29-row MEN leg, +0.269150 [−0.627574, +1.132392], says nothing either way.

L15 control 1: Amendment 14’s diagnosis, checked

The visible dry record said only that L15 control 1 did not fail as required. Amendment 14 diagnosed size, “almost surely”: MEN’s stand-in has 29 rows, where a word-level interval is very wide. The sealed values bear that out. All three MEN legs failed, and all three SimLex legs held:

LegDry (verbs)Half-widthRegistered (nouns)Half-width
SimLex H3 − K3+0.382158 [+0.083644, +0.676847]0.297+0.537568 [+0.374862, +0.697053]0.161
SimLex H3 − S-full*+0.289614 [+0.034003, +0.537366]0.252+0.495191 [+0.354274, +0.635538]0.141
SimLex H3 − W*+0.410633 [+0.127721, +0.690491]0.281+0.611906 [+0.449475, +0.764111]0.157
MEN H3 − K3+0.039902 [−0.832059, +0.805202]0.819+0.408242 [+0.296270, +0.517291]0.111
MEN H3 − S-full*−0.085806 [−1.025030, +0.672455]0.849+0.339798 [+0.239137, +0.440552]0.101
MEN H3 − W*−0.155639 [−0.723732, +0.455424]0.590+0.354840 [+0.249738, +0.459061]0.105

Two details differ from Amendment 14’s figures. It gave a word-level half-width of 0.8–0.9 on MEN’s stand-in; the W* leg’s is narrower, and so are the four dry L15(a) cells on MEN (0.655 to 0.744). Amendment 16 item 4 records the measured half-widths. And its model gave SimLex at least a 0.75 chance of voiding the control when ρ(H3) is at most 0.4; the dry ρ(H3) on SimLex was 0.387478, and all three SimLex legs held, the less likely outcome. Neither changes the diagnosis.

Dry beside registered (verbs against nouns: orientation only)

ValueDry (verbs)Registered (nouns)
L12 ρ: H3 / WN0.644285 / 0.6531890.624924 / 0.785527
L12(a) H3 − F3*+0.211185 [+0.093381, +0.340704]+0.408004 [+0.317121, +0.502786]
L12(b) H3+0.640349 [+0.443096, +0.834050]+0.174322 [+0.061545, +0.292732]
L12(c) H3+0.070389 [−0.018617, +0.171168]+0.399389 [+0.308581, +0.487686]
L12(d) H3 − WN−0.008904 [−0.069285, +0.047977]−0.160603 [−0.220516, −0.112235]
L13 AP of H3 (BLESS; dry: the stand-in)0.6490650.789238
L13(a) H3 − F3+0.232616 [+0.186297, +0.283994]+0.683738 [+0.654393, +0.713618]
L13 direction: H3 / F30.865000 / 0.4950000.979899 / 0.628141
L14(a) d = 2, random: per-seed estimates+0.229188 to +0.262857+0.279870 to +0.316292
L14(b) d = 2: per-seed estimates+0.179040 to +0.242879+0.285241 to +0.366185
L15 ρ(H3) by Sym: SimLex / MEN0.387478 / 0.1816840.589185 / 0.394799
L15(a) SimLex H2 − F2+0.349395 [+0.106964, +0.589871]+0.391403 [+0.261742, +0.519386]
L15(b) HyperLex, H3 − GloVe+0.355078 [+0.177772, +0.526120]+0.109535 [+0.023564, +0.193084]
11

The record, re-read in your browser

This panel reads data/stage2_view_v1.js, checks its SHA-256, and applies each registered rule to the record’s intervals again. It also re-grades the predictions and re-checks the controls’ bands. It decides nothing: the Python reference decided, and the files win.

Live · the decisions, recomputed from the record’s intervals
Loading the view…
PartRegistered ruleThe rule, applied hereThe record
Try it
  • Pick L12 and tick the row bootstrap. Every interval narrows (for (d), the half-width falls from 0.054 to 0.021), and no verdict changes.
  • Pick L15. Its threshold is +0.02, not 0. The MEN F* cells sit closest to it, still above.
  • Pick L14. Each line is one seed of one cell; the rule needs all five of a cell above 0.
What this panel is, and is not

It re-applies the rules to intervals the record already holds. It does not resample anything, and it cannot recompute a ρ or an AP: the rated sets are not committed (most state no licence), only their hashes. The browser’s own check of the scores is the Node re-evaluation of the code files (chapter 3).

12

What this does not show

Amendment 13 item 21 listed what none of this decides. The results add a few limits of their own.

  • Discovery, or knowledge beyond WordNet. Every arm reads WordNet, and HyperLex’s TYPE, most of BLESS’s positives and all of HiT were built from it. L12–L14 say what curvature and learning do with the same information. L15(b) compares two sources of information, not two geometries.
  • Senses in context, verbs and adjectives, words outside WordNet. The decided sets are nouns. Verbs appear only in the dry run.
  • Learned geometry beyond this setting. Tables trained on the closure or all nouns, d ≥ 5, the instance-only synsets, other trainers, and readings from a base point other than the origin. The rates sit at the grid’s top, and where the scores would peak is unknown.
  • Comparison with published numbers. Subsets, WordNet versions, folds, rules for missing words and trainers all differ. Every literature number here is orientation.
  • The sets not decided: SemEval-2018 Task 9, HiT’s mixed-hop split, LEDS, EVALution, Shwartz and BiBLESS (the last four reported only).
  • Whether hyperbolic geometry helps a language model. That is for Stages 3 and 4, by amendment.
  • Exactness. G is ordered by square roots certified at 120 and 240 digits, not computed in rationals. Values within the 240-digit bound count as ties; that rule is part of the metric (A13.6).
  • One run, one machine. One decisive run in one container. The word-level intervals are conservative (Amendment 14 item 3); L12(d) and L15(a) are refuted even so.
  • L12(d) rests on average ranks. A13.7 registered them, and the reason; under tie-pessimistic ρ the order of H3 and WN would flip (chapter 4).
  • L13 compares H with near-chance flat arms. The large gaps describe F as much as H.
  • The browser check covers A13.14’s decision files. The Node re-evaluation reproduced all 106: the exact-arm files the decided comparisons read and the measured tables’ files. Files that only controls read were not re-rendered in Node: L14’s edge-shuffled tables (compared in control 3), its tuning tables and relabelled points (controls 1 and 2), and GloVe’s files for the controls 3 of L13 and L15. Amendment 16 item 1 lists every file not rendered. The independent check reproduced these with its own Python code.