Contents
Where this sits
This page expands answer 5 on the hub: could we grow biochains of language pre-processing data as SHD-CCP chains, as BioChain-AI grows biochains? Its focus node is the Records layer, where every table the front end produces becomes a hashed, signable commitment.
Phase 0 is analysis, seven measurement spikes and a preregistration. No grower, record, manifest or importer code existed then; Phase 3 has since built them and decided claims L8–L11 (Explainer 11; README, roadmap). What did exist: spike S4 grew WordNet gloss shards against Owl Academy’s own code, and spike S7 signed and verified a record in BioChain-AI’s attestation format in two languages. Their numbers are spike results, run on data disjoint from the registered instances, and they decide nothing (S4 output, S7 output). The background comes from a read-only survey of Owl Academy at 7f45f35 and BioChain-AI at c975161, re-checked by an adversarial cross-check whose corrections win.
exact: integers, bytes or hashes, identical in every engine. float: floating point, rounded at every step. REGISTERED: a claim in PREREGISTRATION.md, fixed and undecided; SUPPORTED: decided under its registered rule, as this page’s four claims were in Phase 3 (each claim’s box ends with the result; Explainer 11). estimate: arithmetic on measured numbers, not a measurement. Timings come from the spikes’ “~” lines, which depend on this container (4 CPU cores, Node 22.23.4); the Phase 3 values come from its decisive run, on the same container.
- Biochain
- One record that holds a UTF-8 text of at most 60,000 bytes, rebuilds it exactly, and carries hashes that commit to it. Owl Academy and BioChain-AI grow them in the browser.
- Chunk, frame
- The text is cut into chunks of 720 bytes. Each becomes a frame: a 64-bit packet, a 32-bit length and a residual.
- Residual
- The bits an order-2 rank coder writes for a chunk: for each byte, its rank among the bytes that followed the same two bytes before. Decoding replays the ranks, so the chunk comes back exactly.
- Leaf, Merkle root
- A leaf is one chunk’s SHA-256 hash. The Merkle root hashes the leaves in pairs, level by level, to one value: change any chunk and the root changes.
- Typed commitment
- A pipe-joined string with a kind and version first, such as
GROWN/1|chainId|merkleRoot|…. Its SHA-256 is what gets signed. - Attestation
- An ECDSA P-256 signature over an
ATTEST/1|…payload that contains a commitment’s hash: it says which key committed to the record. - Shard, manifest
- A shard is one biochain holding one slice of a corpus. A manifest is a record that commits to a list of shards, in order.
Biochains today
The biochain is the working SHD-CCP: the code Owl Academy and BioChain-AI actually run. Its pure codec is the same in both repositories and in Engram Mind’s Eye’s pinned copy, biochain_v1.js (Explainer 2).
How growBiochain grows one
- Cut. The text’s UTF-8 bytes are cut into 720-byte chunks. Above
MAX_GROW_BYTES = 60000it throws. - Code. Each chunk goes through the order-2 rank coder, which starts fresh for every chunk and writes Elias-gamma codes, with an 8-bit escape for a byte not seen in its context.
- Crystallize. Each chunk is folded into one 64-bit word of form 9, a “crystal”: form (bits 63..60), even parity (59), spin, a 32-bit digest of the chunk read as four int8 quaternion codes, the chunk length in the 16-bit payload, and a byte-sum checksum in the last 8 bits.
- Hash. Leaf i is SHA-256 of
i + "|" + sha256Hex(String.fromCharCode(...chunk)), over UTF-8. The leaves build a binary Merkle tree that duplicates the last node on odd levels. - Weave. Each chunk’s crystal becomes a unit quaternion; their ordered product is the chiral weave, and the conjugates in reverse order give the mirror weave float.
- Name.
chainId = "BC-" + SHA-256(merkleRoot + "|" + weaveChiral), first 24 hex digits. Frames are concatenated into one hex string, theshdccp/2stream. - Engrams (on by default). One 64-bit engram/1 key per word token, recomputed from the text on verification: the
engramStream.
| Field | What it holds |
|---|---|
| chainId | “BC-” and 24 hex digits, from the root and the chiral weave |
| stream, streamVersion | The frames as one hex string; "shdccp/2", the only version marker |
| leafHashes, merkleRoot | One SHA-256 hex per chunk, and their root |
| weaveChiral, weaveMirror | Four components each, printed with toFixed(12) and joined by commas |
| segCuts, segments | Seven weave segments, as a flat array of 8 cuts (Firestore rejects nested arrays) |
| engramStream, engramCount, engramMeanJ | Optional: 16 hex digits a token, the token count, the mean torsion J |
| origBytes, shippedBytes, value | Text bytes; stream + engrams + 32 B a leaf + 40; value = origBytes / shippedBytes |
Publishing adds the owner, the grower, a lineage list and the certification block. In BioChain-AI that block is grownAttestation; in Owl Academy it is grownSeal. The content fields agree in all three codebases; the certification names and the Firebase projects differ.
What verifyIntegrity checks
Stopping at the first failure: each packet’s parity, decoding every residual, each leaf, the rebuilt Merkle root, the chiral weave, and the engram layer recomputed from the rebuilt text. Three things it does not do, which matter later (chapter 9): leaves and root are checked only when the fields are present; a packet’s crystal and checksum are never compared with the rebuilt chunk; and chainId, the mirror weave, segCuts, origBytes and value are never recomputed. It proves the record is consistent with itself, not who made it.
Authority: typed commitments
Who grew a chain, and where it went, is recorded by signed commitments, each a pipe-joined string built from the record’s own fields.
| Kind | Commits | Used for |
|---|---|---|
| GROWN/1 | chainId|merkleRoot|weaveChiral|weaveMirror|origBytes | Certification by the grower |
| GROWTH/1 | parentChainId|chainId|parentMerkleRoot|merkleRoot|addedBytes | A child epoch’s link to its parent |
| CODEX/1 | codexHash|order|entryCount | A codex: order-2 byte-context counts, a prior for the coder |
| PAIR/1 | chainId|codexHash | Binding a codex to a chain |
| TRANSFER/1 | chainId|identityId|toUid|epochToken | A free transfer, stamped with a sequence number and usable once |
In BioChain-AI each is signed as an ATTEST/1 payload, ATTEST/1|keyId|keyVector|SHA256(commitment)|epochToken, with ECDSA P-256 over SHA-256 through WebCrypto, and checked against a keyRegistry of public keys. A key verifies only while it is active, and revocation is one way. Owl Academy signs the same commitments with its Minor Tome seals (SEAL/1|sealId|sealVector|contentHash|tickToken), and calls its transfer BIOMESH-XFER/1.
Where they live, and who can read them
A published biochain is one Firestore document, biochains/{chainId}, with the stream stored inline as hex. In both repositories’ firestore.rules, any signed-in user may read any biochain, and the stream decodes to plaintext: stored text is public to the whole tenant. On creation the rules check the grower and owner ids, not the record’s body, its chainId or its attestation, and nothing may be deleted. The Lexicon publishes no shards to Firestore this round (README, open choices).
The 60 KB lineage cap
A biochain can grow “further”, but only by regrowing. That puts a hard ceiling on a whole lineage, not on one step.
Both repositories set MAX_GROW_BYTES = 60000, with the comment “Firestore doc budget (stream stored inline)”. A published chain is frozen, so growth mints a new epoch. extendBiochain(parent, newText) verifies the parent, recreates its text, and calls growBiochain(parentText + newText). The child carries parentChainId, parentMerkleRoot, epoch + 1 and addedBytes, sealed by GROWTH/1. Nothing is appended: every epoch regrows everything, so its cost grows with the whole text, and the whole text must stay within 60,000 bytes.
Two of the survey’s reports said larger corpora could grow through many GROWTH/1 epochs; a third said a lineage can never pass 60 KB. The cross-check read biomesh.js again and sided with the third: growBiochain throws above the cap, and extendBiochain passes it the whole text. So corpus growth needs many independent shards.
The arithmetic
Spike S4 cut WordNet 3.0’s verb and adjective glosses into shards and found the full ones hold a mean of 59,930.9 bytes. At that fill:
| Corpus | Text | Shards |
|---|---|---|
| C0, WordNet 3.0 noun glosses (registered) | 82,115 lines | 148 (147 ≥ 59,000 B) |
| BabyLM strict-small, about 10M words | 55–60 MB | 918–1,002 |
| BabyLM strict, about 100M words | 550–600 MB | 9,178–10,012 |
The BabyLM rows are S4’s extrapolation estimate: their text differs from glosses. C0 is fixed by the preregistration: noun synsets in offset order, one line each (offset lemmas gloss), cut at line boundaries into shards of at most 60,000 UTF-8 bytes, their SHA-256s pinned in wordnet_v1.meta.json (smallest 20,962 B, largest 59,999 B). Neither BabyLM corpus could be fetched in Phase 0 (osf.io was blocked); since the network was widened at the start of Phase 1, BabyLM is reachable through Hugging Face, while osf.io’s file downloads stay blocked (Explainer 7).
Shards
Decision D8: shards of at most 60,000 UTF-8 bytes, in unmodified shdccp/2 with the JavaScript codec’s hashing. A shard is an ordinary biochain, so every existing reader accepts it as it is.
Unmodified matters. Owl Academy, BioChain-AI and Engram Mind’s Eye already verify shdccp/2; any change of bytes would cut the shards off from them. So everything the Lexicon adds lives beside the shard, in records (chapter 5), and everything that would change bytes is only proposed, under a future shdccp/3 (Explainer 2).
What spike S4 measured
S4 grew the 22 verb and 33 adjective gloss shards, and 8 synthetic ones, with a spike-only synchronous grower: biochain_v1.js’s verbatim codec, the orbital core’s SHA-256 over UTF-8, and Owl’s leaf, Merkle, weave and chainId rules restated line for line. It then ran Owl Academy’s own growBiochain, unchanged, on the same texts.
| Spike S4 | Result |
|---|---|
| Records equal to Owl’s, field for field (15 fields and their order), engrams on and off | 126/126 |
Owl’s verifyIntegrity accepts the engrams-off records, and the decoded stream equals the shard’s bytes | 63/63 |
Engram’s importBiochain accepts every verb shard and recreates its text byte for byte, engrams off / on | 22/22 · 22/22 |
| The orbital core’s SHA-256 against Node’s, over UTF-8 strings | 1000/1000 |
| Full gloss shards (53): mean bytes; chunks | 59,930.9 B; 52 with 84, 1 with 83 |
It also ran BioChain-AI’s Python kernel, which writes a crystal byte 0x80 as 0x81. Its normalised packets changed the stream of 43 of 63 shards, but never merkleRoot, the weaves or chainId (63/63), and Owl’s verifyIntegrity and EngramCore.verifyBiochain both accepted the normalised records (63/63). The reason is a gap in its own right: leaves and weaves are computed from chunk bytes, never from packets, so nothing that names a chain commits to its packets (Explainer 2).
The quirks a Python port must copy
- Leaf hashing.
String.fromCharCodeturns each byte into a code point and SHA-256 reads the UTF-8 of that string, so every byte from 0x80 up is hashed as two bytes. BioChain-AI’s Python hashes binary, and the two cannot verify each other’s chains. - The 0x80 byte. Written raw, as the JavaScript does: 1.4131% of gloss chunks have a crystal holding one (64 of 4,529; analytic 1.5534%), and 39 of the 55 gloss shards have at least one.
- The float weave. V8’s
Math.hypotalgorithm and an exacttoFixed(12)(chapter 7).
Language shards are Owl-valid biochains, field for field, in JS and Python: our synchronous JS grower and our Python port each give the record Owl Academy’s own growBiochain gives (BioChain-AI’s gives the same); Owl’s and BioChain-AI’s verifyIntegrity accept every record; importBiochain accepts every record and recreates the text; the derived layers and the LPREP/1 strings agree in Python and JS. Instances: C0’s 148 shards and 20 synthetic ones from Rng(1000 + i), each grown with engrams on and off. Controls that must fail: a Python grower with the kernel’s 0x80 normalisation; a grower that hashes leaves over binary bytes; one altered byte must change its leaf and root. S4’s 126/126 is a spike result on verb, adjective and synthetic shards, not this claim’s.
Phase 3. Supported for the JS grower and for the Python port: 0 of 336 records (C0’s 148 shards and the 20 synthetic ones, engrams off and on) differ from Owl’s own growBiochain in any of the fifteen fields, and BioChain-AI’s grower gives the same; Owl’s and BioChain-AI’s verifyIntegrity accept every record, importBiochain recreates every text byte for byte, and the id streams, addrTableRoot and the LPREP/1 strings are identical in Python and JS. The kernel normalisation differed on 20 of 20 synthetic shards (in the stream only), binary leaves on 168 of 168 (all rejected by Owl), and one altered byte changed leaf and root on 168 of 168. 189 of C0’s 12,378 chunks (1.527%) have a crystal holding 0x80 (Explainer 11).
Records
A shard holds text. Everything else the front end produces (the vocabulary, the address table, the token ids, the manifest, the benchmark results) travels as typed commitments, in the pattern GROWN/1 already uses.
shdccp/2 biochain of ≤ 60,000 B: the text, unchanged| Kind | Commits |
|---|---|
| LVOCAB/1 | The tokenizer: vocabulary SHA-256, the normaliser spec’s hash, the Unicode 15.1 tables’ hash, algorithm and commit, training-manifest root, seed, special tokens. Content-addressed like a codex. |
| LADDR/1 | The address table: vocabulary hash, WordNet archive hash, the address rule, and addrTableRoot, an RFC 6962 root over rows id|token|word|len|h3PointSha|h2PointSha; the two trees’ steps, t = 3/4 and 11/13. |
| LPREP/1 | One per shard: corpus id, shard index, chainId, merkleRoot, byte length, SHA-256 of the raw slice, SPDX licence id, source hash, LVOCAB hash, id-stream SHA-256, LADDR hash, exact-weave hash, optional engram-stream hash, tier. |
| LCORPUS/1 | The manifest: corpus id, N, an RFC 6962 root over SHA-256(LPREP/1) in shard order, licence set, source manifest, evaluation-set hashes, a contamination receipt, a Croissant metadata sidecar’s hash. |
| LGROW/1 | corpusId|N_old|root_old|N_new|root_new|SHA256(consistency proof): growth appends shards and proves it; nothing is regrown. |
| LBENCH/1 | A benchmark receipt: claim, subject hash, evaluation-set hash (gated sets by hash only), scorer and version, seeds, a hash of each item’s output, metrics, hardware and versions, class (reproducible or attested). |
| LMODEL/1 later | A model: a per-file hash manifest, training seed, LCORPUS root, mask-schedule seed and checkpoint hashes. |
Built in Phase 3. Amendment 7 fixed these kinds as exact pipe-joined strings before any code, and Phase 3 built LVOCAB/1, LADDR/1, LPREP/1, LCORPUS/1 and LGROW/1 in JavaScript and Python. LVOCAB/1, LADDR/1 and every LPREP/1 are identical in both (L8(d)), and so are the manifest’s roots and proofs (L9). LCORPUS/1’s evaluation-set, contamination and Croissant fields stay empty until Phase 6, and LBENCH/1, left undefined then (Explainer 11), was defined by Amendment 10 item 12 for L6’s tables (the instance, arm, geometry, d, seed and rate, the protocol’s and settings’ hashes, the table hashes and the exact MAPs’ hashes) and written in Phase 4: 367 receipts, one per evaluated table (Explainer 12).
Kinds, not Form values. Decision D7 keeps the namespace in record kinds, never in a Form nibble: every one of the 16 Form values is already claimed by some consumer (Explainer 2), and a biochain document has no type field (publishBiochain drops meta.kind). Each Lexicon record therefore carries {kind, spec: 'hyperbolic-lexicon/0.1', compat: 'biomesh-js'}.
Derived layers are recomputed, not stored. Token ids and addresses are never written per token. They follow the engramStream precedent: a layer that is a pure function of the rebuilt text and pinned tables, which a verifier recomputes and compares by hash. The record holds only the id stream’s SHA-256 (LEB128 varints, one per token) and addrTableRoot. That works only because the normaliser, the tokenizer and the address table are pinned and deterministic, which is what claims L1 and L3 test (Explainer 3). It also avoids the engram layer’s cost: 8 bytes (16 hex digits) for every word token, while S4’s gloss text averaged 6.7 bytes a token, so the layer is larger than the text it describes and more than halves a shard’s value (chapter 10).
Signed in ATTEST/1, as spike S7 measured it
Every record can be signed in BioChain-AI’s exact format. S7 ran BioChain-AI’s own crypto-core.js, signing-key.js and biomesh.js unchanged in Node (with Firestore replaced by an in-memory map) and recomputed its values in Python. It built a sample LPREP/1-style commitment of 15 fields, 582 characters, with placeholders in eight of its fields (the last six hashes are shortened here):
LPREP/1|S7-SPIKE|0|BC-bb5306c48ec981158afab12a|93b6aaf0dc5d47a9fe6ef8213d78eb53d79c872f3ea494b93fb6ea3b68439c6a|59993|92b10065af6d41824898acbd1005a44bb83b087257ca64e3bf2b6670ed51f360|CC0-1.0|cb1469a8…|c45f3f80…|6f4e21c7…|081e7492…|cb27ed23…|5b5f1c1c…|S
| Spike S7 found | So the importer must |
|---|---|
The payload is ATTEST/1|keyId|keyVector|contentHash|epochToken: 137 characters with S7’s 19-character epoch token. keyId = “K-” + the first 24 hex of SHA-256 over the public key’s x|y; keyVector is 16 hex derived from the identity, independent of the key. | Rebuild the payload from the commitment, never trust a stored one |
WebCrypto writes IEEE P1363 signatures (r‖s, 64 bytes, 128 hex). Python’s cryptography verified Node’s 200/200 after conversion to DER and 0/200 raw; WebCrypto accepted Python’s DER 0/200 raw and 200/200 converted. | Convert P1363 ↔ DER at the language boundary |
| DER back to P1363 must left-pad r and s to 32 bytes: one of them falls below 2248 in about 1 signature in 128, and the unpadded form is rejected. | Pad, and test the short cases |
| The high-S twin (r, n − s) of a signature also verifies, in WebCrypto, node:crypto and Python. | Key nothing by signature bytes |
Every single-character change of the payload (137 positions), the commitment (582) and the signature (128 hex) was rejected, in Node and in Python alike. identityId, ownerUid, signedAt and format lie outside the payload and do not affect the verdict. | Take the signer’s identity from the key registry, never from unsigned fields |
S7 also wrote a deterministic golden ATTEST/1 under RFC 6979’s public test key (keyId K-ba70de20e32549d49e6bd616), which verifies in WebCrypto, node:crypto and Python and through BioChain-AI’s verifyAttestationAgainstContent: a known-answer vector for tests, never a signing key.
The manifest
Many shards need one commitment to the whole list, in order, that can grow without being rebuilt and that answers two questions cheaply: is this shard in the corpus, and is today’s corpus an extension of last month’s?
LCORPUS/1 is an RFC 6962 Merkle tree, the one Certificate Transparency logs use, over SHA-256(LPREP/1) in shard order:
- Leaf hash = SHA-256(0x00 ‖ entry). Node hash = SHA-256(0x01 ‖ left ‖ right). The prefixes keep a leaf from ever passing as a node.
- Split rule. A list of n > 1 entries splits at k, the largest power of two below n: the left subtree takes the first k, the right the rest. Nothing is duplicated.
- Inclusion proof. To show entry i is in a tree of N, give the sibling hashes along its path, about log₂ N of them; the verifier rebuilds the root.
- Consistency proof. To show a tree of N extends a smaller tree of m (the first m entries unchanged), give a few subtree hashes from which both roots can be rebuilt. That is what makes
LGROW/1append-only.
| Spike S4, RFC 6962 | N = 1,000 | N = 10,000 |
|---|---|---|
| Inclusion proofs that verify; rejected for the next leaf’s hash; rejected at the next position | 1000/1000 each | 10000/10000 each |
| Inclusion proof length | 8–10 hashes, ≤ 320 B | 8–14 hashes, ≤ 448 B |
| Consistency proofs that verify, from every smaller size m | 999/999 | 9999/9999 |
| Consistency proof length | 1–11 hashes, ≤ 352 B | 1–15 hashes, ≤ 480 B |
| ~ Verifying one inclusion proof | 8.4 µs | 18.7 µs |
S4 also reproduced the Certificate Transparency reference roots for every prefix of its 8-entry vector (9/9) and RFC 6962’s own worked audit paths and consistency proofs (7/7), and got the same roots from the orbital core’s SHA-256 as from Node’s.
Why not the shdccp/2 rule
The biochain rule hashes hex strings in pairs and, on an odd level, copies the last node before pairing. So [a, b, c] pairs as (a, b), (c, c), and [a, b, c, c] pairs as (a, b), (c, c) too: two different lists, one root. This is the CVE-2012-2459 class.
shdccp/2
[a, b, c] → H(H(a b) H(c c))
[a, b, c, c] → H(H(a b) H(c c))
the same root
RFC 6962, with h(x) = H(0 ‖ x)
[a, b, c] → H(1 ‖ H(1 ‖ h(a) ‖ h(b)) ‖ h(c))
[a, b, c, c] → H(1 ‖ H(1 ‖ h(a) ‖ h(b)) ‖ H(1 ‖ h(c) ‖ h(c)))
different roots
Spike S4 checked every K from 1 to 200. Under the shdccp/2 rule, a list and the same list with its last element repeated had equal roots at K = 3, 5, 7, …, 199: every odd K ≥ 3, 99 values. K = 1 is safe, because a one-leaf root is the leaf itself and is never hashed. Repeating the last two elements collided at K = 6, 10, 14, …, 198 (49 values). RFC 6962 roots of the same lists were equal at 0 of 200.
Inside one shard the damage is limited, because the stream fixes how many leaves there are. A manifest is different: a root alone must fix the list it commits to. So shards keep their shdccp/2 root for compatibility, and every list the Lexicon commits to (shards, address-table rows) uses RFC 6962 (Lemma 15 note; the lab below shows both rules side by side).
LCORPUS/1 is an RFC 6962 tree over SHA-256(LPREP/1) in shard order. Every inclusion proof verifies for its own leaf and position and fails for any other; every consistency proof from a smaller prefix size m verifies, so LGROW/1 is append-only; every single alteration (replace, delete, insert, adjacent swap, duplicate-last, truncation with the old root kept) is detected; JS and Python give identical roots and proofs. Instances: the manifest over C0 (K = 148) and synthetic lists with K ∈ {1, 2, 3, 7, 84, 200, 1000, 1001}. The control that must fail: the shdccp/2 rule must collide at every odd K ≥ 3 among them.
Phase 3. Supported. On C0’s manifest (K = 148, root f567700d…) and the eight synthetic lists, every root and proof is equal in JS and Python, every alteration is detected (739 on C0’s manifest; 2,000 drawn at each of K = 1,000 and 1,001), and every proof is rejected for a wrong leaf or position; inclusion proofs need at most 8 hashes at K = 148 and 10 at K = 1,000 and 1,001. The shdccp/2 rule collided at K = 3, 7 and 1,001 (Explainer 11).
The weave
A Merkle root says which chunks a chain holds. The weave is a second summary, built to change when chunks swap places. Today it is computed in floats, and that makes it fragile across languages.
The float weave
Each chunk’s 32-bit crystal is read as four signed bytes s, made into (0.5 + s₀/254, s₁/127, s₂/127, s₃/127), and normalised with Math.hypot. The weave multiplies these unit quaternions in order, renormalising after each step, and prints each component with toFixed(12) float. Because chainId hashes that string, a rounding difference that reaches the twelfth decimal changes the chain’s name.
Spike S4 ported the weave to Python two ways and compared both with Owl’s JavaScript on 1,000 synthetic shards. A faithful port (V8’s Math.hypot algorithm, exact toFixed(12)) differed on 0 of 1,000. A naive port (Python’s math.hypot and '%.12f') wrote a different weave string on 2 of 1,000. The cause was hypot: Python’s differed from V8’s on 39,408 of 127,152 calls (30.9928%), while the formatting differed on 0 of 8,000 components.
The exact norm-5 weave
The Lexicon proposes an exact replacement, built from code Engram Mind’s Eye already ships. Take the first 8 bytes of each leaf hash as a 64-bit value. engram_core.chainQuaternion writes each value as a 28-letter word of the norm-5 tree (each letter one of the integer quaternions 1±2i, 1±2j, 1±2k) and multiplies the whole chain out into one integer quaternion exact. Its norm is exactly 528K for K leaves, and right division (engram_core.unfold) reads the values back, in order (Explainer 1).
| Spike S4, verb shard 00 | Result |
|---|---|
| Leaves K; letters | 84; 2,352 |
| Norm (sum of the four squared components) = 52352 exactly | true |
| Bits per component | 2730, 2730, 2729, 2730 |
unfold(q, 84) returns the 84 values in order | true |
| The ordered product of the 7 segment sub-weaves (cuts 0, 12, 24, 36, 48, 60, 72, 84) equals it | true |
| ~ chainQuaternion; unfold; segment products | 16.3; 27.2; 3.3 ms |
Float weave (shdccp/2) | Exact weave (proposed) | |
|---|---|---|
| Input | The crystals, 32 bits a chunk | Leaf-hash prefixes, 64 bits a leaf |
| Arithmetic | float one rounding per step | exact BigInt, no rounding |
| Across languages | Only with a port of V8’s hypot | Integers, with nothing to round (L9(b) tests it) |
| Invertible | No | Yes: unfolds to the prefixes, in order |
| Size | Four 12-decimal strings | About 32.5 bits a leaf per component; 2,730 bits at K = 84 |
The exact weave carries order; collision resistance still comes from the SHA-256 leaves underneath it. Shards keep the float weave, because shdccp/2 readers check it; the exact weave travels beside them, as weave3Sha in LPREP/1, and is proposed in place of the float weave for shdccp/3 (Explainer 2).
Each shard’s exact norm-5 weave, engram_core.chainQuaternion over the first 8 bytes of each leaf hash in order, has norm exactly 528K, unfolds in order to those prefixes, equals the ordered product of its 7 segment sub-weaves exactly, and has identical components in JS and Python, on every C0 shard. The control that must fail: an XOR summary of the same prefixes must miss at least one adjacent swap that the weave detects.
Phase 3. Supported. All 148 C0 weaves are exact: norm exactly 528K (52352 for the 147 shards of 84 chunks, 5840 for the last, of 30), unfolded in order, equal to the product of their 7 segment sub-weaves, with components of 971–2,731 bits equal in JS and Python (~45.9 ms a shard in JS, 51.3 ms in Python). The XOR summary missed every adjacent swap it was given, and the weave detected each (Explainer 11).
Two tiers
Every signed-in user can read every biochain, and a stream decodes to plaintext. Much language data may not be shared that way. So the design has two tiers: the text travels only when its licence allows; otherwise only commitments do.
Tier S · the stream
Plaintext travels losslessly in the shdccp/2 stream, with its leaves, for text whose licence allows redistribution: WordNet with its notice kept (licence), UD English EWT (CC BY-SA 4.0), BLiMP (CC BY), public-domain Gutenberg texts, and the team’s own text. Anyone holding a shard can rebuild and check it.
Tier P · public commitments
Restricted corpora (BNC, OpenSubtitles, CHILDES/TalkBank, gated EWoK) publish only hashes, roots, counts, SPDX ids, metrics and receipts. Their streams stay on the holder’s disk, or in object storage keyed by root. SNOMED CT is excluded; EWoK appears by hash only.
Salted commitments
A bare hash of a known text can be confirmed by anyone who hashes the same text. So a Tier-P shard’s public LPREP/1 carries saltedRoot = SHA-256('LPREP-SALT/1'|salt|merkleRoot|chainId), with a 32-byte secret salt per shard, and omits chainId. Without the salt, guessing the text confirms nothing. Phase 3 tested Tier P as a format only: 26 Tier-P twins of C0 shards, salted with the registered test salts (which anyone can compute, so they hide nothing), were accepted, and all 50 checked with the wrong salt (L10’s T6) were rejected; the challenge audits below were not built (Explainer 11).
Challenge audits
Nobody else can recompute a Tier-P shard, so it is checked by challenge. The auditor derives k shard indices from SHA-256(LCORPUS root | ‘audit’ | n); the holder reveals those shards, their salts and their inclusion proofs, privately. If a fraction f of the shards were bad, the audit misses all of them with probability (1 − f)k:
| Bad fraction f | k = 10 | k = 30 | k = 100 | k = 300 |
|---|---|---|---|---|
| 1% | 0.904 | 0.740 | 0.366 | 0.049 |
| 5% | 0.599 | 0.215 | 0.0059 | 2.1×10−7 |
| 10% | 0.349 | 0.042 | 2.7×10−5 | 1.9×10−14 |
The table is arithmetic on the formula. Its lesson: an audit catches widespread problems quickly and rare ones slowly. With one bad shard in 1,000 (f = 0.001), even k = 300 misses it with probability about 0.74.
It is attestation plus audit, not public recomputation, and the design says so plainly. A signature proves who committed a record. It does not prove that the licence holds or that the data is correct.
Contamination receipts. LCORPUS/1 also commits to a receipt: the 13-gram SHA-256 sets of the evaluation items, intersected with the training id streams. Only the counts are published. The full licence table is in Explainer 7.
The hardened import
The verifier in use today proves that a record agrees with itself. Spike S4 measured how much that leaves open. The Lexicon’s importer closes it, and reports two verdicts instead of one.
What verifyIntegrity accepts
S4 altered verb shard 00 in eight ways and ran Owl Academy’s verifyIntegrity, unchanged, beside EngramCore.verifyBiochain. The two agreed every time.
| Alteration (spike S4) | Owl’s verdict | The hardened check that catches it |
|---|---|---|
T1: leafHashes and merkleRoot deleted | accepts | Leaves and root are mandatory |
| T2: packet 0’s crystal bit 0 flipped, parity re-fixed | accepts | Re-crystallize each chunk |
| Packet 0’s checksum bit flipped, parity re-fixed | accepts | Re-crystallize each chunk |
T8: chainId’s last hex digit changed | accepts | Recompute chainId |
| One padding bit of a residual flipped | accepts | Re-encode each chunk |
| One bit of a raw-byte rank code flipped (decodes the same) | accepts | Re-encode each chunk |
| T7: frames 0 and 1 swapped (a control) | rejects: leaf 0 mismatch | Already caught |
| Packet 0’s parity bit flipped (a control) | rejects: parity failed | Already caught |
Residual aliasing. The last two rows need a word. When the decoder reads a rank larger than the number of bytes seen in a context, it reads the next 8 bits as a raw byte, whatever the rank was. So an escape code has aliases: several bit patterns decode to the same chunk, and leaves, root, weave and chainId all commit to decoded bytes only. S4 flipped every single bit of frame 0’s residual in verb shards 00–03: 689 of 19,400 flips (3.5515%) decoded to the same 720 bytes (681 inside a raw-byte rank code, 8 in the last byte’s padding, 0 elsewhere). A re-encode check (encode the decoded chunk and require the stored residual, byte for byte) rejected 689 of 689. S4 also altered one byte of each verb shard’s stream; importBiochain rejected 21 of 22 copies, and the one it accepted decodes to the same bytes.
The checks
- Packet parity, as today.
- Decode, then re-encode each chunk and require the stored residual, byte for byte: only the canonical encoding passes.
- Re-crystallize each chunk and require the stored packet: crystal and checksum are checked at last.
- Leaves and root mandatory, recomputed with the JavaScript rule.
- Both weaves and
chainIdrecomputed. - The derived layers (the V0 id stream,
addrTableRoot) and the exact weave recomputed; theLPREP/1commitment rebuilt from them. - The ATTEST/1 signature verified against a pinned snapshot of the key registry, refusing revoked keys and key-vector mismatches; P1363 converted to DER for Python, and back with r and s left-padded.
- Manifest proofs: the shard’s inclusion in
LCORPUS/1, and the consistency of eachLGROW/1.
Self-consistent
Steps 1–6 and 8: the record, its derived layers and its place in the manifest agree with one another. Anyone can check this, but anyone can also produce it, by growing an altered text.
Authentic
Step 7: a registered, unrevoked key committed to exactly this record. Today nothing in Engram’s import path calls verifyGrown, and the Firestore rules do not check attestations.
Engram’s importBiochain is not changed; the Lexicon’s importer wraps it.
The lexicon importer, in JS (WebCrypto) and Python (cryptography), accepts every clean attested record and rejects every record in the registered tamper set, naming the failing check; its verdicts are reported separately as self-consistent and authentic, and are identical in both languages; Owl’s verifyIntegrity accepts the classes that only the lexicon importer’s checks catch. Instances: 50 C0 shards, each LPREP/1 attested with a P-256 key generated per run, and 9 tamper classes of 50 records each: T1 leaves and root removed, T2 a crystal changed with parity re-fixed, T3 an LPREP/1 field changed, T4 a revoked key, T5 a valid attestation of a different shard, T6 a Tier-P commitment with a wrong salt, T7 two frames swapped, T8 chainId altered, T9 a residual bit flipped that still decodes the same; 450 tampered records and 50 clean. Controls that must fail: Owl’s verifyIntegrity must accept at least one T1, T2, T8 and T9 record; Python without the P1363-to-DER conversion must reject every valid signature.
Phase 3. Supported. Of the 500 attested packages, JS (WebCrypto) and Python (cryptography) made 0 false accepts and 0 false rejects, with identical verdicts on all 500. The verifier runs the eight steps above as ten named checks, and Amendment 8 made its two verdicts follow the definitions above. Every class failed where it should: T1 at leaves, T2 at the crystal check, T3 and T6 at the commitment, T4 and T5 at the attestation (self-consistent, not authentic), T7 at leaves (48) or the canonical check (2), T8 at the weaves and T9 at the canonical check. Owl’s verifyIntegrity accepted 50 of 50 in each of T1, T2, T8 and T9, and Python without the conversion accepted 0 of 126 valid signatures (Explainer 11).
Costs
A design for corpora has to be cheap per shard. Spikes S4 and S7 measured the pieces on this container; claim L11 fixes the threshold the whole importer must meet.
Size, per full 60 KB gloss shard
| Spike S4, 53 full gloss shards | Value |
|---|---|
| Chunks | 84 |
| Stream: hex characters per text byte | 1.7120 |
| Residual: bits per text byte (the rank coder alone) | 6.7134 |
| Value, engrams off (median, range) | 1.1016 (1.0334–1.3070) |
| Value, engrams on (median, range) | 0.4860 (0.4426–0.5194) |
| Record JSON: characters per text byte, engrams off / on | 1.8131 / 4.2144 |
| Chunks whose crystal holds a 0x80 byte | 1.4131% (analytic 1.5534%) |
A value near 1 means the stream is about as large as the text. The rank coder is an integrity container, not compression; for context only, S4 measured gzip -9 at 3.1230 bits per byte on the same shards, and nothing is claimed from it. On C0’s 148 shards in Phase 3 the median value was 1.102 with engrams off and 0.469 with them on, from the per-shard values; the reported medians, 1.095 and 0.466, are over all 168 shards, the 20 synthetic ones included (Explainer 11).
Time, per full shard
| ~ Median, this container | ms |
|---|---|
| Encode / decode / crystallize (S4) | 24.9 / 22.3 / 4.3 |
| Whole synchronous grow, engrams off (S4) | 40.9 |
Owl’s own growBiochain, engrams off (S4) | 51.0 |
| Owl’s verbatim engram layer on verb 00, 9,482 tokens: 731.2 µs a token (S4) | 6,933.7 |
The same layer in engram_core, 52.7× faster (S4) | 131.5 |
Codec part of the hardened verification, per verb shard: parse and parity 0.2, decode 21.3, crystal re-check 4.7, leaves and root 3.8, weaves and chainId 9.0, exact weave 2.0 (S4, range 33.0–43.4) | 41.0 |
| The re-encode check, on top, per verb shard (S4) | +23.5 |
| The attestation step as BioChain-AI writes it; WebCrypto verify alone, key imported once (S7) | 0.2341; 0.1186 |
| ECDSA verify in Python, with the P1363-to-DER conversion (S7) | 0.0590 |
Two things stand out. Signatures are nearly free: S7 puts the attestation step for 1,000 shards at 0.23 s in Node. And the engram layer is the expensive part: Owl’s verbatim code recomputes each letter’s grid once for every one of its 64 bits. The Lexicon keeps engram/1 off by default for corpora and stores only hashes of derived layers. Phase 3 bore both out: on C0 the attestation check took a median 0.71 ms inside a full verification (of engrams-off packages; the L11 box below), and over all 168 shards Owl’s own grower took a median 9,319 ms a shard with engrams on against 70.5 ms with them off (~, run beside other work; Explainer 11).
At BabyLM scale
| S4’s extrapolation estimate | strict-small, 55–60 MB | strict, 550–600 MB |
|---|---|---|
| Shards | 918–1,002 | 9,178–10,012 |
| Stream hex | 94.2–102.7 MB | 941.6–1027.2 MB |
| Record JSON, engrams off | 99.7–108.8 MB | 997.2–1087.9 MB |
| Manifest inclusion proofs | ≤ 10 hashes (320 B) | ≤ 14 hashes (448 B) |
| ~ Whole synchronous grow, single thread | 37.5–41.0 s | 375.3–409.4 s |
| ~ Codec verification, single thread | 37.6–41.1 s | 376.2–410.4 s |
~ Engram layer: Owl’s verbatim code / engram_core | 1.8–1.9 h / 119.8–130.7 s | 17.7–19.3 h / 1197.5–1306.3 s |
A holder verifies a full C0 shard with every hardened check of L10 (and the V0 id stream and addrTableRoot recomputed) in a median of at most 250 ms in Node, single-threaded, on this container; a 1,000-shard corpus therefore verifies in under about 5 minutes. Instances: the 147 C0 shards of at least 59,000 bytes, 3 timed repetitions each, the first discarded. Supported if the upper end of the median’s bootstrap interval is ≤ 250 ms; refuted if the lower end exceeds 500 ms; inconclusive otherwise. The control: a run with the attestation step skipped must be faster. Predicted: a median of about 50–150 ms.
Phase 3. Supported, on packages grown with engrams off. The registration did not say which; Amendment 8 fixed off, the Lexicon’s default for corpora, after dry runs had put the median on engrams-on packages at 243.5–265.5 ms, near the bar, and it says so plainly. The 147 full C0 shards verified in a median 155.4 ms, 95% interval [154.2, 156.0], just above the predicted range; a 1,000-shard corpus takes about 156.7 s. The same shards’ engrams-on packages took 243.6 ms (reported). Amendment 8 also replaced the control, since a comparison of medians could not resolve a 0.7–0.8 ms step against 10–28 ms of noise, with the verifier’s own record of its checks, which held: all 441 timed full verifications ran the attestation inside the timed window, and all 441 skipped ones recorded it skipped (Explainer 11).
Adding up what the spikes timed, the codec part, the re-encode and the attestation come to about 41.0 + 23.5 + 0.2 ≈ 65 ms a shard estimate. The id stream, addrTableRoot and the LPREP/1 rebuild were not timed then: L11 is timed in Node, and Node got the tokenizer and the addresses in Phase 2 (Explainer 10) and the records in Phase 3. That is why L11 sets a threshold with headroom and predicts a range, not a number.
Measured in Phase 3 (medians over the 147 C0 shards, Node, engrams-off packages, “~”): the canonical check, which decodes and re-encodes every chunk, 78.9 ms, against S4’s 21.3 + 23.5 on verb shards; the derived layers (the V0 id stream and the exact weave) 46.4 ms; weaves and chainId 13.3 ms; the crystal re-check 7.2 ms; leaves and root 4.6 ms; the commitment 1.8 ms; parity 1.1 ms; the attestation 0.71 ms, against S7’s 0.23 ms; the manifest proofs 0.3 ms. The address table is rebuilt once per session, in 1,277.8 ms; rebuilt inside every verification instead, a shard takes 1,350.2 ms. Python takes 195.9 ms a shard (Explainer 11).
Grow a shard, live
This lab grows a biochain from your text in the browser, with Engram Mind’s Eye’s biochain_v1.js (Owl Academy’s synchronous code, verbatim) and the orbital core’s SHA-256, then verifies it with EngramCore.verifyBiochain. Then you can break it, and watch which checks notice.
The frames
| # | Bytes | Packet (form 9) | Crystal | Residual | Leaf hash |
|---|
Break it
One list, two Merkle rules
| K | shdccp/2: L and L + [last] | RFC 6962: L and L + [last] |
|---|
The grower follows the steps of spike S4’s synchronous grower: biochain_v1.js for encoding, crystals, framing and the weave quaternions, OrbitalCore.sha256 over UTF-8 for leaves, root and chainId. S4 found that grower equal to Owl’s growBiochain on 126/126 records; this page’s copy was checked against the three Owl-grown records in Engram’s bridges_v1.json, not separately registered. The left box is EngramCore.verifyBiochain, which repeats Owl’s verifyIntegrity step for step. The right box sketches the codec checks L10 adds; it is not the Phase 3 importer, and the derived layers, the LPREP/1 commitment, the signature and the manifest proofs are not in it. The Phase 3 grower and verifier themselves run in Explainer 11’s lab. “Flip one byte” flips the top bit of one byte of frame 0’s residual (by default the middle byte, as S4 did). The Merkle panel uses the list L = SHA-256(“leaf 0”), SHA-256(“leaf 1”), …: the shdccp/2 rule pairs their hex strings, RFC 6962 hashes their 32 bytes with the 0x00 and 0x01 prefixes. Generated texts are repetitive word lists, so their sizes and values are not those of real text; S4’s values are for WordNet glosses.
- Flip one byte: the decoder either runs out of bits (“residual truncated”) or rebuilds different bytes (“leaf 0 mismatch”). Slide the byte along: now and then a flip lands inside an escape’s rank code, which still reads as an escape, and the chunk decodes the same. Then Owl’s steps accept, and only the re-encode check objects.
- Delete leaves and root, change a crystal, alter chainId: Owl’s steps still accept. These are S4’s T1, T2 and T8.
- Three chunks, then swap frames 0 and 1: rejected at leaf 0, because the stored leaves fix the order. The shard’s own three leaves also show the duplicate-last collision in the Merkle panel.
- 60,001 B: the error
growBiochainthrows. One byte over is the lineage cap. - Non-ASCII: each byte from 0x80 up is hashed as two bytes in the leaf, the quirk a Python port must copy.
Into Engram Mind’s Eye
Every Lexicon shard is a shdccp/2 biochain, so the bridge into Engram Mind’s Eye already exists. What it lacks is a key whose bits mean something; that is the longer bridge.
EngramCore.importBiochain runs verifyBiochain (Owl’s verifyIntegrity, step for step), then checks that Engram’s own engram stream of the rebuilt text equals the record’s, and brings the text in as one turn of words, tokenized as biomesh.js tokenizes them (Engram’s bridges). S4 measured it accepting all 22 verb shards, engrams off and on, with the text recreated byte for byte; with engrams off its median was 170.6 ms a shard. It computes Engram’s own engram stream of the text on every import, and that layer alone took a median 130.5 ms a shard in S4. It checks no signature.
Now
Shards import unchanged through importBiochain; claim L8(c) tested it in Phase 3 on all of C0 and 20 synthetic shards, and it accepted every record, engrams off and on, recreating each text byte for byte (Explainer 11). The Lexicon’s importer wraps it with the hardened checks and the attestation.
Phase 5
The 2D and 3D Lexicon pages get a shard panel, and read Engram session exports with engram_core’s own importers, showing each token’s lexicon address beside its engram/1 key (Explainer 3).
Phase 8
engram/2: a subword id plus the lexicon address, registered by amendment in Engram’s own preregistration once the address claims are decided. Placement by content would then make Engram’s fence a novelty screen, which its Scope reserves for engram/2.
engram/1 itself stays what Explainer 1 found it to be: letter-count parity, a compatibility checksum for Owl Academy, off by default for corpora and never an input. The Owl notes and the BioChain-AI proposal carry this design to those repositories as documents; nothing in them is changed.
Out of reach until others act
- Existing BioChain-AI Python records cannot be cross-verified: that Python disagrees with the JavaScript on the 0x80 byte and on leaf hashing.
- GROWTH/1 as corpus growth: the cap holds for a whole lineage.
- Meaning in the crystal word: the 64-bit crystal is a digest of the chunk, not a representation of it.
Nothing. L8–L11 were registered and undecided when it was written; Phase 3 built the grower in JS and Python, the records, the manifests, the exact weave and the hardened importer, and decided them on the instances fixed in PREREGISTRATION.md, all supported (Explainer 11; ledger, Explainer 8).