Reproducing Canon, mHC, and Engram

Large training runs are expensive enough that I do not want architectural taste choosing them.

Before I spend the compute, I want a smaller experiment to force an idea to make a specific prediction. Not “the loss might go down,” which is the prediction every new module makes when it wants funding, but what capability should move, on what task, under what measurement, and what failure would make us stop?

Canon, mHC, and Engram arrived as three very different bets. Canon inserts local convolutions at specific positions in the transformer block, betting that token-level substructure can be made more useful to later reasoning. mHC constrains residual flow with manifold-aware hyper-connections, betting on a better-conditioned route through depth. Engram adds hashed bigram memory with a learned gate, betting that some associations are better retrieved than repeatedly recomputed.

This experiment was our screening round. It was never meant to prove that a small synthetic harness predicts production pretraining. Its job was to decide whether any of these ideas produced a clear enough signal, with an understandable enough failure mode, to deserve a larger run.

The short answer is: Canon, yes. mHC, maybe, for a narrower reason than we started with. Engram, yes, but not before dealing with the fact that a shared hash address is not the same thing as shared meaning.

Three bets need three ways to win

Our first pass put all three primitives through the same generic task mix: definition-to-property QA (depo), symbolic grammar (lano), arithmetic (mano), and the usual loss, token-accuracy, and exact-match summaries. The results barely separated. The architectures were still unresolved; the screen was inconclusive.

A review against the PhysicsLM4 reference implementation showed why. Our depo task used single-token “words,” making retrieval much easier than the multi-token version. We were grading Canon mostly with loss and token accuracy even though its structured-grammar claim concerns the distribution of legal next tokens. We had implemented one Canon hook rather than the full A/B/C/D placement set, and Canon-D assumed SwiGLU semantics our MLP did not yet have. Engram, meanwhile, was being averaged into reasoning tasks where memory had no particular reason to help.

So we sharpened the screen rather than declaring a winner from noise. depo_v2 got multi-token words, a mini-vocabulary, an end-of-word marker, multiple QA pairs, and answer-only masking. lano_cfg became a context-free grammar whose ground-truth next-token distribution can be computed by dynamic programming. The MLP moved to SwiGLU and Canon used all four placements. Engram got its own memory suite.

This refinement matters, but it is not the point of the story. A screening harness has one job: give each architectural bet a fair chance to reveal the mechanism for which we might later pay real money.

Canon made the clearest case

Canon's signal appeared when we measured the whole grammar distribution. Token accuracy asks whether the model's favorite next token is legal. DP-KL asks whether probability is spread across all legal next tokens in the right proportions. A model can satisfy the first while learning a poor approximation to the grammar underneath it.

We ran six variants across three seeds on the corrected structured suite:

Canon reduced mean DP-KL by about 80%, from 0.196 to 0.040, and also produced the largest loss and token-accuracy improvements. That is not proof it will survive the move to a larger model, a richer corpus, or a production kernel. It is exactly what a screening result should look like: the mechanism moved the metric it claimed to move, across three seeds, by enough that the next run has something concrete to test.

Canon earned promotion. The larger experiment should preserve the ABCD placement semantics and SwiGLU contract rather than scaling a convenient approximation and calling any difference a test of Canon.

mHC made a quieter argument

mHC moved DP-KL from 0.196 to 0.163, about 17%, and improved loss and token accuracy modestly. When combined with Canon, however, it did not add to Canon's result at this scale: DP-KL was 0.045 for mHC + Canon versus 0.040 for Canon alone.

There are at least two readings. Canon may already do most of the representational work this small harness can expose. Or mHC's value may live in optimization stability and depth regimes that a six-layer, 2000-step screen cannot reach. The current matrix cannot choose between them.

That makes mHC a weaker promotion than Canon, not a rejection. I would carry it into a larger run only with a stability-oriented measurement plan (depth, conditioning, gradient or update behavior, and failure rate), rather than a generic promise that stacking it beside Canon must improve final loss. The small experiment earned a question, not yet a victory lap.

Engram was promising exactly where it was dangerous

Engram looked neutral on the structured suite (0.197 DP-KL versus 0.196 for baseline), which is unsurprising once we stop demanding that a memory primitive improve every form of reasoning. Its real test was whether it could exploit a repeated transition table without forcing stale memory into examples where retrieval was useless or ambiguous.

We built three slices. ngram uses a stable bigram table. ngram_polysemy has modes A and B share hash addresses while requiring different answers. ngram_scrambled changes transitions per sample, so stored memory should learn to stay out of the way.

The seed-0 answer-region token accuracies were:

Engram produced the strongest kind of early architectural result: a large win and a specific failure. It helped substantially on stable lookup, stayed neutral when the table was useless, and badly damaged one side of the polysemous lookup.

That −59% mode-B result is seed 0, not a multiseed law. It was still important enough to diagnose before scaling. If memory retrieved the right answer early and later layers overwrote it, more depth or better residual control might help. If the lookup was wrong from the start, scaling the same mechanism would merely buy a larger machine for making the same mistake.

We built LogitScope for that distinction: project every layer to logits and compute cross-entropy against the actual correct answer, rather than asking only whether an intermediate state resembles a plausible token.

Mode B was worse than baseline essentially from the start. PLE+Ngrammer showed the same qualitative pattern, which makes unresolved hash collision a more interesting suspect than an Engram-specific late-layer bug, but the evidence remains one diagnostic seed.

Engram therefore earns a larger experiment only after an architectural change that can disambiguate retrieval: a larger table, context-aware hashing, or a gate that can reject an ambiguous memory before injecting it. The screen did not tell us to abandon memory. It told us what must be different before more compute becomes informative.

Attention ratio did not earn the same priority

We also swept global-attention ratios in a 30-layer, 4000-step screen at sequence length 4096 with a local window of 64, using the mHC + Canon + Engram stack and 100%, 50%, 10%, and 0% global-layer schedules across three seeds.

Some global layers helped optimization a little, but the differences were small and mostly within the observed variation. That may mean the task mix did not isolate a truly global regime, or that Canon and mHC already carried much of the work here. Either way, attention ratio did not produce the kind of first-order signal that should drive the next expensive run.

What deserves the larger run

The point of this exercise was not to produce a tiny-model leaderboard. It was to spend a cheap harness buying better large-run decisions.

Canon made the strongest case. It moved the right metric across three seeds and should advance as a faithful ABCD implementation. mHC produced a modest standalone signal and deserves a targeted stability test at greater depth, not an assumption of additivity. Engram proved that learned memory can help dramatically on its own turf and exposed a concrete collision that should be fixed before scale. The global-attention ratio, on this task mix, can wait.

That ordering is provisional by design. The harness covers small models, synthetic tasks, and short runs. Larger pretraining may reverse an interaction, expose a throughput cost, or reveal a benefit the physics tasks cannot represent. But now those larger runs have reasons, measurements, and stop conditions. We are no longer choosing architecture by how persuasive its diagram looks.

Receipt and provenance map: 0010. The structured table combines three recorded seeds; the Engram slice and layerwise collision diagnostic are seed 0; the attention sweep uses three seeds. The cited physics/... artifacts are internal provenance paths rather than payloads bundled with this site.

The screen did what I wanted. It did not tell me which architecture is true. It told me which ideas had earned the right to become expensive.