Ternary weights: a smaller model on paper, but we have not made one work

A ternary model stores each weight as -1, 0 or +1 instead of a 16-bit number. In theory that shrinks the weights by about 16 times and could suit plain CPUs. That is arithmetic about storage, not something we measured, and we did not verify any speed gain. The open question is whether it can still be trained to be good. We tried, with free compute and small models.

Our attempts failed their own targets. That is a result about our configuration and budget, not proof that ternary models do not work.

Larger run, 2000 steps

Ternary perplexity 158.21 against 13.85 for the comparison model. It also failed its entry check, and the run continued anyway.

Distillation run (Exp8)

Teacher: frozen SmolLM2-360M. Goal: perplexity under 100 by step 1000. Result: 352.71. Not met.

Lower perplexity is better. The chart shows the distillation run against the earlier run without it.

Source: exp8_summary.json (preregistration SHA-256 0f33380de08a6dc5953eaa5221683a9c7b85532909d13125ae53ed1b32281c6b). The earlier run has no step 750 value. It is compared at the same step count, not at the same compute cost, so this is not a fair cost comparison.

• Small models and short training. Nothing here speaks to large ternary models.
• Only results backed by recorded evaluation output are shown.
• The weights and reproducibility files are stored privately. Nothing is released.