← Bezawit Abebaw

Building a Transformer Beyond Backpropagation

A predictive coding approach where each layer learns from its own local prediction error.

The question

Backpropagation requires every layer to wait for an error signal computed at the output and passed back through the entire network. That creates a strict sequential dependency, needs the whole computation graph held in memory, and has no clear counterpart in biological neural systems. Real neurons do not appear to have access to a global error term.

Predictive coding proposes something local instead. Each layer holds a latent state, emits a prediction of the layer above, and receives back only the discrepancy between that prediction and what actually arrived. It updates itself from that. Everything a layer needs is available at the layer.

The theory is well established in computational neuroscience. Whether you can build a working transformer language model on it was much less clear. That was the question I took on.

How the architecture works

The model keeps the familiar transformer shape, with token and position embeddings, multi-head attention, an MLP block and an output projection to vocabulary logits. What changes is the learning rule inside every one of those components.

Diagram of the PC-Transformer architecture showing embedding, attention, MLP and output layers connected by prediction and error pathways
Predictions flow one way, errors the other, and no gradient crosses a layer boundary.

Each layer runs an iterative refinement loop over T inference steps:

  1. Initialise the layer's latent state x.
  2. Predict the activity of the next layer, giving μ.
  3. Compare against what the next layer actually shows: ε = target − μ.
  4. Correct both the state and the weights from that error alone.
x(t+1) = x(t) + η · ε(t) ΔW = η · ε(t)ᵀ · x(t)

Both update rules read only quantities available at that layer. T, the number of times this settling loop runs before weights are committed, turns out to be the most consequential knob in the system.

Detail of a single predictive coding layer showing latent state, prediction, error computation and the local weight update
Inside one predictive coding layer: state, prediction, error, update, all local.

Adapting attention

Attention was the hardest part to port. An early version updated the query, key and value projections through a shared pathway, which pushed the heads toward redundancy and stopped them specialising. I reworked it so each head's Q, K and V update independently from their own error signals, restoring the head diversity that makes multi-head attention worth having. I also added optional lateral connections as a separate module, so the inhibitory dynamics predictive coding theory calls for could be tested without entangling them with the core update rule.

Making the experiments work

A research claim is only as good as the infrastructure that tests it. Much of this work was making the experiments fast enough, controlled enough and repeatable enough to be worth believing.

  • I adapted distributed training for predictive coding rather than relying on the standard gradient reduction approach used by DistributedDataParallel (DDP). Since predictive coding updates weights locally and does not produce a global gradient to average across GPUs, I synchronised the model parameters after each update instead.
  • Predictive coding performs multiple inference steps for each input, making autoregressive generation significantly more expensive than a standard single pass forward loop. I added key-value caching so previously computed attention keys and values could be reused across generation steps, reducing redundant computation and making iterative sampling practical.
  • I replaced the borrowed GPT-2 tokenizer with a BPE tokenizer trained directly on the project corpus. This produced a vocabulary matched to the data rather than relying on tokenization learned from a different language distribution, giving the model more appropriate subword units for the experiments.
  • I used Optuna to search over both architectural choices and predictive coding parameters. The first phase explored the broader configuration space, while the second narrowed the search around promising regions. The objective combined model energy and language-model perplexity, so configurations had to balance predictive coding behaviour with actual language modelling performance.
  • Every run was seeded and recorded separately, with benchmark configurations committed to the repository and structured logs used to track metrics and compare experiments across runs.

Initial results

The first comparison used the same training setup for both models, with the PC-Transformer running for 2 inference steps. Both models learned over the five training epochs, but the gap in language modelling performance was large.

The PC-Transformer reached a test perplexity of 834.32, compared with 117.35 for the Standard Transformer. Its validation perplexity also remained around 830 by the end of training, while its energy decreased from roughly 1.38 to 1.35. The Standard Transformer reached a test cross-entropy loss of 4.77, compared with 6.73 for the PC-Transformer.

At this stage, the model could clearly train without backpropagation, but its language modelling performance was far behind the standard baseline. This raised the next question: was the gap caused by the learning method, or was the PC-Transformer simply not being given enough inference steps to settle its latent states?

Training and validation curves comparing PC-Transformer and Standard Transformer performance across epochs
Training and validation curves comparing PC-Transformer and Standard Transformer performance across epochs

What the controlled experiments showed

With the infrastructure in place I ran controlled sweeps varying exactly one thing at a time, holding every other hyperparameter at its tuned value, five epochs per configuration.

Iterative inference is where the gains are

Three charts showing training energy, validation energy and validation perplexity against the number of inference iterations T from 2 to 10
Sweeping T from 2 to 10. Perplexity falls steadily; energy climbs, then spikes.

Validation perplexity improves monotonically as T rises, from roughly 835 at T=2 down to about 100 at T=10. That is an order-of-magnitude gain from nothing but letting each layer settle longer before committing an update. Depth of inference substitutes for a good deal of what backpropagation buys you.

Energy tells a different story. It rises gently through T=8 and then jumps sharply at T=10, to roughly ten times its previous value, at exactly the point where perplexity is at its best. At the useful end of the range the two metrics move in opposite directions.

Stacking more blocks does not help

Three charts showing training energy, validation energy and validation perplexity against the number of transformer blocks from 2 to 6
Sweeping depth from 2 to 6 blocks. Energy rises steadily; perplexity goes nowhere.

Across 2 to 6 blocks, validation perplexity stays inside a narrow band of roughly 834 to 851, with no trend, just noise. Energy meanwhile increases with every block added. More depth costs more compute and more prediction error, and returns nothing.

The finding: for this architecture at this scale, iterative inference is the axis that matters and architectural depth is not. Spend compute on letting layers settle, not on stacking more of them.

Energy taken alone would have pointed the wrong way on both sweeps. It prefers shallow models and few iterations, which is the opposite of what the language modelling metric wants.

Where it stands

This is ongoing research, so I would rather be precise than promotional about the results. At this scale, the predictive coding transformer does not match a well-tuned backpropagation baseline on perplexity. That was not the near-term goal. The focus was to understand whether the architecture trains stably, responds consistently to its hyperparameters, and reveals useful patterns about where predictive coding has an effect. So far, the experiments provide evidence for all three.

The work continues in FabricPC, a JAX library from SingularityNET where I contributed the transformer components. It trains the same architecture with either predictive coding or backpropagation, which makes the comparison a much more direct one.

Two findings stood out to me. The first was a negative result: increasing depth did not produce a meaningful improvement. This only became clear through controlled sweeps with pinned configurations. The second was that the choice of objective matters more than it might initially seem. Optimising the theoretically principled objective alone would have selected models that performed worse, showing that the objective itself is an important research decision rather than just an implementation detail.

← Back to all work