Does it replicate?
A Nature Communications paper claimed a biologically motivated alternative to backpropagation could learn generative models competitively. We checked.
The premise
In 2022, Ororbia and Kifer published the Neural Coding Framework for Learning Generative Models, a system where learning is driven by predictive processing rather than a global gradient. The results were strong enough to be worth taking seriously, which is exactly the kind of result that deserves independent verification rather than citation.
Replication is unglamorous and it is the part of research that most needs doing. This project asked three questions. Do the published numbers reproduce? How do these models compare to conventional generative models under matched conditions? And what do the learned representations actually contain?
What we built
We worked from the NGC model family and rebuilt the evaluation harness around it, so every model could be trained and assessed through one consistent path rather than through the ad hoc scripts each had accumulated.
Three axes of evaluation
- Reconstruction quality. How faithfully a model rebuilds an input it has compressed.
- Likelihood estimation. Fitting a Gaussian mixture density over the latent space to estimate how well the model captures the data distribution, rather than judging sample quality by eye.
- Downstream classification. Training a classifier on the frozen latent representations, with early stopping, to test whether what the model learned is useful beyond reconstruction.
I added a fourth probe that was not in the original evaluation: pattern completion via masked MSE. Mask part of an input, ask the model to fill it in, and measure error only on the masked region. A model that reconstructs well but completes badly has learned to copy rather than to represent, and that distinction does not show up in a reconstruction score.
The control group
A replication that only runs the proposed method tells you very little. We trained backpropagation baselines under the same conditions, including a Gaussian variational autoencoder, a regularised autoencoder and an adversarial autoencoder, so every NGC number had a conventionally trained counterpart beside it.
On reproducibility. The original work targeted Ubuntu 18.04 and a specific TensorFlow lineage, which by the time we started was genuinely difficult to stand up. We packaged the environment as a Docker image and moved every hyperparameter into version-controlled config files. The comparison can be re-run from a clean machine with two commands, which is the minimum bar for a replication study to mean anything.
My contribution
I led this work and wrote roughly 70% of the repository.
- Centralised the training and evaluation workflow. Before, each model had its own script and the numbers were not strictly comparable. Afterwards, one modular pipeline.
- Moved every model to declarative config files with parameters matching those stated in the paper, so any deviation from the original setup was explicit rather than buried in code.
- Implemented the masked-MSE pattern completion metric and the latent-space classifier.
- Built the density estimation path for likelihood evaluation, including how fitted mixture models are persisted and reloaded.
- Wrote the experiment runner so the full set of models could be trained and analysed in one sweep, and reviewed the team's contributions into the repository.
Why this shaped everything after
This project is where the predictive coding work began. Rebuilding someone else's framework carefully enough to test it is the fastest way to understand its real mechanics, including the parts papers tend to compress into a sentence. The instincts I brought to PC-Transformers came from here: pin the configs, seed the runs, make one pipeline serve every model, and never compare numbers produced by different code paths.