Aadarsh Agarwal https://www.aadarwal.com/blog/ Technical notes by Aadarsh Agarwal. quarto-1.10.18 Mon, 05 Oct 2026 05:00:00 GMT Dealing with Context Blowup in Open Source Models https://www.aadarwal.com/blog/posts/context-blowup-in-open-source-models/ Cross-listed with the nonlocally blog.

]]> https://www.aadarwal.com/blog/posts/context-blowup-in-open-source-models/ Mon, 05 Oct 2026 05:00:00 GMT Lean scaling laws https://www.aadarwal.com/blog/posts/lean-scaling-laws/
Carbonara, Renzo. n.d. A Type of Programming. https://atypeofprogramming.com/.
Gwern. 2025. “Lean Software Scaling Laws.” https://gwern.net/lean-scaling.
]]> https://www.aadarwal.com/blog/posts/lean-scaling-laws/ Mon, 05 Oct 2026 05:00:00 GMT Learning without a backward pass https://www.aadarwal.com/blog/posts/forward-forward/ Notes on Hinton (2022).

Forward-Forward

Layer input (previous layer’s activities), normalized by length only:

Goodness, and probability that the input is positive (real) data:

Local loss:

Update, using :

Inputs to the update: , , , all local to the layer.

Negative data:

  • supervised: one-hot label in the first 10 pixels; correct label = positive, wrong label = negative
  • unsupervised: , with a random bit image blurred by and thresholded at 0.5

Classification (supervised), one pass per label :

Backprop

Inputs to the update: , at every layer from up, stored and .

Physical layers :

Each is a derivative of the physical device at its operating point.

Cost per example, digital

= arithmetic of one forward pass, = number of layers.

arithmetic activations held
backprop layers
backprop + checkpointing (Chen et al. 2016) layers
FF ( with hard-negative mining) 1 layer

MNIST (Hinton 2022, sec. 3): backprop ≈ 20 epochs, FF 60 epochs, so FF ≈ 4–5× the arithmetic.

45 nm energies (Horowitz 2014): FP32 multiply 3.7 pJ, FP32 add 0.9 pJ, DRAM access 1.3–2.6 nJ.

Activation stash vs. arithmetic for a layer of width , at 640 pJ per 32-bit DRAM access (Han et al. 2016):

Measured, ResNet-18 on CIFAR-10, Jetson Orin Nano (Ma et al. 2025):

training energy
FP32 backprop 18,764 J
INT8 backprop 10,291 J
INT8 FF (loss not purely local) 9,926 J

Physical hardware

Resistive crossbar (Gokmen and Vlasov 2016):

  • forward:
  • backward: drive the columns to multiply by
  • update: by pulse coincidence, all devices in parallel

FF update in the same rank-1 form: , .

Optical goodness: total optical power.

Ways to get gradients on physical hardware:

method requires
physics-aware training (Wright et al. 2022) digital model of the device for the backward pass
in situ backprop (Hughes et al. 2018; Pai et al. 2023) reciprocal device, forward + adjoint field measurements, digital nonlinearities
weight perturbation (Jabri and Flower 1992) no model; learning rate
ADCs + digital gradients ADCs: 58% of tile power in ISAAC (Shafiee et al. 2016)
equilibrium propagation (Scellier and Bengio 2017) energy-based system; free phase + weakly clamped phase
FF , , per layer

Reference points: optical MNIST inference at ≈ 90% accuracy with < 1 detected photon per multiplication (Wang et al. 2022); human brain ≈ 20 W (Balasubramanian 2021).

Results

Test error, Hinton (2022) §3–4:

task backprop FF
MNIST, fully connected, permutation-invariant ≈ 1.4% (≈ 20 epochs) 1.36% (60 epochs, 4 × 2000 ReLUs)
MNIST, unsupervised + linear classifier — 1.37% (100 epochs); 1.16% local receptive fields (60 epochs)
MNIST, ±2-pixel jitter ≈ 0.6% (CNN) 0.64% (500 epochs)
CIFAR-10, local receptive fields, 2 / 3 layers 37% / 39% 41% / 41%

Since:

work result
Oguz et al. (2023) multimode fiber as a fixed layer, MNIST subset: FF 90.8% → 94.4% accuracy with the fiber; LeNet-5 backprop 95.0%
Momeni et al. (2023) model-free FF-style local learning on acoustic and microwave hardware and a modeled optical system; vowel classification
Dooms et al. (2024) CIFAR-10 ≈ 84% (backprop 89%); CIFAR-100 36% (backprop 60%)

Claims in the paper

  • §1: FF “is unlikely to replace backpropagation for applications where power is not an issue.”
  • §7: “FF has no problem learning if we insert a black box between layers that are being trained by FF.”
  • §8: “The use of two forward passes instead of a forward and a backward pass should make these A-to-D converters unnecessary.”
  • §9: “If you want your trillion parameter neural net to only consume a few watts, mortal computation may be the only option.”

References

Balasubramanian, Vijay. 2021. “Brain Power.” Proceedings of the National Academy of Sciences 118 (32): e2107022118. https://doi.org/10.1073/pnas.2107022118.
Chen, Tianqi, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training Deep Nets with Sublinear Memory Cost. https://arxiv.org/abs/1604.06174.
Dooms, Thomas, Ing Jyh Tsang, and Jose Oramas. 2024. “The Trifecta: Three Simple Techniques for Training Deeper Forward-Forward Networks.” Transactions on Machine Learning Research. https://arxiv.org/abs/2311.18130.
Gokmen, Tayfun, and Yurii Vlasov. 2016. “Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices: Design Considerations.” Frontiers in Neuroscience 10: 333. https://doi.org/10.3389/fnins.2016.00333.
Han, Song, Xingyu Liu, Huizi Mao, et al. 2016. “EIE: Efficient Inference Engine on Compressed Deep Neural Network.” 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), 243–54. https://doi.org/10.1109/ISCA.2016.30.
Hinton, Geoffrey. 2022. The Forward-Forward Algorithm: Some Preliminary Investigations. https://arxiv.org/abs/2212.13345.
Horowitz, Mark. 2014. “Computing’s Energy Problem (and What We Can Do about It).” 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), 10–14. https://doi.org/10.1109/ISSCC.2014.6757323.
Hughes, Tyler W., Momchil Minkov, Yu Shi, and Shanhui Fan. 2018. “Training of Photonic Neural Networks Through in Situ Backpropagation and Gradient Measurement.” Optica 5 (7): 864–71. https://doi.org/10.1364/OPTICA.5.000864.
Jabri, Marwan, and Barry Flower. 1992. “Weight Perturbation: An Optimal Architecture and Learning Technique for Analog VLSI Feedforward and Recurrent Multilayer Networks.” IEEE Transactions on Neural Networks 3 (1): 154–57. https://doi.org/10.1109/72.105429.
Ma, Jingxiao, Priyadarshini Panda, and Sherief Reda. 2025. “FF-INT8: Efficient Forward-Forward DNN Training on Edge Devices with INT8 Precision.” 62nd Design Automation Conference (DAC). https://arxiv.org/abs/2506.22771.
Momeni, Ali, Babak Rahmani, Matthieu Malléjac, Philipp del Hougne, and Romain Fleury. 2023. “Backpropagation-Free Training of Deep Physical Neural Networks.” Science 382 (6676): 1297–303. https://doi.org/10.1126/science.adi8474.
Oguz, Ilker, Junjie Ke, Qifei Weng, et al. 2023. “Forward–Forward Training of an Optical Neural Network.” Optics Letters 48 (20): 5249–52. https://doi.org/10.1364/OL.496884.
Pai, Sunil, Zhanghao Sun, Tyler W. Hughes, et al. 2023. “Experimentally Realized in Situ Backpropagation for Deep Learning in Photonic Neural Networks.” Science 380 (6643): 398–404. https://doi.org/10.1126/science.ade8450.
Scellier, Benjamin, and Yoshua Bengio. 2017. “Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation.” Frontiers in Computational Neuroscience 11: 24. https://doi.org/10.3389/fncom.2017.00024.
Shafiee, Ali, Anirban Nag, Naveen Muralimanohar, et al. 2016. “ISAAC: A Convolutional Neural Network Accelerator with in-Situ Analog Arithmetic in Crossbars.” 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), 14–26. https://doi.org/10.1109/ISCA.2016.12.
Wang, Tianyu, Shi-Yuan Ma, Logan G. Wright, Tatsuhiro Onodera, Brian C. Richard, and Peter L. McMahon. 2022. “An Optical Neural Network Using Less Than 1 Photon Per Multiplication.” Nature Communications 13: 123. https://doi.org/10.1038/s41467-021-27774-8.
Wright, Logan G., Tatsuhiro Onodera, Martin M. Stein, et al. 2022. “Deep Physical Neural Networks Trained with Backpropagation.” Nature 601 (7894): 549–55. https://doi.org/10.1038/s41586-021-04223-6.
]]> https://www.aadarwal.com/blog/posts/forward-forward/ Mon, 05 Oct 2026 05:00:00 GMT The Old World is Dead https://www.aadarwal.com/blog/posts/the-old-world-is-dead/ https://www.aadarwal.com/blog/posts/the-old-world-is-dead/ Mon, 05 Oct 2026 05:00:00 GMT What will be New in India https://www.aadarwal.com/blog/posts/what-will-be-new-in-india/ https://www.aadarwal.com/blog/posts/what-will-be-new-in-india/ Mon, 05 Oct 2026 05:00:00 GMT