Learning without a backward pass
Notes on Hinton (2022).
Forward-Forward
Layer input x (previous layer’s activities), normalized by length only:
\hat x = \frac{x}{\lVert x \rVert}, \qquad y = \mathrm{ReLU}(W \hat x)
Goodness, and probability that the input is positive (real) data:
G = \sum_j y_j^2, \qquad p = \sigma(G - \theta)
Local loss:
\mathcal{L} = \begin{cases} -\log p & \text{(positive)}\\ -\log(1 - p) & \text{(negative)} \end{cases}
Update, using \partial G / \partial W = 2\, y\, \hat x^{\top}:
\Delta W = \begin{cases} +2\eta\,(1-p)\; y\, \hat x^{\top} & \text{(positive)}\\ -2\eta\, p\; y\, \hat x^{\top} & \text{(negative)} \end{cases}
Inputs to the update: \hat x, y, p, all local to the layer.
Negative data:
- supervised: one-hot label in the first 10 pixels; correct label = positive, wrong label = negative
- unsupervised: m \odot x_1 + (1 - m) \odot x_2, with m a random bit image blurred by [\tfrac14, \tfrac12, \tfrac14] and thresholded at 0.5
Classification (supervised), one pass per label c:
\hat c = \arg\max_c \sum_{\ell \ge 2} G_\ell(x, c)
Backprop
\delta_\ell = \big(W_{\ell+1}^{\top}\, \delta_{\ell+1}\big) \odot f'(a_\ell), \qquad \Delta W_\ell = -\eta\, \delta_\ell\, h_{\ell-1}^{\top}
Inputs to the update: W_{\ell+1}, \dots, W_L, f' at every layer from \ell up, stored a_\ell and h_{\ell-1}.
Physical layers h_{\ell+1} = f_\ell(h_\ell; \theta_\ell):
\nabla_{\theta_\ell} L = J_{\theta_\ell}^{\top}\, J_{\ell+1}^{\top} \cdots J_{L}^{\top}\, \nabla_{h_{L+1}} L, \qquad J_k = \frac{\partial f_k}{\partial h_k}, \quad J_{\theta_\ell} = \frac{\partial f_\ell}{\partial \theta_\ell}
Each J_k is a derivative of the physical device at its operating point.
Cost per example, digital
F = arithmetic of one forward pass, L = number of layers.
| arithmetic | activations held | |
|---|---|---|
| backprop | 3F | L layers |
| backprop + checkpointing (Chen et al. 2016) | \approx 4F | O(\sqrt{L}) layers |
| FF | 4F (5F with hard-negative mining) | 1 layer |
MNIST (Hinton 2022, sec. 3): backprop ≈ 20 epochs, FF 60 epochs, so FF ≈ 4–5× the arithmetic.
45 nm energies (Horowitz 2014): FP32 multiply 3.7 pJ, FP32 add 0.9 pJ, DRAM access 1.3–2.6 nJ.
Activation stash vs. arithmetic for a layer of width d, at 640 pJ per 32-bit DRAM access (Han et al. 2016):
\frac{E_{\text{stash}}}{E_{\text{arith}}} \approx \frac{2 \times 640\ \text{pJ}}{3d \times 4.6\ \text{pJ}} \approx \frac{93}{d}
Measured, ResNet-18 on CIFAR-10, Jetson Orin Nano (Ma et al. 2025):
| training | energy |
|---|---|
| FP32 backprop | 18,764 J |
| INT8 backprop | 10,291 J |
| INT8 FF (loss not purely local) | 9,926 J |
Physical hardware
Resistive crossbar (Gokmen and Vlasov 2016):
- forward: I_i = \sum_j G_{ij} V_j
- backward: drive the columns to multiply by W^{\top}
- update: \Delta G_{ij} \propto x_i\, \delta_j by pulse coincidence, all devices in parallel
FF update in the same rank-1 form: \Delta W = s\, y\, \hat x^{\top}, s \in \{\, 2\eta(1-p),\ -2\eta p \,\}.
Optical goodness: G = \sum_j |E_j|^2 \propto total optical power.
Ways to get gradients on physical hardware:
| method | requires |
|---|---|
| physics-aware training (Wright et al. 2022) | digital model of the device for the backward pass |
| in situ backprop (Hughes et al. 2018; Pai et al. 2023) | reciprocal device, forward + adjoint field measurements, digital nonlinearities |
| weight perturbation (Jabri and Flower 1992) | no model; learning rate \propto 1/(\text{number of perturbed variables}) |
| ADCs + digital gradients | ADCs: 58% of tile power in ISAAC (Shafiee et al. 2016) |
| equilibrium propagation (Scellier and Bengio 2017) | energy-based system; free phase + weakly clamped phase |
| FF | \hat x, y, p per layer |
Reference points: optical MNIST inference at ≈ 90% accuracy with < 1 detected photon per multiplication (Wang et al. 2022); human brain ≈ 20 W (Balasubramanian 2021).
Results
Test error, Hinton (2022) §3–4:
| task | backprop | FF |
|---|---|---|
| MNIST, fully connected, permutation-invariant | ≈ 1.4% (≈ 20 epochs) | 1.36% (60 epochs, 4 × 2000 ReLUs) |
| MNIST, unsupervised + linear classifier | — | 1.37% (100 epochs); 1.16% local receptive fields (60 epochs) |
| MNIST, ±2-pixel jitter | ≈ 0.6% (CNN) | 0.64% (500 epochs) |
| CIFAR-10, local receptive fields, 2 / 3 layers | 37% / 39% | 41% / 41% |
Since:
| work | result |
|---|---|
| Oguz et al. (2023) | multimode fiber as a fixed layer, MNIST subset: FF 90.8% → 94.4% accuracy with the fiber; LeNet-5 backprop 95.0% |
| Momeni et al. (2023) | model-free FF-style local learning on acoustic and microwave hardware and a modeled optical system; vowel classification |
| Dooms et al. (2024) | CIFAR-10 ≈ 84% (backprop 89%); CIFAR-100 36% (backprop 60%) |
Claims in the paper
- §1: FF “is unlikely to replace backpropagation for applications where power is not an issue.”
- §7: “FF has no problem learning if we insert a black box between layers that are being trained by FF.”
- §8: “The use of two forward passes instead of a forward and a backward pass should make these A-to-D converters unnecessary.”
- §9: “If you want your trillion parameter neural net to only consume a few watts, mortal computation may be the only option.”