Saumya Gandhi, Ritu Gala, Jonah Kornberg et al. · arXiv (Cornell University) · 2023
The Forward Forward algorithm, proposed by Geoffrey Hinton in November 2022, is a novel method for training neural networks as an alternative to backpropagation. In this project, we replicate Hinton's experiments on the MNIST dataset, and subsequently extend the scope of the method with two significant contributions. First, we establish a baseline performance for the Forward Forward network on the IMDb movie reviews dataset. As far as we know, our results on this sentiment analysis task marks the first instance of the algorithm's extension beyond computer vision. Second, we introduce a novel pyramidal optimization strategy for the loss threshold - a hyperparameter specific to the Forward Forward method. Our pyramidal approach shows that a good thresholding strategy causes a difference of up to 8% in test error. Lastly, we perform visualizations of the trained parameters and derived several significant insights, such as a notably larger (10-20x) mean and variance in the weights acquired by the Forward Forward network. Repository: https://github.com/Ads-cmu/ForwardForward
Cited by 5Open access
Xing Chen, Dongshu Liu, Jérémie Laydevant et al. · arXiv (Cornell University) · 2024
Agents that operate autonomously benefit from lifelong learning capabilities. However, compatible training algorithms must comply with the decentralized nature of these systems, which imposes constraints on both the parameter counts and the computational resources. The Forward-Forward (FF) algorithm is one of these. FF relies only on feedforward operations, the same used for inference, for optimizing layer-wise objectives. This purely forward approach eliminates the need for transpose operations required in traditional backpropagation. Despite its potential, FF has failed to reach state-of-the-art performance on most standard benchmark tasks, in part due to unreliable negative data generation methods for unsupervised learning. In this work, we propose the Self-Contrastive Forward-Forward (SCFF) algorithm, a competitive training method aimed at closing this performance gap. Inspired by standard self-supervised contrastive learning for vision tasks, SCFF generates positive and negative inputs applicable across various datasets. The method demonstrates superior performance compared to existing unsupervised local learning algorithms on several benchmark datasets, including MNIST, CIFAR-10, STL-10, and Tiny ImageNet. We extend FF's application to training recurrent neural networks, expanding its utility to sequential data tasks. These findings pave the way for high-accuracy, real-time learning on resource-constrained edge devices.
Cited by 0Open access
Xing Chen, Dongshu Liu, Jérémie Laydevant et al. · Nature Communications · 2025
Agents that operate autonomously benefit from lifelong learning capabilities. However, compatible training algorithms must comply with the decentralized nature of these systems which imposes constraints on both the parameters counts and the computational resources. The Forward-Forward (FF) algorithm is one of these. FF relies only on feedforward operations, the same used for inference, for optimizing layer-wise objectives. This purely forward approach eliminates the need for transpose operations required in traditional backpropagation. Despite its potential, FF has failed to reach state-of-the-art performance on most standard benchmark tasks, in part due to unreliable negative data generation methods for unsupervised learning. In this work, we propose Self-Contrastive Forward-Forward (SCFF) algorithm, a competitive training method aimed at closing this performance gap. Inspired by standard self-supervised contrastive learning for vision tasks, SCFF generates positive and negative inputs applicable across various datasets. The method demonstrates superior performance compared to existing unsupervised local learning algorithms on several benchmark datasets, including MNIST, CIFAR-10, STL-10 and Tiny ImageNet. We extend FF's application to training recurrent neural networks, expanding its utility to sequential data tasks. These findings pave the way for high-accuracy, real-time learning on resource-constrained edge devices.
Cited by 6Open access
John L. Fan · 2001
Cited by 12
Abel A. Reyes-Angulo, Sidike Paheding · 2024
The back-propagation algorithm has long been the de-facto standard in optimizing weights and biases in neural networks, particularly in cutting-edge deep-learning models. Its widespread adoption in fields like natural language processing, computer vision, and remote sensing has revolutionized automation in various tasks. The popularity of back-propagation stems from its ability to achieve outstanding performance in tasks such as classification, detection, and segmentation. Nevertheless, back-propagation is not without its limitations, encompassing sensitivity to initial conditions, vanishing gradients, overfitting, and computational complexity. The recent introduction of a forward-forward algorithm (FFA), which computes local goodness functions to optimize network parameters, alleviates the dependence on substantial computational resources and the constant need for architectural scaling. This study investigates the application of FFA for hyperspectral image classification. Experimental results and comparative analysis are provided with the use of the traditional back-propagation algorithm. Preliminary results show the potential behind FFA and its promises.
Cited by 3
Shin-ya Matsushita · Numerical Algebra Control and Optimization · 2022
In this paper, we propose a forward-backward-forward splitting algorithm with alternated inertial step for finding a zero of the sum of two monotone operators in a real Hilbert space. Then, by making use of primal-dual techniques we derive the alternated inertial primal-dual splitting algorithm of forward-backward-forward type for solving structured monotone inclusion problems involving parallel sums and compositions of maximally monotone operators with linear continuous ones and convex minimization problems. Finally, we demonstrate numerically that our proposed algorithm performs better than the existing algorithms.
Cited by 1Open access
Bob Adamson · ArXiv.org · 2025
The Forward-Forward algorithm is an alternative learning method which consists of two forward passes rather than a forward and backward pass employed by backpropagation. Forward-Forward networks employ layer local loss functions which are optimized based on the layer activation for each forward pass rather than a single global objective function. This work explores the dynamics of model and layer accuracy changes in Forward-Forward networks as training progresses in pursuit of a mechanistic understanding of their internal behavior. Treatments to various system characteristics are applied to investigate changes in layer and overall model accuracy as training progresses, how accuracy is impacted by layer depth, and how strongly individual layer accuracy is correlated with overall model accuracy. The empirical results presented suggest that layers deeper within Forward-Forward networks experience a delay in accuracy improvement relative to shallower layers and that shallower layer accuracy is strongly correlated with overall model accuracy.
Cited by 0Open access
Guy Lorberbom, Itai Gat, Yossi Adi et al. · Proceedings of the AAAI Conference on Artificial Intelligence · 2024
Backpropagation, which uses the chain rule, is the de-facto standard algorithm for optimizing neural networks nowadays. Recently, Hinton (2022) proposed the forward-forward algorithm, a promising alternative that optimizes neural nets layer-by-layer, without propagating gradients throughout the network. Although such an approach has several advantages over back-propagation and shows promising results, the fact that each layer is being trained independently limits the optimization process. Specifically, it prevents the network's layers from collaborating to learn complex and rich features. In this work, we study layer collaboration in the forward-forward algorithm. We show that the current version of the forward-forward algorithm is suboptimal when considering information flow in the network, resulting in a lack of collaboration between layers of the network. We propose an improved version that supports layer collaboration to better utilize the network structure, while not requiring any additional assumptions or computations. We empirically demonstrate the efficacy of the proposed version when considering both information flow and objective metrics. Additionally, we provide a theoretical motivation for the proposed method, inspired by functional entropy theory.
Cited by 6Open access
Sudhanshu Thakur, Reha Dhawan, Parth Bhargava et al. · Communications in computer and information science · 2024
Cited by 0
Ortiz Torres, Mauricio, Lange, Markus, Raulf, Arne Peter · elib (German Aerospace Center) · 2025
The Forward-Forward algorithm has evolved in machine learning research, tackling more complex tasks that mimic real-life applications. In recent years, it has been improved by several techniques to perform better than its original version, handling a challenging dataset like CIFAR10 without losing its flexibility and low memory usage. We have shown in our results that improvements are achieved through a combination of convolutional channel grouping, learning rate schedules, and independent block structures during training that lead to a 20% decrease in test error percentage. Additionally, to approach further implementations on low-capacity hardware projects, we have presented a series of lighter models that achieve low test error percentages within (21±3)% and a number of trainable parameters between 164,706 and 754,386. This serves as a basis for our future study on complete verification and validation of these kinds of neural networks.
Cited by 0Open access
Amin Aminifar, Baichuan Huang, Azra Abtahi et al. · Frontiers in artificial intelligence and applications · 2024
The human brain performs tasks with an outstanding energy efficiency, i.e., with approximately 20 Watts. The state-of-the-art Artificial/Deep Neural Networks (ANN/DNN), on the other hand, have recently been shown to consume massive amounts of energy. The training of these ANNs/DNNs is done almost exclusively based on the back-propagation algorithm, which is known to be biologically implausible. This has led to a new generation of forward-only techniques, including the Forward-Forward algorithm. In this paper, we propose a lightweight inference scheme specifically designed for DNNs trained using the Forward-Forward algorithm. We have evaluated our proposed lightweight inference scheme in the case of the MNIST and CIFAR datasets, as well as two real-world applications, namely, epileptic seizure detection and cardiac arrhythmia classification using wearable technologies, where complexity overheads/energy consumption is a major constraint, and demonstrate its relevance. Our code is available at https://github.com/AminAminifar/LightFF.
Cited by 5Open access
Lică, Mircea-Tudor, Dinucu-Jianu, David · arXiv (Cornell University) · 2023
This paper aims to explore the separation of the two forward passes in the Forward-Forward algorithm from a biological perspective in the context of sleep. We show the size of the gap between the sleep and awake phase influences the learning capabilities of the algorithm and highlight the importance of negative data in diminishing the devastating effects of sleep deprivation.
Cited by 0Open access
Mauricio Ortiz Torres, Markus Lange, Arne Raulf · elib (German Aerospace Center) · 2025
The Forward-Forward algorithm has evolved in machine learning research, tackling more complex tasks that mimic real-life applications. In recent years, it has been improved by several techniques to perform better than its original version, handling a challenging dataset like CIFAR10 without losing its flexibility and low memory usage. We have shown in our results that improvements are achieved through a combination of convolutional channel grouping, learning rate schedules, and independent block structures during training that lead to a 20% decrease in test error percentage. Additionally, to approach further implementations on low-capacity hardware projects, we have presented a series of lighter models that achieve low test error percentages within (21±3)% and a number of trainable parameters between 164,706 and 754,386. This serves as a basis for our future study on complete verification and validation of these kinds of neural networks.
Cited by 0Open access
Marta E. Torres, Markus Lange, Arne Raulf · ArXiv.org · 2025
The Forward-Forward algorithm has evolved in machine learning research, tackling more complex tasks that mimic real-life applications. In the last years, it has been improved by several techniques to perform better than its original version, handling a challenging dataset like CIFAR10 without losing its flexibility and low memory usage. We have shown in our results that improvements are achieved through a combination of convolutional channel grouping, learning rate schedules, and independent block structures during training that lead to a 20\% decrease in test error percentage. Additionally, to approach further implementations on low-capacity hardware projects, we have presented a series of lighter models that achieve low test error percentages within (21$\pm$3)\% and number of trainable parameters between 164,706 and 754,386. This serves as a basis for our future study on complete verification and validation of these kinds of neural networks.
Cited by 0Open access
Spencer Connolly · Cornerstone (Minnesota State University, Mankato) · 2025
The Forward-Forward algorithm (FF) is yet another novel invention by Geoffrey Hinton, the creator of the famous backpropagation algorithm (BP). Since its proposal, many papers have been published exploring its potential, and good progress has been made in increasing its viability. Though FF continually falls short of BP, its purpose is not to replace BP and preliminary research shows that there is plenty of room for growth. In this paper, we present a literature review for FF algorithms applied to Convolution Neural Networks (CNN) for image classification tasks and set the stage for applying FF to more complex datasets. The proposed Ripple algorithm creates spatially distributed class representations that maintain label visibility throughout convolutional layers, addressing a key limitation in applying FF to CNN architectures. Our Ripple algorithm plays a key role in expanding the embedding space which provides the goodness function greater flexibility to fit the data. We evaluate our approach using the MNIST and CIFAR-10 datasets. Our experimental results demonstrate that the Ripples technique enables FF-trained CNNs to achieve competitive performance while maintaining the theoretical advantages of forward-only learning.
Cited by 0Open access
Marc Lassonde, Ludovic Nagesseur · Journal of Mathematical Analysis and Applications · 2013
Cited by 3Open access
Sepp Kollmorgen (248487), Richard H. R. Hahnloser (201245) · Figshare · 2015
Cited by 0Open access
Mitra Bakhshi · 2024
Incorporating the Forward-Forward algorithm into neural network training represents a transformative shift from traditional methods, introducing a dual-forward mechanism that streamlines the learning process by bypassing the complexities of derivative propagation. This method is noted for its simplicity and efficiency and involves executing two forward passes-the first with actual data to promote positive reinforcement, and the second with synthetically generated negative data to enable discriminative learning. Our experiments confirm that the ForwardForward algorithm is not merely an experimental novelty but a viable training strategy that competes robustly with conventional multi-layer perceptron (MLP) architectures. To overcome the limitations inherent in traditional saliency techniques, which predominantly rely on gradient-based methods, we developed a bespoke saliency algorithm specifically tailored for the ForwardForward framework. This innovative algorithm enhances the intuitive understanding of feature importance and network decision-making, providing clear visualizations of the data features most influential in model predictions. By leveraging this specialized saliency method, we gain deeper insights into the internal workings of the model, significantly enhancing our interpretative capabilities beyond those offered by standard approaches. Our evaluations, utilizing the MNIST and Fashion MNIST datasets, demonstrate that our method performs comparably to traditional MLP-based models.
Cited by 3
Yonghong Yao, Shang-Hong Ke, Yekini Shehu et al. · Optimization · 2025
Cited by 0
Gerardo Hernández-Hernández, Alejandro Rodríguez-Molina, Máximo Eduardo Sánchez-Gutiérrez et al. · 2024
This paper proposes using the Forward-Forward (FF) algorithm to train hybrid neural networks. We implement the FF algorithm in Morphological-Linear Neural Networks (MLNNs), which consist of two layers. The first layer is made up of morphological neurons and the second consists of perceptron-type neurons. For this we adapt the FF training algorithm in the calculation of the goodness and loss functions derived from the morphological neuron layer. The calculation of the goodness and the loss functions was not modified for the perceptron-type neuron layer. The experimental results show that it is possible to train morphological-linear neural networks with the FF algorithm, obtaining results similar to those obtained with the original FF algorithm. The comparison between the original training algorithm and the proposed one is carried out with low-dimensionality and binary classification datasets from the IMDB natural language processing dataset. Likewise, the ability of the FF algorithm is analyzed and compared in terms of its ability to disentangle patterns for classification tasks.
Cited by 0