Desmond Y. M. Tang · arXiv (Cornell University) · 2023
The backpropagation algorithm, despite its widespread use in neural network learning, may not accurately emulate the human cortex's learning process. Alternative strategies, such as the Forward-Forward Algorithm (FFA), offer a closer match to the human cortex's learning characteristics. However, the original FFA paper and related works on the Forward-Forward Algorithm only mentioned very limited types of neural network mechanisms and may limit its application and effectiveness. In response to these challenges, we propose an integrated method that combines the strengths of both FFA and shallow backpropagation, yielding a biologically plausible neural network training algorithm which can also be applied to various network structures. We applied this integrated approach to the classification of the Modified National Institute of Standards and Technology (MNIST) database, where it outperformed FFA and demonstrated superior resilience to noise compared to backpropagation. We show that training neural networks with the Integrated Forward-Forward Algorithm has the potential of generating neural networks with advantageous features like robustness.
Cited by 1Open access
Julien Becker (283239), Francis Maes (283240), Louis Wehenkel (32624) · Figshare · 2015
<p>In order to identify the relevant feature function set, the algorithm requires four components: a dataset, a list of candidate feature functions, a base learner and a criterion to optimize.</p>
Cited by 0Open access
National Cancer Institute · Definitions · 2020
Ope n Pe e r Re v ie w on Qe ios
Cited by 0Open access
2014
Cited by 1
Feifan Liu (361011), Hong Yu (6528) · Figshare · 2015
Cited by 0Open access
Süleyman Emir Akın, Toygar Akgün · 2024
The forward-forward algorithm is a recently introduced method that can be used to train artificial neural networks. Currently, the canonical approach to training artificial neural networks is using (batched) stochastic gradient descent with backpropagation to minimize a carefully designed cost function. As successful as it may be in realizing artificial neural networks on modern digital computers, the backpropagation algorithm is biologically implausible. Unlike backpropagation, the forward-forward algorithm does not propagate the partial derivatives of the cost function backwards throughout the whole network and does not require storing or re-computing layer activations. As a result, the forward-forward algorithm is closer to being biologically plausible. As mentioned in Hinton's original paper, the forward-forward algorithm is well-suited for low-power platforms, indicating the possibility of on-device/on-edge training. Coincidentally, event-based (neuromorphic) visual sensors, praised for their power efficiency and high dynamic range, are becoming increasingly commercially available. In this paper we apply the forward-forward algorithm to train relatively small network models to classify event-based visual inputs. We first implement the forward-forward algorithm to reproduce the results obtained with the MNIST dataset in the original paper. Next, we switch the input data to the event-based version of the MNIST dataset and propose novel modifications to the forward-forward training process to make it applicable to event-based visual inputs. Finally, we present a comparative performance analysis. All PyTorch scripts that generated the results in this paper are available here: https://github.com/SuleymanEmirAkin/Event-Based_Forward-Forward
Cited by 0
Luis M. Briceño-Arias, Damek Davis · arXiv (Cornell University) · 2017
Tseng's algorithm finds a zero of the sum of a maximally monotone operator and a monotone continuous operator by evaluating the latter twice per iteration. In this paper, we modify Tseng's algorithm for finding a zero of the sum of three operators, where we add a cocoercive operator to the inclusion. Since the sum of a cocoercive and a monotone-Lipschitz operator is monotone and Lipschitz, we could use Tseng's method for solving this problem, but implementing both operators twice per iteration and without taking into advantage the cocoercivity property of one operator. Instead, in our approach, although the {continuous monotone} operator must still be evaluated twice, we exploit the cocoercivity of one operator by evaluating it only once per iteration. Moreover, when the cocoercive or {continuous-monotone} operators are zero it reduces to Tseng's or forward-backward splittings, respectively, unifying in this way both algorithms. In addition, we provide a {preconditioned} version of the proposed method including non self-adjoint linear operators in the computation of resolvents and the single-valued operators involved. This approach allows us to {also} extend previous variable metric versions of Tseng's and forward-backward methods and simplify their conditions on the underlying metrics. We also exploit the case when non self-adjoint linear operators are triangular by blocks in the primal-dual product space for solving primal-dual composite monotone inclusions, obtaining Gauss-Seidel type algorithms which generalize several primal-dual methods available in the literature. Finally we explore {applications to the obstacle problem, Empirical Risk Minimization, distributed optimization and nonlinear programming and we illustrate the performance of the method via some numerical simulations.
Cited by 1Open access
Luis M. Briceño-Arias, Damek Davis · SIAM Journal on Optimization · 2018
Tseng's algorithm finds a zero of the sum of a maximally monotone operator and a monotone continuous operator by evaluating the latter twice per iteration. In this paper, we modify Tseng's algorithm for finding a zero of the sum of three operators, where we add a cocoercive operator to the inclusion. Since the sum of a cocoercive and a monotone-Lipschitz operator is monotone and Lipschitz, we could use Tseng's method for solving this problem, but implementing both operators twice per iteration and without taking into advantage the cocoercivity property of one operator. Instead, in our approach, although the continuous monotone operator must still be evaluated twice, we exploit the cocoercivity of one operator by evaluating it only once per iteration. Moreover, when the cocoercive or continuous-monotone operators are zero it reduces to Tseng's algorithm or forward-backward splittings, respectively, unifying in this way both algorithms. In addition, we provide a preconditioned version of the proposed method including non-self-adjoint linear operators in the computation of resolvents and the single-valued operators involved. This approach allows us to also extend previous variable metric versions of the Tseng and forward-backward methods and simplify their conditions on the underlying metrics. We also exploit the case in which non-self-adjoint linear operators are triangular by blocks in the primal-dual product space for solving primal-dual composite monotone inclusions, obtaining Gauss--Seidel-type algorithms which generalize several primal-dual methods available in the literature. Finally we explore applications to the obstacle problem, empirical risk minimization, distributed optimization, and nonlinear programming and we illustrate the performance of the method via some numerical simulations.
Cited by 79
Luis M. Briceño-Arias, Damek Davis · arXiv (Cornell University) · 2017
Tseng's algorithm finds a zero of the sum of a maximally monotone operator and a monotone continuous operator by evaluating the latter twice per iteration. In this paper, we modify Tseng's algorithm for finding a zero of the sum of three operators, where we add a cocoercive operator to the inclusion. Since the sum of a cocoercive and a monotone-Lipschitz operator is monotone and Lipschitz, we could use Tseng's method for solving this problem, but implementing both operators twice per iteration and without taking into advantage the cocoercivity property of one operator. Instead, in our approach, although the {continuous monotone} operator must still be evaluated twice, we exploit the cocoercivity of one operator by evaluating it only once per iteration. Moreover, when the cocoercive or {continuous-monotone} operators are zero it reduces to Tseng's or forward-backward splittings, respectively, unifying in this way both algorithms. In addition, we provide a {preconditioned} version of the proposed method including non self-adjoint linear operators in the computation of resolvents and the single-valued operators involved. This approach allows us to {also} extend previous variable metric versions of Tseng's and forward-backward methods and simplify their conditions on the underlying metrics. We also exploit the case when non self-adjoint linear operators are triangular by blocks in the primal-dual product space for solving primal-dual composite monotone inclusions, obtaining Gauss-Seidel type algorithms which generalize several primal-dual methods available in the literature. Finally we explore {applications to the obstacle problem, Empirical Risk Minimization, distributed optimization and nonlinear programming and we illustrate the performance of the method via some numerical simulations.
Cited by 0Open access
佳璇 魏, Xiaoting Xu, Peng Luo et al. · SSRN Electronic Journal · 2025
Cited by 0Open access
Hossein Aghagolzadeh, Mehdi Ezoji · Neural Networks · 2025
Cited by 0
Riccardo Scodellaro, Ajinkya Kulkarni, Frauke Alves et al. · Scientific Reports · 2025
Recent successes in image analysis with deep neural networks are achieved almost exclusively with Convolutional Neural Networks (CNNs), typically trained using the backpropagation (BP) algorithm. In a 2022 preprint, Geoffrey Hinton proposed the Forward-Forward (FF) algorithm as a biologically inspired alternative, where positive and negative examples are jointly presented to the network and training is guided by a locally defined goodness function. Here, we extend the FF paradigm to CNNs. We introduce two spatially extended labeling strategies, based on Fourier patterns and morphological transformations, that enable convolutional layers to access label information across all spatial positions. On CIFAR10, we show that deeper FF-trained CNNs can be optimized successfully and that morphology-based labels prevent shortcut solutions on dataset with more complex and fine features. On CIFAR100, carefully designed label sets scale effectively to 100 classes. Class Activation Maps reveal that FF-trained CNNs learn meaningful and complementary features across layers. Together, these results demonstrate that FF training is feasible beyond fully connected networks, provide new insights into its learning dynamics and stability, and highlight its potential for neuromorphic computing and biologically inspired learning.
Cited by 4Open access
PARDI, MATTEO · Electronic Theses and Dissertations Repository (University of Pisa) · 2023
In the last decade, deep learning has achieved remarkable success in a wide range of learning tasks, with increasingly larger datasets being used to train increasingly larger deep neural networks via end-to-end (E2E) backpropagation. However, training deep neural networks via E2E backpropagation presents challenges, such as the exploding/vanishing gradient problem, that can lead to slow convergence or even complete training failures. Moreover, deep training demands high amounts of computational resources, which can be limiting in resource-constrained scenarios. These factors prompted the research for alternative methods. An alternative to E2E training is represented by greedy layer-wise (GLW) approaches. In GLW methods, the network is constructed by training one layer at a time, adding the next layer only after the previous one has been trained and frozen, avoiding deep backpropagation. However, this approach may limit the effectiveness of hidden representations since the hidden layers may not be optimally trained to effectively transfer information to subsequent layers. The Forward-Forward Algorithm (FFA), introduced by Hinton in 2022, is a recent GLW approach that has shown promise in early experiments. In this thesis, we propose a novel approach to enhance the training of FFA layers by introducing an entropy-based regularization term in the loss function, aiming to maximize the information content within hidden layer representations. With our method, each layer is trained not only to solve the prediction task at hand, but also to transfer a high amount of useful information to subsequent layers, discouraging correlations among units in favor of richer hidden representations. The research consists of three main parts: (i) We examine the challenge of estimating entropy from a dataset, in order to identify a method for defining the entropy regularization term in the loss function. Specifically, we evaluate selected algorithms by conducting experiments on several synthetic datasets to understand their accuracy and computational cost. (ii) We conduct experiments on vectorial datasets to assess the effectiveness of entropy regularization on the FFA, exploring different definitions of the entropy regularization term. (iii) We investigate the performance of the FFA approach both with and without entropy regularization on more complex data structures, focusing on graph data. In this context, the depth of a model plays a more crucial role compared to the vectorial case. The layers construct node embeddings by performing neighborhood-based processing. As a result, the depth of a layer is associated with the spatial scale of graph analysis, with early layers focusing on small-scale analysis and deeper layers on larger-scale one. We evaluate the performance of two models: Graph Forward-Foward (GFF), introduced by Paliotta et al. in 2023, and our own novel model, referred to as Forward-Forward for Graphs (FF4G). Experimental results show that entropy regularization consistently improves the accuracy of FFA and GFF. Moreover, FF4G surpasses the accuracy of several E2E trained models on the considered benchmarks.
Cited by 0
Hossein Aghagolzadeh, Mehdi Ezoji · ArXiv.org · 2025
Although backpropagation is widely accepted as a training algorithm for artificial neural networks, researchers are always looking for inspiration from the brain to find ways with potentially better performance. Forward-Forward is a novel training algorithm that is more similar to what occurs in the brain, although there is a significant performance gap compared to backpropagation. In the Forward-Forward algorithm, the loss functions are placed after each layer, and the updating of a layer is done using two local forward passes and one local backward pass. Forward-Forward is in its early stages and has been designed and evaluated on simple multi-layer perceptron networks to solve image classification tasks. In this work, we have extended the use of this algorithm to a more complex and modern network, namely the Vision Transformer. Inspired by insights from contrastive learning, we have attempted to revise this algorithm, leading to the introduction of Contrastive Forward-Forward. Experimental results show that our proposed algorithm performs significantly better than the baseline Forward-Forward leading to an increase of up to 10% in accuracy and accelerating the convergence speed by 5 to 20 times. Furthermore, if we take Cross Entropy as the baseline loss function in backpropagation, it will be demonstrated that the proposed modifications to the baseline Forward-Forward reduce its performance gap compared to backpropagation on Vision Transformer, and even outperforms it in certain conditions, such as inaccurate supervision.
Cited by 0Open access
Riccardo Scodellaro, Ajinkya Kulkarni, Frauke Alves et al. · arXiv (Cornell University) · 2023
Recent successes in image analysis with deep neural networks are achieved almost exclusively with Convolutional Neural Networks (CNNs), typically trained using the backpropagation (BP) algorithm. In a 2022 preprint, Geoffrey Hinton proposed the Forward-Forward (FF) algorithm as a biologically inspired alternative, where positive and negative examples are jointly presented to the network and training is guided by a locally defined goodness function. Here, we extend the FF paradigm to CNNs. We introduce two spatially extended labeling strategies, based on Fourier patterns and morphological transformations, that enable convolutional layers to access label information across all spatial positions. On CIFAR10, we show that deeper FF-trained CNNs can be optimized successfully and that morphology-based labels prevent shortcut solutions on dataset with more complex and fine features. On CIFAR100, carefully designed label sets scale effectively to 100 classes. Class Activation Maps reveal that FF-trained CNNs learn meaningful and complementary features across layers. Together, these results demonstrate that FF training is feasible beyond fully connected networks, provide new insights into its learning dynamics and stability, and highlight its potential for neuromorphic computing and biologically inspired learning.
Cited by 3Open access
Matteo Pardi, Domenico Tortorella, Alessio Micheli · 2023
The forward-forward algorithm (FFA) is a recently proposed alternative to end-to-end backpropagation in deep neural networks.FFA builds networks greedily layer by layer, thus being of particular interest in applications where memory and computational constraints are important.In order to boost layers' ability to transfer useful information to subsequent layers, in this paper we propose a novel regularization term for the layerwise loss function that is based on Renyi's quadratic entropy.Preliminary experiments show accuracy is generally significantly improved across all network architectures.In particular, smaller architectures become more effective in addressing our classification tasks compared to the original FFA.
Cited by 1Open access
Fernando Roldán · Journal of Optimization Theory and Applications · 2024
Cited by 1
Ghader, Mohammadnavid, Kheradpisheh, Saeed Reza, Farahani, Bahar et al. · ArXiv.org · 2025
Spiking Neural Networks (SNNs) offer a biologically inspired computational paradigm that emulates neuronal activity through discrete spike-based processing. Despite their advantages, training SNNs with traditional backpropagation (BP) remains challenging due to computational inefficiencies and a lack of biological plausibility. This study explores the Forward-Forward (FF) algorithm as an alternative learning framework for SNNs. Unlike backpropagation, which relies on forward and backward passes, the FF algorithm employs two forward passes, enabling layer-wise localized learning, enhanced computational efficiency, and improved compatibility with neuromorphic hardware. We introduce an FF-based SNN training framework and evaluate its performance across both non-spiking (MNIST, Fashion-MNIST, Kuzushiji-MNIST) and spiking (Neuro-MNIST, SHD) datasets. Experimental results demonstrate that our model surpasses existing FF-based SNNs on evaluated static datasets with a much lighter architecture while achieving accuracy comparable to state-of-the-art backpropagation-trained SNNs. On more complex spiking tasks such as SHD, our approach outperforms other SNN models and remains competitive with leading backpropagation-trained SNNs. These findings highlight the FF algorithm's potential to advance SNN training methodologies by addressing some key limitations of backpropagation.
Cited by 0Open access
Suraj R Gautham, Swapnil Nair, Suresh Jamadagni et al. · 2024
This research explores the Forward-Forward Algorithm, proposed as an alternative to Backpropagation, in the context of Convolutional and Recurrent Neural Networks for multiclass classification. The authors' investigation aims to evaluate the algorithm's efficiency, advantages, and disadvantages for real-world applications. Results offer insights into the untapped potential and nuances of this approach, guiding researchers in optimizing neural network performance and potentially advancing a more human-like approach in Deep Learning.
Cited by 2
Taewook Hwang, Hyein Seo, Sangkeun Jung · Journal of KIISE · 2026
Forward-Forward 알고리즘은 역전파를 대체할 새로운 방법론으로 인간의 뇌의 동작 방식과 유사하게 각각의 레이어마다 독립적으로 학습한다. 그러나 이러한 학습 방법은 레이어 간 정보 전달 능력에 의문점이 존재한다. 본 연구에서는 인간의 기억능력을 모방하여 각 레이어에 독립된 메모리 공간을 할당하여 레이어 간 정보전달에 활용할 수 있는 방법을 제안한다. 각 레이어의 라벨별 출력값을 평균내어 메모리에 저장하고, 저장된 값과 현재 레이어 출력값을 각도기반 손실함수로 계산하여 각각의 라벨마다 고유한 각도에 분포되게 한다. 또한, 이전 레이어의 메모리 값과 현재 레이어의 출력값을 각도기반 손실함수로 계산하여 이전 레이어와 유사한 각도를 갖도록 학습하였다. 실험 결과, 데이터가 극히 적은 초기 단계에서는 한계가 있었으나, 데이터가 일정 수준(100-shot) 이상 확보되었을 때 제안하는 방법은 MNIST 데이터셋에서 90.71%의 정확도를 달성하여, 기존 Forward-Forward 알고리즘(89.63%) 대비 향상된 성능을 보임을 확인하였다.
Cited by 0