Zhiqiang Liao, Sheng Dai, Eunji Lim et al. · arXiv (Cornell University) · 2024
Convex regression is a method for estimating the convex function from a data set. This method has played an important role in operations research, economics, machine learning, and many other areas. However, it has been empirically observed that convex regression produces inconsistent estimates of convex functions and extremely large subgradients near the boundary as the sample size increases. In this paper, we provide theoretical evidence of this overfitting behavior. To eliminate this behavior, we propose two new estimators by placing a bound on the subgradients of the convex function. We further show that our proposed estimators can reduce overfitting by proving that they converge to the underlying true convex function and that their subgradients converge to the gradient of the underlying function, both uniformly over the domain with probability one as the sample size is increasing to infinity. An application to Finnish electricity distribution firms confirms the superior performance of the proposed methods in predictive power over the existing methods.
Cited by 1Open access
Asma’ Amro, Mousa Al-Akhras, Khalil El Hindi et al. · Journal of Intelligent Systems · 2021
Abstract Decision trees learning is one of the most practical classification methods in machine learning, which is used for approximating discrete-valued target functions. However, they may overfit the training data, which limits their ability to generalize to unseen instances. In this study, we investigated the use of instance reduction techniques to smooth the decision boundaries before training the decision trees. Noise filters such as ENN, RENN, and ALLKNN remove noisy instances while DROP3 and DROP5 may remove genuine instances. Extensive empirical experiments were conducted on 13 benchmark datasets from UCI machine learning repository with and without intentionally introduced noise. Empirical results show that eliminating border instances improves the classification accuracy of decision trees and reduces the tree size, which reduces the training and classification times. In datasets without intentionally added noise, applying noise filters without the use of the built-in Reduced Error Pruning gave the best classification accuracy. ENN, RENN, and ALLKNN outperformed decision trees learning without pruning in 9, 9, and 8 out of 13 datasets, respectively. The datasets reduced using ENN and RENN without built-in pruning were more effective when noise was intentionally introduced in different ratios.
Cited by 55Open access
Xiaoyue Feng, Yanchun Liang, Xiaohu Shi et al. · Entropy · 2017
Overfitting is an important problem in machine learning. Several algorithms, such as the extreme learning machine (ELM), suffer from this issue when facing high-dimensional sparse data, e.g., in text classification. One common issue is that the extent of overfitting is not well quantified. In this paper, we propose a quantitative measure of overfitting referred to as the rate of overfitting (RO) and a novel model, named AdaBELM, to reduce the overfitting. With RO, the overfitting problem can be quantitatively measured and identified. The newly proposed model can achieve high performance on multi-class text classification. To evaluate the generalizability of the new model, we designed experiments based on three datasets, i.e., the 20 Newsgroups, Reuters-21578, and BioMed corpora, which represent balanced, unbalanced, and real application data, respectively. Experiment results demonstrate that AdaBELM can reduce overfitting and outperform classical ELM, decision tree, random forests, and AdaBoost on all three text-classification datasets; for example, it can achieve 62.2% higher accuracy than ELM. Therefore, the proposed model has a good generalizability.
Cited by 30Open access
Asma' Amro, Mousa Al-Akhras, Khalil El Hindi et al. · Greater South Information System · 2021
Abstract Decision trees learning is one of the most practical classification methods in machine learning, which is used for approximating discrete-valued target functions. However, they may overfit the training data, which limits their ability to generalize to unseen instances. In this study, we investigated the use of instance reduction techniques to smooth the decision boundaries before training the decision trees. Noise filters such as ENN, RENN, and ALLKNN remove noisy instances while DROP3 and DROP5 may remove genuine instances. Extensive empirical experiments were conducted on 13 benchmark datasets from UCI machine learning repository with and without intentionally introduced noise. Empirical results show that eliminating border instances improves the classification accuracy of decision trees and reduces the tree size, which reduces the training and classification times. In datasets without intentionally added noise, applying noise filters without the use of the built-in Reduced Error Pruning gave the best classification accuracy. ENN, RENN, and ALLKNN outperformed decision trees learning without pruning in 9, 9, and 8 out of 13 datasets, respectively. The datasets reduced using ENN and RENN without built-in pruning were more effective when noise was intentionally introduced in different ratios.
Cited by 0Open access
Liao, Zhiqiang · Aaltodoc (Aalto University) · 2025
Overfitting is a common pitfall in many statistical models. In the context of regression analysis, it refers to a phenomenon where the statistical model fits not only the underlying relationship between variables but also the noise in the data, often caused by measurement errors. An overfitted model is usually characterized by excessive complexity and poor generalization, thus failing to make accurate predictions on unseen or new data. Convex regression, a classical nonparametric regression method, aims to estimate a convex function from a data set. Given its high flexibility, especially near the boundary, convex regression is more prone to suffer from overfitting compared to simpler regression methods like linear regression. Therefore, this thesis focuses on developing new convex nonparametric least squares (CNLS) estimators for reducing overfitting and exploring their applications in the regulation of electricity distribution networks in Nordic countries. Essay I proposes a machine learning-based approach combining support vector regression and convex regression. This approach is motivated by the regulation of Finnish electricity distribution networks, in which the Energy Authority collects historical data on labor, capital, and cost over the past 10 years. The regulatory policy is related to the predictive power of CNLS, to provide the right incentive policies for regulated electricity distribution companies. Given that the CNLS estimator suffers from the overfitting problem, the proposed convex support vector regression offers a useful tool to combine machine learning techniques and flexible nonparametric modelling, thereby allowing one to alleviate overfitting and improve the accuracy of out-of-sample predictions. Essay II focuses on investigating the overfitting behavior of the CNLS estimator at the boundary from a statistical perspective. Drawing from numerical evidence of overfitting in the literature, this essay seeks to provide theoretical evidence about how the CNLS estimator performs near the boundary of the domain. In this regard, Essay II bridges the gap between empirical observations and theoretical properties in convex regression and, more importantly, sheds light on remedies for reducing overfitting. Based on the theoretical results, this study presents two techniques that help tackle the overfitting problem. Statistical theorems and numerical experiments confirm that these methods outperform the original CNLS estimator. Essay III addresses the problem of sparse regression when high-dimensional data are available. Selecting a sparse model is important for model interpretation and improving predictive power, especially when the number of variables is large. Essay III proposes the structured Lasso method by combining ℓ1-norm and ℓ∞-norm. This approach is also extended to a relaxed version to create a trade-off between variable selection and shrinkage. This variable selection technique, along with its extensions, demonstrates superior performance in numerical experiments compared to other methods regarding sparsity and accuracy.
Cited by 0Open access
Lin Yang, Xinyu Ma, Xu Chu et al. · Knowledge-Based Systems · 2025
Cited by 9
Sara Silva, Inês Magessi, Leonardo Vanneschi · IEEE Transactions on Evolutionary Computation · 2025
Like other machine learning methods, Genetic Programming (GP) frequently faces the issue of overfitting when applied to supervised learning tasks. Traditional regularization techniques, though well-studied, are challenging to apply to GP due to the free-form nature of the evolved models. This work proposes a novel approach that prevents overfitting while inherently improving the interpretability of GP models. It involves a dual optimization process that minimizes loss while penalizing functional complexity using multi-objective selection mechanisms. The improved complexity measure used in this study approximates the mathematical curvature of a function in linear time. While loss minimization is common in GP, penalizing functional complexity is an additional step aimed at evolving robust and smooth functions, less prone to overfitting and potentially more interpretable. Experimental results demonstrate the effectiveness of the two variants of our method, benchmarked against standard GP and two of the seemingly best overfitting-reduction methods found in the literature. By focusing on both loss and complexity, our approach achieves state-of-the-art generalization on difficult problems and a strong feature selection that improves interpretability, making it a unified improvement of GP.
Cited by 0Open access
Alex Barros, Denis Rosário, Eduardo Cerqueira et al. · 2021
Federated learning (FL) is a framework to train machine learning models using decentralized data, especially unbalanced and non-iid. Adaptive methods can be used to accelerate convergence, reducing the number of rounds of local computation and communication to a centralized server. This paper proposes an adaptive controller to adapt the number of epochs needed that employs Poisson distribution to avoid overfitting of the aggregated model, promoting fast convergence. Our results indicate that increasing the local update of the model should be avoided, but yet some complementary mechanism is needed to model performance. We evaluate the impact of an increasing number of epochs of FedAVG and FedADAM.
Cited by 7Open access
Abdul Kawsar Tushar, Akm Ashiquzzaman, Md. Rashedul Islam · 2017
Hand signs and signals are the staple form of expression for the hearing and speech impaired people. Human Computer Interaction technology enable people to interact with computer machine using hand gestures. Common sign languages use separate hand signals to communicate different decimals. Recent developments in Deep Convolutional Neural Networks (DCNN) have opened the door to recognize and classify this visual form of gestures more accurately. In this paper, a layer-wise optimized neural network architecture is proposed where batch normalization contributes to faster convergence of training, and introduction of dropout technique mitigates data overfitting. Batch normalization forces each training batch toward zero mean and unit variance, leading to improved flow of gradients through the model and convergence in shorter time. Dropout forces neurons of neural network to regularize, resulting in reduced overfitting. A constructed numerical hand gesture data set is used for validating the claims based on American Sign Language system. The proposed model is shown to surpass other methods in classifying these numerical hand signs successfully.
Cited by 19
Fatemeh Ahmadi Abkenari, Amin Milani Fard, Sara Khanchi · Journal of Cybersecurity and Privacy · 2023
An intrusion detection system (IDS), whether as a device or software-based agent, plays a significant role in networks and systems security by continuously monitoring traffic behaviour to detect malicious activities. The literature includes IDSs that leverage models trained to detect known attack behaviours. However, such models suffer from low accuracy or high overfitting. This work aims to enhance the performance of the IDS by making a model based on the observed traffic via applying different single and ensemble classifiers and lowering the classifier’s overfitting on a reduced set of features. We implement various feature reduction techniques, including Linear Regression, LASSO, Random Forest, Boruta, and autoencoders on the CSE-CIC-IDS2018 dataset to provide a training set for classifiers, including Decision Tree, Naïve Bayes, neural networks, Random Forest, and XGBoost. Our experiments show that the Decision Tree classifier on autoencoders-based reduced sets of features yields the lowest overfitting among other combinations.
Cited by 6Open access
Erapaneni Gayatri, S. L. Aarthy · Journal of X-Ray Science and Technology · 2023
Finally, augmented data with focal loss is given a good classification performance with 98.85% accuracy, 95.52% precision, and 95.93% recall. Matthews Correlation coefficient (MCC) is the best metric to evaluate the quality of multi-class images. It has given outstanding performance by using augmented data and focal loss.
Cited by 12Open access
Hendriansyah Santosa, Kusrini Kusrini · Dinasti International Journal of Education Management And Social Science · 2025
This study aims to predict book sales using a Bidirectional Long Short-Term Memory (LSTM) model combined with clipping and early stopping techniques to handle outliers and reduce overfitting. The dataset consists of daily book sales records with temporal and categorical variables. The preprocessing process includes feature engineering, logarithmic transformation, standardization, and clipping on the target variable. The dataset is formed in time-series format with a sliding window approach. The model is evaluated using MSE, MAE, RMSE, and R². The results show that the integration of clipping and early stopping provides optimal prediction performance, with an R² value of 0.87 and an RMSE of 0.44. These findings demonstrate the effectiveness of the Bidirectional LSTM approach in forecasting complex and dynamic book sales. This paper is part of the author’s undergraduate thesis at Universitas Amikom Yogyakarta.
Cited by 0Open access
Karima EL Hachimi, Ghizlane Orhanou, Lahoucine Ballihi · Expert Systems with Applications · 2025
Cited by 1
Poningsih Poningsih, Agus Perdana Windarto, Putrama Alkhairi · Jurnal Ilmiah Teknik Elektro Komputer dan Informatika · 2024
Overfitting presents a significant challenge in developing text classification models using neural networks, as it occurs when models learn too much from the training data, including noise and specific details, resulting in poor performance on new, unseen data. This study addresses this issue by exploring overfitting reduction techniques to enhance the generalization of neural networks in text classification tasks using the IMDB movie review dataset from Kaggle. The research aims to provide insights into effective methods to reduce overfitting, thereby improving the performance and reliability of text classification models in practical applications. The methodology involves developing two LSTM neural network models: a standard model without overfitting reduction techniques and an enhanced model incorporating dropout and early stopping. The IMDB dataset is preprocessed to convert reviews into sequences suitable for input into the LSTM models. Both models are trained, and their performances are compared using various metrics. The model without overfitting reduction techniques shows a test loss of 0.4724 and a test accuracy of 86.81%. Its precision, recall, and F1-score for classifying negative reviews are 0.91, 0.82, and 0.86, respectively, and for positive reviews are 0.84, 0.92, and 0.87. The enhanced model, incorporating dropout and early stopping, demonstrates improved performance with a lower test loss of 0.2807 and a higher test accuracy of 88.61%. For negative reviews, its precision, recall, and F1-score are 0.92, 0.84, and 0.88, and for positive reviews are 0.86, 0.93, and 0.89. Overall, the enhanced model achieves better metrics, with an accuracy of 89%, and macro and weighted averages for precision, recall, and F1-score all at 0.89. The applying overfitting reduction techniques significantly enhances the model's performance.
Cited by 0Open access
Akash Das Sarma · Advances in computational intelligence and robotics book series · 2025
This chapter provides a detailed understanding of various types of regularization techniques in generative models, and explain how these regularization techniques can help for reduction of overfitting and improves model generalization. The chapter begins with a brief introduction about what is an AI model, its importance and their uses. Further, explain what regularization is and why is it significant. Briefly outline the terms underfitting, overfitting, just right and mention techniques to detect overfitting. Also, provides a list of different types of AI models. Moreover, explain various types of regularizations. Besides, outline the common goals of regularization. Further, present various key parameters that affect overfitting in generative models and explain briefly. Finally, present various regularization techniques that prevent overfitting in different modalities such as text, image and multi-models.
Cited by 0
Wenwen Qiang, Zeen Song, Ziyin Gu et al. · arXiv (Cornell University) · 2024
Self-supervised learning (SSL) methods learn from unlabeled data and achieve high generalization performance on downstream tasks. However, they may also suffer from overfitting to their training data and lose the ability to adapt to new tasks. To investigate this phenomenon, we conduct experiments on various SSL methods and datasets and make two observations: (1) Overfitting occurs abruptly in later layers and epochs, while generalizing features are learned in early layers for all epochs; (2) Coding rate reduction can be used as an indicator to measure the degree of overfitting in SSL models. Based on these observations, we propose Undoing Memorization Mechanism (UMM), a plug-and-play method that mitigates overfitting of the pre-trained feature extractor by aligning the feature distributions of the early and the last layers to maximize the coding rate reduction of the last layer output. The learning process of UMM is a bi-level optimization process. We provide a causal analysis of UMM to explain how UMM can help the pre-trained feature extractor overcome overfitting and recover generalization. We also demonstrate that UMM significantly improves the generalization performance of SSL methods on various downstream tasks.
Cited by 0Open access
Yiwen Guo, Xiaoqing Ding, Changsong Liu et al. · IEEE Transactions on Image Processing · 2016
Canonical correlation analysis (CCA) is an effective way to find two appropriate subspaces in which Pearson's correlation coefficients are maximized between projected random vectors. Due to its well-established theoretical support and relatively efficient computation, CCA is widely used as a joint dimension reduction tool and has been successfully applied to many image processing and computer vision tasks. However, as reported, the traditional CCA suffers from overfitting in many practical cases. In this paper, we propose sufficient CCA (S-CCA) to relieve CCA's overfitting problem, which is inspired by the theory of sufficient dimension reduction. The effectiveness of S-CCA is verified both theoretically and experimentally. Experimental results also demonstrate that our S-CCA outperforms some of CCA's popular extensions during the prediction phase, especially when severe overfitting occurs.
Cited by 11
С. В. Козырев, I. Lopatin, Alexander Pechen · Entropy · 2024
While there are many works on the applications of machine learning, not so many of them are trying to understand the theoretical justifications to explain their efficiency. In this work, overfitting control (or generalization property) in machine learning is explained using analogies from physics and biology. For stochastic gradient Langevin dynamics, we show that the Eyring formula of kinetic theory allows to control overfitting in the algorithmic stability approach-when wide minima of the risk function with low free energy correspond to low overfitting. For the generative adversarial network (GAN) model, we establish an analogy between GAN and the predator-prey model in biology. An application of this analogy allows us to explain the selection of wide likelihood maxima and ab overfitting reduction for GANs.
Cited by 1Open access
Kozyrev, Sergei V., Lopatin, Ilya A, Pechen, Alexander N · arXiv (Cornell University) · 2024
While there are many works on the applications of machine learning, not so many of them are trying to understand the theoretical justifications to explain their efficiency. In this work, overfitting control (or generalization property) in machine learning is explained using analogies from physics and biology. For stochastic gradient Langevin dynamics, we show that the Eyring formula of kinetic theory allows to control overfitting in the algorithmic stability approach - when wide minima of the risk function with low free energy correspond to low overfitting. For the generative adversarial network (GAN) model, we establish an analogy between GAN and the predator-prey model in biology. An application of this analogy allows us to explain the selection of wide likelihood maxima and overfitting reduction for GANs.
Cited by 0Open access
Trần Kim Toại, Võ Thị Hạnh, Võ Minh Huân · 2023
This paper proposes the hybrid random forest and long short-term memory (LSTM) to mitigate overfitting issue in time series data in stock market. There are many techniques that reduce the overfitting such as data augmentation, regularization, feature selection, dimension reduction, and so on. We propose the model based on feature selection to reduce the model complexity. First, the model selects the stock data features by random forest model. As the result, the selected features are inputted to the LSTM to predict the stock price. By doing so, the proposed model can improve model accuracy in both training and test dataset and generalize well unseen data to mitigate overfitting. The hybrid random forest and LSTM is compared with hybrid ridge and LSTM, and single LSTM model in ability to mitigate overfitting. The MAE, RSME and R2 are used as performance evaluation metrics. We also conduct the study on various stock datasets to evaluate the performance of overcoming the overfitting problems.
Cited by 3