Crouther, Paul · Papyrus : Institutional Repository (Université de Montréal) · 2023
The primary goal of this work is to propose a methodology for discovering hyperparameters. \nHyperparameters aid systems in convergence when well-tuned and handcrafted. However, \nto this end, poorly chosen hyperparameters leave practitioners in limbo, between concerns \nwith implementation or improper choice in hyperparameter and system configuration. We \nspecifically analyze the choice of learning rate in stochastic gradient descent (SGD), a popular \nalgorithm. As a secondary goal, we attempt the discovery of fixed points using smoothing of \nthe loss landscape by exploiting assumptions about its distribution to improve the update \nrule in SGD. Smoothing of the loss landscape has been shown to make convergence possible in \nlarge-scale systems and difficult black-box optimization problems. However, we use stochastic \nvalue gradients (SVG) to smooth the loss landscape by learning a surrogate model and then \nbackpropagate through this model to discover fixed points on the real task SGD is trying to \nsolve. Additionally, we construct a gym environment for testing model-free algorithms, such \nas Proximal Policy Optimization (PPO) as a hyperparameter optimizer for SGD. For tasks, \nwe focus on a toy problem and analyze the convergence of SGD on MNIST using model-free \nand model-based reinforcement learning methods for control. The model is learned from \nthe parameters of the true optimizer and used specifically for learning rates rather than for \nprediction. In experiments, we perform in an online and offline setting. In the online setting, \nwe learn a surrogate model alongside the true optimizer, where hyperparameters are tuned \nin real-time for the true optimizer. In the offline setting, we show that there is more potential \nin the model-based learning methodology than in the model-free configuration due to this \nsurrogate model that smooths out the loss landscape and makes for more helpful gradients \nduring backpropagation.
Cited by 0Open access
Kurt Price · Western CEDAR (Western Washington University) · 2018
Machine learning models can learn to recognize subtle patterns in complex data, making them useful in a wide variety of regression and classification tasks. The process of training a machine learning model almost always involves setting hyperparameters. Hyperparameters can be thought of as knobs that affect the learned model; some hyperparameters affect the architecture of the model itself, while others affect the optimization algorithm used the train it. To find (near) optimal hyperparameter configurations, researchers typically often hand-tune the models through tedious trial-and-error or simple heuristics. In this work, we are developing a web service that offers researchers easy access to powerful hyperparameter optimization routines, in order to eliminate the guesswork currently involved with hyperparameter tuning. Clients of our service will first provide possible ranges for the hyperparameters they wish to tune, creating a session. The service can then be queried to provide hyperparameter configurations for the client to try and, afterwards, report the results of this configuration. These results will then inform subsequent hyperparameter suggestions based on the chosen optimization module. Although some hyperparameter optimization services and tools currently exist, they are either prohibitively expensive or limited in functionality and cumbersome to use, making them inappropriate for many researchers and small companies. With our free service and easy-to-use interface, we aim to provide a valuable service to the long tail of machine learning model users.
Cited by 0
Jonsson, Leif, Magnusson, Måns · Zenodo (CERN European Organization for Nuclear Research) · 2018
Implementations of various fast parallelized samplers for LDA, including Partially Collapsed LDA, Light LDA, Partially Collapsed Light LDA and a very efficient Polya-Urn LDA
Cited by 0Open access
Martin Wistuba, Nicolas Schilling, Lars Schmidt-Thieme · 2015
Hyperparameter optimization is often done manually or by using a grid search. However, recent research has shown that automatic optimization techniques are able to accelerate this optimization process and find hyperparameter configurations that lead to better models. Currently, transferring knowledge from previous experiments to a new experiment is of particular interest because it has been shown that it allows to further improve the hyperparameter optimization. We propose to transfer knowledge by means of an initialization strategy for hyperparameter optimization. In contrast to the current state of the art initialization strategies, our strategy is neither limited to hyperparameter configurations that have been evaluated on previous experiments nor does it need meta-features. The initial hyperparameter configurations are derived by optimizing for a meta-loss formally defined in this paper. This loss depends on the hyperparameter response function of the data sets that were investigated in past experiments. Since this function is unknown and only few observations are given, the meta-loss is not differentiable. We propose to approximate the response function by a differentiable plug-in estimator. Then, we are able to learn the initial hyperparameter configuration sequence by applying gradient-based optimization techniques. Extensive experiments are conducted on two meta-data sets. Our initialization strategy is compared to the state of the art for initialization strategies and further methods that are able to transfer knowledge between data sets. We give empirical evidence that our work provides an improvement over the state of the art.
Cited by 85
Muntequa Imtiaz Siraji (17458184), Ahnaf Akif Rahman (17458187), Mirza Muntasir Nishat (17458190) et al. · Figshare · 2023
<div><p>Depression is a psychological state of mind that often influences a person in an unfavorable manner. While it can occur in people of all ages, students are especially vulnerable to it throughout their academic careers. Beginning in 2020, the COVID-19 epidemic caused major problems in people’s lives by driving them into quarantine and forcing them to be connected continually with mobile devices, such that mobile connectivity became the new norm during the pandemic and beyond. This situation is further accelerated for students as universities move towards a blended learning mode. In these circumstances, monitoring student mental health in terms of mobile and Internet connectivity is crucial for their wellbeing. This study focuses on students attending an International University of Bangladesh to investigate their mental health due to their continual use of mobile devices (e.g., smartphones, tablets, laptops etc.). A cross-sectional survey method was employed to collect data from 444 participants. Following the exploratory data analysis, eight machine learning (ML) algorithms were used to develop an automated normal-to-extreme severe depression identification and classification system. When the automated detection was incorporated with feature selection such as Chi-square test and Recursive Feature Elimination (RFE), about 3 to 5% increase in accuracy was observed by the method. Similarly, a 5 to 15% increase in accuracy has been observed when a feature extraction method such as Principal Component Analysis (PCA) was performed. Also, the SparsePCA feature extraction technique in combination with the CatBoost classifier showed the best results in terms of accuracy, F1-score, and ROC-AUC. The data analysis revealed no sign of depression in about 44% of the total participants. About 25% of students showed mild-to-moderate and 31% of students showed severe-to-extreme signs of depression. The results suggest that ML models, incorporating a proper feature engineering method can serve adequately in multi-stage depression detection among the students. This model might be utilized in other disciplines for detecting early signs of depression among people.</p></div>
Cited by 0Open access
Aaron Klein · FreiDok plus (Universitätsbibliothek Freiburg) · 2020
Automated machine learning emerged as a new research field inside of machine learning that tries to progressively automate different steps of common machine learning pipelines which are traditionally executed by humans. One of its core tasks is the automated search for the right hyperparameters of a given machine learning algorithm which in practice is often essential to achieve good performance. Compare to other optimization problems, hyperparameter optimization is usually particularly expensive, since in each iteration, it requires to train and validate the underlying algorithm. One of the most successful approaches for hyperparameter optimization is Bayesian optimization. At its core, Bayesian optimization fits a probabilistic model of the objective function, which together with an additional acquisition function is used to guide the search towards the global optimum. In this thesis we present several extensions to standard Bayesian optimization to improve its performance for hyperparameter optimization problems. First, we introduce a new probabilistic model based on Bayesian neural networks, that allows to model the performance of hyperparameter configurations across different tasks and thereby scales much better with the number of data points and dimensions than Gaussian processes which are traditionally used inside Bayesian optimization. In hyperparameter optimization, often approximations, so-called fidelities, of the objective function are available which are much cheaper to evaluate. We present two new Bayesian optimization methods that can leverage such fidelities, such as learning curves or dataset subsets, to improve the overall search process in terms of wall-clock time by orders of magnitude. Furthermore, based on our proposed Bayesian neural network model, we present a new neural network architecture which models the learning curve of iterative machine learning methods, such as neural networks. Finally, due to the high computational cost of hyperparameter optimization, thorough benchmarking and evaluation of new developed methods is often prohibitively expensive. We show that one can approximate continuous and discrete benchmarks by surrogate benchmarks that capture the characteristics of the original benchmark but take only milliseconds to evaluate. This allows us to performa rigorous analysis and comparison of various different hyperparameter optimization methods from the literature.
Cited by 0Open access
André Miguel Ferreira da Cruz · Open Repository of the University of Porto (University of Porto) · 2020
In recent years, increased usage of machine learning algorithms has been accompanied by several reports of machine bias in areas from recidivism assessment, to job-applicant screening tools, and estimating mortgage default risk. Additionally, recent advances in machine learning have prominently featured so-called "black-box" models (e.g. neural networks), in which we can see its inputs and outputs, but with limited capability for inspecting its decision-making process. As a result, it is increasingly imperative to monitor and control fairness of developed models for detecting discrimination against sub-groups of the population (e.g. based on race, gender, or age). State-of-the-art machine learning algorithms require the definition of a large number of hyperparameters to govern how they learn and generalize to unseen data. Current hyperparameter search algorithms aim to tune these knobs in order to optimize for a global performance metric (e.g. accuracy). At the same time, fairness metrics are equally impacted by varying hyperparameter values, but there is comparatively little research on optimizing for multiple objectives. Consequently, we aim to study how to achieve efficient hyperparameter optimization for multi-objective goals, and corresponding trade-offs. We develop a hyperparameter optimization framework that supports the definition of secondary objectives or constraints, and experiment with multiple fairness metrics (e.g. equality of opportunity). Furthermore, we explore a fraud detection case study, and assess the framework's effectiveness in this context.
Cited by 0Open access
ARTESANI, ALESSIA · Zenodo (CERN European Organization for Nuclear Research) · 2025
Bayesian Hyperparameter Optimization Code refers to the paper: Artesani A et al., (2025) Development and Evaluation of a Bayesian Optimization FDG Population-Based Input Function for Implementing Parametric Imaging in the Clinical Practice Abstract. Aim. Parametric imaging from dynamic positron emission tomography (PET) has gained interest for tumour diagnostics and treatment response evaluation. However, the lack of a standardized method for generating the input function ‒ reference curve for kinetic modelling ‒ has led to inconsistent descriptors, contributing to uncertainties in parametric imaging reliability. This study aims to address this challenge by proposing a hyperparametric optimization method for deriving FDG population-based input function (PBIF), independent of acquisition and injection protocols.Method. This study included ten patients undergoing FDG PET scans using a standard axial field of view scanner. Image-derived input functions (IDIF) were extracted from the descending aorta, normalized, and utilized as input for PBIF modelling. Bayesian hyperparameter optimization was employed to estimate global optima for ten parameters that describe the input function through independent runs of up to 600 iterations each. The injection profile was integrated as a double rectangular profile, representing both the tracer injection and the saline flush tracer residual. Results. The Bayesian optimization successfully modelled patient-specific IDIFs (R²=0.97). The algorithm estimated injection and flush durations in agreement with recorded values. Parameter distributions showed low variability, with median amplitude and time constant values varying by around 15%. The glucose-affine molecule dynamics were characterized by distinct time constants of 6 seconds, 4 minutes, and 70 minutes. Analytical and numerical comparisons of parametric imaging from IDIF and PBIF show that Patlak analysis is unaffected by the injection characteristics.Conclusion. The study highlights the benefits of Bayesian optimization for modelling PBIF without prior knowledge of injection characteristics. These findings support the existence of unified FDG PBIF, facilitating the utilization of parametric imaging across PET centres. Although the present study is based on a limited, single-centre cohort, this methodological development is intended as a foundational study to further multi-centre validation on larger population.
Cited by 0Open access
Daniel Schirmacher (19475634), Ümmünur Armagan (19475637), Yang Zhang (30734) et al. · Figshare · 2024
<p>aiSEGcell instances trained with different learning rates on the D1 training set were evaluated with respect to on the D1 test set (n = 3,153 images, N = 12 experiments). All other hyperparameters were identical between hyperparameter conditions. For each learning rate three random weight initializations were trained. The instance with the highest (cyan square) was used for all subsequent experiments.</p> <p>(TIF)</p>
Cited by 0Open access
Crouther, Paul · Papyrus : Institutional Repository (Université de Montréal) · 2023
The primary goal of this work is to propose a methodology for discovering hyperparameters. Hyperparameters aid systems in convergence when well-tuned and handcrafted. However, to this end, poorly chosen hyperparameters leave practitioners in limbo, between concerns with implementation or improper choice in hyperparameter and system configuration. We specifically analyze the choice of learning rate in stochastic gradient descent (SGD), a popular algorithm. As a secondary goal, we attempt the discovery of fixed points using smoothing of the loss landscape by exploiting assumptions about its distribution to improve the update rule in SGD. Smoothing of the loss landscape has been shown to make convergence possible in large-scale systems and difficult black-box optimization problems. However, we use stochastic value gradients (SVG) to smooth the loss landscape by learning a surrogate model and then backpropagate through this model to discover fixed points on the real task SGD is trying to solve. Additionally, we construct a gym environment for testing model-free algorithms, such as Proximal Policy Optimization (PPO) as a hyperparameter optimizer for SGD. For tasks, we focus on a toy problem and analyze the convergence of SGD on MNIST using model-free and model-based reinforcement learning methods for control. The model is learned from the parameters of the true optimizer and used specifically for learning rates rather than for prediction. In experiments, we perform in an online and offline setting. In the online setting, we learn a surrogate model alongside the true optimizer, where hyperparameters are tuned in real-time for the true optimizer. In the offline setting, we show that there is more potential in the model-based learning methodology than in the model-free configuration due to this surrogate model that smooths out the loss landscape and makes for more helpful gradients during backpropagation.
Cited by 0Open access
Yaoyang Yi (21627403) · Figshare · 2025
<div> In light of global climate change and the objective of carbon neutrality, the carbon market has become an important tool for the international community to combat climate change. Nonetheless, due to the complexity and non-linear nature of the carbon price, its accurate prediction has always been a research difficulty. This work presents a hybrid model incorporating comprehensive feature screening, optimized quadratic decomposition, and the Optuna-Attention-LSTM prediction method, aiming to improve the accuracy and stability of carbon price prediction. First, the improved complete ensemble empirical mode decomposition with adaptive noise (ICEEMDAN) is used to decompose the carbon price time series once, extract high-frequency and low-frequency components, and denoise the high-frequency components using stacked denoising autoencoder (SDAE). Then, the variational mode decomposition (VMD) method is subsequently employed to execute a secondary decomposition on the reconstructed signal, with the decomposition hyperparameters optimized via crested porcupine optimization (CPO). Subsequently, Boruta and least absolute shrinkage and selection operator (Lasso) regression are employed to identify significant external features; finally, a long short-term memory (LSTM) model integrated with an attention mechanism is utilized for prediction, and optuna is introduced to optimize the hyperparameters. This paper evaluates the performance of the proposed model using the carbon markets of Guangdong, Hubei, and Shanghai in China as examples. The experimental results indicate that compared with the traditional model, the proposed model achieves average reductions of 67.30%, 47.68%, 48.42%, and 48.79% in the mean squared error (MSE), root mean squared error (RMSE), mean absolute error (MAE), and mean absolute percentage error (MAPE), respectively, demonstrating higher prediction accuracy and robustness. Shapley additive explanations (SHAP) analysis results indicate that carbon prices in the Guangdong carbon market are dominated by macroeconomic and regional environmental factors, while those in the Hubei carbon market are mainly driven by changes in the energy market. The Shanghai carbon market, on the other hand, is more significantly influenced by global carbon market dynamics and international trade activities. The research not only verifies the efficacy of the decomposition ensemble prediction framework, but also provides a scientific basis for decision-making for carbon market participants and policymakers. </div>
Cited by 0Open access
Lennart Schneider, Bernd Bischl, Matthias Feurer · ArXiv.org · 2025
Hyperparameter optimization (HPO) aims to identify an optimal hyperparameter configuration (HPC) such that the resulting model generalizes well to unseen data. As the expected generalization error cannot be optimized directly, it is estimated with a resampling strategy, such as holdout or cross-validation. This approach implicitly assumes that minimizing the validation error leads to improved generalization. However, since validation error estimates are inherently stochastic and depend on the resampling strategy, a natural question arises: Can excessive optimization of the validation error lead to overfitting at the HPO level, akin to overfitting in model training based on empirical risk minimization? In this paper, we investigate this phenomenon, which we term overtuning, a form of overfitting specific to HPO. Despite its practical relevance, overtuning has received limited attention in the HPO and AutoML literature. We provide a formal definition of overtuning and distinguish it from related concepts such as meta-overfitting. We then conduct a large-scale reanalysis of HPO benchmark data to assess the prevalence and severity of overtuning. Our results show that overtuning is more common than previously assumed, typically mild but occasionally severe. In approximately 10% of cases, overtuning leads to the selection of a seemingly optimal HPC with worse generalization error than the default or first configuration tried. We further analyze how factors such as performance metric, resampling strategy, dataset size, learning algorithm, and HPO method affect overtuning and discuss mitigation strategies. Our results highlight the need to raise awareness of overtuning, particularly in the small-data regime, indicating that further mitigation strategies should be studied.
Cited by 0Open access
Jiayue Zhang (7832669), Xinxin Yi (3932963), Heng Wang (76752) · Figshare · 2025
Visualization of hyperparameter optimization results.
Cited by 0Open access
Doaa Sami Khafaga (21463870), El-Sayed M. El-kenawy (14581088), Faris H. Rizk (21755909) et al. · Figshare · 2025
Classification performance after hyperparameter optimization.
Cited by 0Open access
Thomas Bartz–Beielstein · 2023
Abstract This chapter reflects on advantages and sense of use of Hyperparameter Tuning (HPT) and its disadvantages. In particular it shows how important it is, to keep the human in the loop, even if HPT works perfectly. The chapter presents a collection of HPT studies. First, HPT applications in Machine Learning (ML) and Deep Learning (DL) are described. A special focus lies on automated ML, neural architecture search, and combined approaches. HPT software is presented. Finally, model based approaches, especially applications with Sequential Parameter Optimization Toolbox (SPOT) are discussed.
Cited by 5Open access
André F. Cruz, Pedro Saleiro, Catarina Belém et al. · arXiv (Cornell University) · 2021
Considerable research effort has been guided towards algorithmic fairness but real-world adoption of bias reduction techniques is still scarce. Existing methods are either metric- or model-specific, require access to sensitive attributes at inference time, or carry high development or deployment costs. This work explores the unfairness that emerges when optimizing ML models solely for predictive performance, and how to mitigate it with a simple and easily deployed intervention: fairness-aware hyperparameter optimization (HO). We propose and evaluate fairness-aware variants of three popular HO algorithms: Fair Random Search, Fair TPE, and Fairband. We validate our approach on a real-world bank account opening fraud case-study, as well as on three datasets from the fairness literature. Results show that, without extra training cost, it is feasible to find models with 111% mean fairness increase and just 6% decrease in performance when compared with fairness-blind HO.
Cited by 6Open access
Kirill Akhmetzyanov, A. A. Yuzhakov · arXiv (Cornell University) · 2020
We present a new method for searching optimal hyperparameters among several tasks and several criteria. Multi-Task Multi Criteria method (MTMC) provides several Pareto-optimal solutions, among which one solution is selected with given criteria significance coefficients. The article begins with a mathematical formulation of the problem of choosing optimal hyperparameters. Then, the steps of the MTMC method that solves this problem are described. The proposed method is evaluated on the image classification problem using a convolutional neural network. The article presents optimal hyperparameters for various criteria significance coefficients.
Cited by 2Open access
Masoud Ghodsian (22298480), Shima Mohammadbeigi (22298483) · Figshare · 2025
<div> Leakage in water distribution systems poses a global challenge, not only due to resource loss but also through soil erosion and sinkhole formation, which risk infrastructure collapse. This study investigates the mechanisms of soil fluidization, a process in which pressurized pipeline leakage generates turbulent water-soil mixtures, forming expanding fluidized zones. Experimental tests using a custom leakage simulation apparatus, combined with dimensional analysis, were conducted to identify key factors influencing fluidization dynamics. Empirical equations were developed to predict fluidized zone height and area (R² = 0.753–0.915) for both upward and downward leakages. These models were validated against 150 experimental datasets from current and prior studies, covering a wide range of leakage rates and soil types. Three ensemble machine learning models—Random Forest, XGBoost, and a Stacking model integrating support vector regression, multilayer perceptron, and linear regression—were employed to enhance predictive accuracy and stability. The results of evaluation metrics (R<sup>2</sup>, RMSE and correlation coefficient) showed that although XGBoost outperformed other models regarding accuracy (with R<sup>2</sup> = 0.91 in test splits), this model exhibited the lowest stability in predicting dimensionless height and area of fluidized zone (with ΔR² = 0.07–0.08). The Random Forest model had the lowest accuracy (R<sup>2</sup> = 0.907–0.912 in train phase) though the most capability in generalization through the minimum differences between train and test splits (with ΔR² = 0.02–0.04). Regarding Stacking model, both accuracy and stability maintained in balanced conditions with moderate performance. The conclusion from findings of all evaluation metrics were the same as deterministic coefficient. The equations derived from dimensional analysis, especially equations for downward leakage direction, also showed comparable performance with the most accurate ensemble model like XGBoost (with R² up to 0.915). Temporal analysis revealed the progression of fluidization through seven distinct stages over approximately 75 seconds. Fluidization initiated at a specific discharge rate, beginning with the formation of a 2 cm cavity and culminating in vortex bifurcation and dimensional stabilization. Critical pore pressure thresholds, observed around 20 seconds, induced suspended particle states and fountain flow, while vertical cavity growth predominated between 25 and 65 seconds. Sensitivity analysis highlighted the densimetric Froude number and soil uniformity as dominant factors, with their omission reducing model R² by up to 84.1% and 71.3%, respectively. In contrast, the particle size-to-leak area ratio exhibited marginal effect (<10% reduction). These findings provide critical insights into sinkhole risk mitigation, offering practical tools for infrastructure vulnerability assessment. </div>
Cited by 0Open access
Md. Shafak Shahriar Sozol (21212880), M. Rubaiyat Hossain Mondal (8415447), Achmad Husni Thamrin (21212883) · Figshare · 2025
Genetic algorithm for hyperparameter optimization.
Cited by 0Open access
Schneider, Lennart · Figshare · 2025
Trajectories of Hyperparameter Optimization (HPO) runs as analyzed in the paper Overtuning in Hyperparameter Optimization.Code for the analysis of the datasets (.csv files) can be found in the accompanying GitHub repository: https://github.com/slds-lmu/paper_2025_overtuning.`overtuning_csvs.zip` contains already postprocessed HPO trajectories of different HPO runs with columns being self-explanatory.`reshuffling_raw_csvs.zip` contains raw HPO trajectories as produced in the experiments of the paper Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimization.In contrast to https://doi.org/10.6084/m9.figshare.27627504.v1 these are the raw HPO trajectories and not the trajectories over the incumbents.<br>Additionally, HPO runs of HEBO with early stopping are included.<br>This data contains the following columns (and some additional ones not explained here which are self-explanatory) where the validation and test performance of each proposed hyperparameter configuration are tracked over time in the form of trajectoriesiteration (iteration of an HPO run)valid (validation performance)test_retrained (test performance after retraining)seed (replication id)classifier (learning algorithm)data_id (data set id)train_valid_size (size of the set used for training and validation)resampling (resampling method)metric (performance metric)method (post selection method and resampling method)optimizer (HPO algorithm)<br>
Cited by 0Open access