Luca Franceschi, Michele Donini, Valerio Perrone et al. · arXiv (Cornell University) · 2024
Hyperparameters are configuration variables controlling the behavior of machine learning algorithms. They are ubiquitous in machine learning and artificial intelligence and the choice of their values determines the effectiveness of systems based on these technologies. Manual hyperparameter search is often time-consuming and becomes infeasible when the number of hyperparameters is large. Automating the search is an important step towards advancing, streamlining, and systematizing machine learning, freeing researchers and practitioners alike from the burden of finding a good set of hyperparameters by trial and error. In this survey, we present a unified treatment of hyperparameter optimization, providing the reader with examples, insights into the state-of-the-art, and numerous links to further reading. We cover the main families of techniques to automate hyperparameter search, often referred to as hyperparameter optimization or tuning, including random and quasi-random search, bandit-, model-, population-, and gradient-based approaches. We further discuss extensions, including online, constrained, and multi-objective formulations, touch upon connections with other fields, such as meta-learning and neural architecture search, and conclude with open questions and future research directions.
Cited by 5Open access
Josif Grabocka, Maciej Janowski, Arlind Kadra et al. · 2023
Cited by 0
Fabián Pedregosa · arXiv (Cornell University) · 2016
Most models in machine learning contain at least one hyperparameter to control for model complexity. Choosing an appropriate set of hyperparameters is both crucial in terms of model accuracy and computationally challenging. In this work we propose an algorithm for the optimization of continuous hyperparameters using inexact gradient information. An advantage of this method is that hyperparameters can be updated before model parameters have fully converged. We also give sufficient conditions for the global convergence of this method, based on regularity conditions of the involved functions and summability of errors. Finally, we validate the empirical performance of this method on the estimation of regularization constants of L2-regularized logistic regression and kernel Ridge regression. Empirical benchmarks indicate that our approach is highly competitive with respect to state of the art methods.
Cited by 139Open access
Joydeb Kumar Sana (14204182), Mohammad Zoynul Abedin (13488619), M. Sohel Rahman (12056885) et al. · Figshare · 2022
<p><b>(Left.)</b> For logistic regression, the parameter C is tuned. The best performance is attained when <i>C</i> = 5. <b>(Right.)</b> The hyperparameters <i>epoch</i> and <i>batch_size</i> are optimized for the Feed-forward Neural Network. The best performance is attained when <i>epoch</i> = 10, <i>batch</i>_<i>size</i> = 10.</p>
Cited by 0Open access
Segel, Sarah, Graf, Helena, Tornede, Alexander et al. · Zenodo (CERN European Organization for Nuclear Research) · 2023
Hyperparameter optimization (HPO) methods can determine well-performing hyperparameter configurations efficiently but often lack insights and transparency. We propose to apply symbolic regression to meta-data collected with Bayesian optimization (BO) during HPO. In contrast to prior approaches explaining the effects of hyperparameters on model performance, symbolic regression allows for obtaining explicit formulas quantifying the relation between hyperparameter values and model performance. Overall, our approach aims to make the HPO process more explainable and human-centered, addressing the needs of multiple user groups: First, providing insights into the HPO process can support data scientists and machine learning practitioners in their decisions when using and interacting with HPO tools. Second, obtaining explicit formulas and inspecting their properties could help researchers understand the HPO loss landscape better. In an experimental evaluation, we find that naively applying symbolic regression directly to meta-data collected during HPO is affected by the sampling bias introduced by BO. However, the true underlying loss landscape can be approximated by fitting the symbolic regression on the surrogate model trained during BO. By penalizing longer formulas, symbolic regression furthermore allows the user to decide how to balance the accuracy and explainability of the resulting formulas.
Cited by 0Open access
Schneider, Lennart · Figshare · 2025
Trajectories of Hyperparameter Optimization (HPO) runs as analyzed in the paper Overtuning in Hyperparameter Optimization.Code for the analysis of the datasets (.csv files) can be found in the accompanying GitHub repository: https://github.com/slds-lmu/paper_2025_overtuning.`overtuning_csvs.zip` contains already postprocessed HPO trajectories of different HPO runs with columns being self-explanatory.`reshuffling_raw_csvs.zip` contains raw HPO trajectories as produced in the experiments of the paper Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimization.In contrast to https://doi.org/10.6084/m9.figshare.27627504.v1 these are the raw HPO trajectories and not the trajectories over the incumbents.<br>Additionally, HPO runs of HEBO with early stopping are included.<br>This data contains the following columns (and some additional ones not explained here which are self-explanatory) where the validation and test performance of each proposed hyperparameter configuration are tracked over time in the form of trajectoriesiteration (iteration of an HPO run)valid (validation performance)test_retrained (test performance after retraining)seed (replication id)classifier (learning algorithm)data_id (data set id)train_valid_size (size of the set used for training and validation)resampling (resampling method)metric (performance metric)method (post selection method and resampling method)optimizer (HPO algorithm)<br>
Cited by 0Open access
Balraj Preet Kaur (20370832), Harpreet Singh (677440), Rahul Hans (20370835) et al. · Figshare · 2024
<div><p>In the current era, a lot of research is being done in the domain of disease diagnosis using machine learning. In recent times, one of the deadliest respiratory diseases, COVID-19, which causes serious damage to the lungs has claimed a lot of lives globally. Machine learning-based systems can assist clinicians in the early diagnosis of the disease, which can reduce the deadly effects of the disease. For the successful deployment of these machine learning-based systems, hyperparameter-based optimization and feature selection are important issues. Motivated by the above, in this proposal, we design an improved model to predict the existence of respiratory disease among patients by incorporating hyperparameter optimization and feature selection. To optimize the parameters of the machine learning algorithms, hyperparameter optimization with a genetic algorithm is proposed and to reduce the size of the feature set, feature selection is performed using binary grey wolf optimization algorithm. Moreover, to enhance the efficacy of the predictions made by hyperparameter-optimized machine learning models, an ensemble model is proposed using a stacking classifier. Also, explainable AI was incorporated to define the feature importance by making use of Shapely adaptive explanations (SHAP) values. For the experimentation, the publicly accessible Mexico clinical dataset of COVID-19 was used. The results obtained show that the proposed model has superior prediction accuracy in comparison to its counterparts. Moreover, among all the hyperparameter-optimized algorithms, adaboost algorithm outperformed all the other hyperparameter-optimized algorithms. The various performance assessment metrics, including accuracy, precision, recall, AUC, and F1-score, were used to assess the results.</p></div>
Cited by 0Open access
Song, Y · Figshare · 2025
<pre>Hyperparameter optimization stacking framework code.</pre>
Cited by 0Open access
Bagge Carlson, Fredrik · Lund University Publications (Lund University) · 2018
Cited by 0
Maria Mahbub (11914235), Sudarshan Srinivasan (11914238), Ioana Danciu (11914241) et al. · Figshare · 2022
Cited by 0Open access
Matías Martínez, Jean‐Rémy Falleri, Martin Monperrus · arXiv (Cornell University) · 2020
Computing the differences between two versions of the same program is an essential task for software development and software evolution research. AST differencing is the most advanced way of doing so, and an active research area. Yet, AST differencing algorithms rely on configuration parameters that may have a strong impact on their effectiveness. In this paper, we present a novel approach named DAT (Diff Auto Tuning) for hyperparameter optimization of AST differencing. We thoroughly state the problem of hyper-configuration for AST differencing. We evaluate our data-driven approach DAT to optimize the edit-scripts generated by the state-of-the-art AST differencing algorithm named GumTree in different scenarios. DAT is able to find a new configuration for GumTree that improves the edit-scripts in 21.8% of the evaluated cases.
Cited by 1Open access
Tanay Agrawal · Apress eBooks · 2020
In the previous chapter, you learned what hyperparameters are and how they affect the performance of an algorithm. Now that you know how important it is to tune hyperparameters, this chapter introduces you to some simple yet powerful uses of algorithms implemented in the scikit-learn library for hyperparameter optimization. Scikit-learn is one of the most widely used open source libraries for machine learning practices. It’s simple to use and really effective in predictive analysis.
Cited by 45
Elad Hazan, Adam R. Klivans, Yuan Yang · arXiv (Cornell University) · 2017
We give a simple, fast algorithm for hyperparameter optimization inspired by techniques from the analysis of Boolean functions. We focus on the high-dimensional regime where the canonical example is training a neural network with a large number of hyperparameters. The algorithm --- an iterative application of compressed sensing techniques for orthogonal polynomials --- requires only uniform sampling of the hyperparameters and is thus easily parallelizable. Experiments for training deep neural networks on Cifar-10 show that compared to state-of-the-art tools (e.g., Hyperband and Spearmint), our algorithm finds significantly improved solutions, in some cases better than what is attainable by hand-tuning. In terms of overall running time (i.e., time required to sample various settings of hyperparameters plus additional computation time), we are at least an order of magnitude faster than Hyperband and Bayesian Optimization. We also outperform Random Search 8x. Additionally, our method comes with provable guarantees and yields the first improvements on the sample complexity of learning decision trees in over two decades. In particular, we obtain the first quasi-polynomial time algorithm for learning noisy decision trees with polynomial sample complexity.
Cited by 27Open access
Matías Martínez, Jean‐Rémy Falleri, Martin Monperrus · IEEE Transactions on Software Engineering · 2023
Computing the differences between two versions of the same program is an essential task for software development and software evolution research. AST differencing is the most advanced way of doing so, and an active research area. Yet, AST differencing algorithms rely on configuration parameters that may have a strong impact on their effectiveness. In this paper, we present a novel approach named <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DAT</monospace> (D <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">iff <u>A</u>uto <u>T</u>uning</i> ) for hyperparameter optimization of AST differencing. We thoroughly state the problem of hyper-configuration for AST differencing. We evaluate our data-driven approach <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DAT</monospace> to optimize the edit-scripts generated by the state-of-the-art AST differencing algorithm named GumTree in different scenarios. <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DAT</monospace> is able to find a new configuration for GumTree that improves the edit-scripts in 21.8% of the evaluated cases.
Cited by 12Open access
Pedregosa, Fabian · Base Institutionnelle de Recherche de l'université Paris-Dauphine (BIRD) (University Paris-Dauphine) · 2016
Most models in machine learning contain at least one hyperparameter to control for model complexity. Choosing an appropriate set of hyperparameters is both crucial in terms of model accuracy and computationally challenging. In this work we propose an algorithm for the optimization of continuous hyperparameters using inexact gradient information. An advantage of this method is that hyperparameters can be updated before model parameters have fully converged. We also give sufficient conditions for the global convergence of this method, based on regularity conditions of the involved functions and summability of errors. Finally, we validate the empirical performance of this method on the estimation of regularization constants of L2-regularized logistic regression and kernel Ridge regression. Empirical benchmarks indicate that our approach is highly competitive with respect to state of the art methods.
Cited by 0Open access
Katharina Eggensperger, Frank Hutter, Holger H. Hoos et al. · 2014
Abstract. Since hyperparameter optimization is crucial for achiev-ing peak performance with many machine learning algorithms, an active research community has formed around this problem in the last few years. The evaluation of new hyperparameter optimization techniques against the state of the art requires a set of benchmarks. Because such evaluations can be very expensive, early experiments are often performed using synthetic test functions rather than using real-world hyperparameter optimization problems. However, there can be a wide gap between the two kinds of problems. In this work, we introduce another option: cheap-to-evaluate surrogates of real hyperparameter optimization benchmarks that share the same hyper-parameter spaces and feature similar response surfaces. Specifically, we train regression models on data describing a machine learning algorithm’s performance under a wide range of hyperparameter con-figurations, and then cheaply evaluate hyperparameter optimization methods using the model’s performance predictions in lieu of the real algorithm. We evaluate the effectiveness for using a wide range of regression techniques to build these surrogate benchmarks, both in terms of how well they predict the performance of new configurations and of how much they affect the overall performance of hyperparame-ter optimizers. Overall, we found that surrogate benchmarks based on random forests performed best: for benchmarks with few hyperparam-eters they yielded almost perfect surrogates, and for benchmarks with more complex hyperparameter spaces they still yielded surrogates that were qualitatively similar to the real benchmarks they model. 1
Cited by 8
Marc Becker, Michel Lang, Jakob Richter et al. · 2019
Hyperparameter optimization package of the 'mlr3' ecosystem. It features highly configurable search spaces via the 'paradox' package and finds optimal hyperparameter configurations for any 'mlr3' learner. 'mlr3tuning' works with several optimization algorithms e.g. Random Search, Iterated Racing, Bayesian Optimization (in 'mlr3mbo') and Hyperband (in 'mlr3hyperband'). Moreover, it can automatically optimize learners and estimate the performance of optimized models with nested resampling.
Cited by 2Open access
Daniel Schirmacher (19475634), Ümmünur Armagan (19475637), Yang Zhang (30734) et al. · Figshare · 2024
<p>D1 pretrained aiSEGcell, retrained on (a) D3 (hyperparameter: binary cross entropy loss weights; evaluation: average image-wise intersection over union (IOU)) and (b) D4 (hyperparameters: binary cross entropy loss weights, learning rate; evaluation: ). All other hyperparameters were identical between hyperparameter conditions. For each hyperparameter combination three random weight initializations were trained. Models were evaluated on the respective validation set (colored square is best model; n = 10 unique field of views, N = 1 biological replicate).</p> <p>(TIF)</p>
Cited by 0Open access
Huan Liu, Shizheng Qu, Shuai Zhang et al. · PLoS ONE · 2025
The parameter values of neural networks will directly affect the performance of the network, so it is very important to choose the appropriate parameter tuning method to improve the performance of the neural network. In this paper, the improved beluga whale optimization hyperparameter optimization ResNet model is used to construct a new model, EBWO-ResNet. Firstly, in order to solve the problem that the initial population of the original beluga whale optimization is not rich enough, the Tent chaotic map is introduced into the beluga whale optimization, and a new algorithm EBWO is constructed. Secondly, in order to solve the problems of low accuracy and difficult parameter tuning of ResNet, the EBWO algorithm was integrated into ResNet to construct a new model EBWO-ResNet. Finally, in order to verify the effectiveness of the EBWO algorithm, the EBWO algorithm was applied to three engineering problems and compared with other five swarm intelligent algorithms, and in order to verify the effectiveness of the EBWO-ResNet model, EBWO-ResNet was applied to maize disease identification,in order to improve the accuracy of corn identification and ensure corn yield,and the other seven models were compared based on three evaluation indexes. The experimental results show that the EBWO algorithm provides the best solutions in the three engineering problems, and the EBWO-ResNet has the best performance in identifying maize diseases, with an accuracy of 96.3%,which is 0.2-1.5 percentage points higher than that of other models.
Cited by 0Open access
Fehring, Lukas, Wever, Marcel, Spliethöver, Maximilian et al. · arXiv (Cornell University) · 2025
Bayesian optimization (BO) is a widely used approach to hyperparameter optimization (HPO). However, most existing HPO methods only incorporate expert knowledge during initialization, limiting practitioners' ability to influence the optimization process as new insights emerge. This limits the applicability of BO in iterative machine learning development workflows. We propose DynaBO, a BO framework that enables continuous user control of the optimization process. Over time, DynaBO leverages provided user priors by augmenting the acquisition function with decaying, prior-weighted preferences while preserving asymptotic convergence guarantees. To reinforce robustness, we introduce a data-driven safeguard that detects and can be used to reject misleading priors. We prove theoretical results on near-certain convergence, robustness to adversarial priors, and accelerated convergence when informative priors are provided. Extensive experiments across various HPO benchmarks show that DynaBO consistently outperforms our state-of-the-art competitors across all benchmarks and for all prior kinds. Our results demonstrate that DynaBO enables reliable and efficient collaborative BO, bridging automated and manually controlled model development.
Cited by 0Open access