Fehring, Lukas, Wever, Marcel, Spliethöver, Maximilian et al. · ArXiv.org · 2025
Bayesian optimization (BO) is a widely used approach to hyperparameter optimization (HPO). However, most existing HPO methods only incorporate expert knowledge during initialization, limiting practitioners' ability to influence the optimization process as new insights emerge. This limits the applicability of BO in iterative machine learning development workflows. We propose DynaBO, a BO framework that enables continuous user control of the optimization process. Over time, DynaBO leverages provided user priors by augmenting the acquisition function with decaying, prior-weighted preferences while preserving asymptotic convergence guarantees. To reinforce robustness, we introduce a data-driven safeguard that detects and can be used to reject misleading priors. We prove theoretical results on near-certain convergence, robustness to adversarial priors, and accelerated convergence when informative priors are provided. Extensive experiments across various HPO benchmarks show that DynaBO consistently outperforms our state-of-the-art competitors across all benchmarks and for all prior kinds. Our results demonstrate that DynaBO enables reliable and efficient collaborative BO, bridging automated and manually controlled model development.
Cited by 0Open access
Bruno G. Galuzzi, Ilaria Giordani, Antonio Candelieri et al. · Computational Management Science · 2020
Abstract Recommender systems represent one of the most successful applications of machine learning in B2C online services, to help the users in their choices in many web services. Recommender system aims to predict the user preferences from a huge amount of data, basically the past behaviour of the user, using an efficient prediction algorithm. One of the most used is the matrix-factorization algorithm. Like many machine learning algorithms, its effectiveness goes through the tuning of its hyper-parameters, and the associated optimization problem also called hyper-parameter optimization. This represents a noisy time-consuming black-box optimization problem. The related objective function maps any possible hyper-parameter configuration to a numeric score quantifying the algorithm performance. In this work, we show how Bayesian optimization can help the tuning of three hyper-parameters: the number of latent factors, the regularization parameter, and the learning rate. Numerical results are obtained on a benchmark problem and show that Bayesian optimization obtains a better result than the default setting of the hyper-parameters and the random search.
Cited by 45Open access
B James, D Yamins, D Cox et al. · International Journal of Machine Learning and Computing · 2022
This paper proposes a new hyperparameter search method involving elliptical grid transformations and rotations of a grid of probe points. This technique is termed "rotated grid search". We begin by motivating the method by discussing the limitations of random search. A new formalism for more efficiently probing a hyperparameter search space is then proposed. Next, we build a theoretical framework to compare hyperparameter optimization performance of rotated grid search against random search. We then evaluate both search methods empirically to quantify the marginal benefit of using one over the other. Monte-Carlo simulations on various synthetic objective functions show that rotated grid search outperforms random search over the full range of anisotropy explored in this study. Finally, we conduct a case study on a real dataset, rectangles-images, and show that rotated grid search outperforms random search in a high dimensional space.
Cited by 2Open access
Martínez Martínez, Pablo · RECERCAT (Consorci de Serveis Universitaris de Catalunya) · 2017
In this work we propose a SSGD based gamma optimization algorithm for Support Vector Machines model. This algorithm is combined with a nouvelle regularization technique called model dropout that make our algorithm competitive with the state of the art.
Cited by 0Open access
André Mendes, Paulo Trigo, Joel Paulo · Procedia Computer Science · 2024
The detection and identification of impulsive sounds applied in a specific context, particularly in sports events, enables the analysis and synthesis of various metrics and statistics associated with the game or even the player's performance. In this context, the automatic identification of an impulsive sound, such as a ball being hit by a player, is a major contribution to the construction of a data source on which game-specific analysis can be performed. Considering all the characteristics (features) of a particular type of impulsive sound in various conditions/environments involves dealing with numerous variables, making it equally challenging to efficiently find the values of the hyperparameters that allow obtaining the best configuration for a given algorithm to be executed in a machine learning process. The contribution of this work is to explore the hyperparameter space in search of values that optimize the performance of the entire process of automatic impulsive sound classification. This process begins with the generation of the dataset to be processed, continues with the training of classification models, and ends with the evaluation of the learned models. The experiments consider a binary classification problem, where a distinction must be made between the intended event and noise. The validation of the process resorts to an audio extracted from videos of sports event competitions, specifically in tennis and padel, where the goal is to identify the sounds of racket hits on the ball. This work is currently in progress, but the preliminary results already enable to evidence the impact of hyperparameter optimization on the accuracy of the overall learning process.
Cited by 2Open access
Hakim Benkirane (14725564), Maria Vakalopoulou (21560771), David Planchard (5843483) et al. · Figshare · 2025
<b>a.</b> Evolution of Lung False Positive Rate (LFPR) across different RNAseq dropout rates during testing, with baseline Missing at Random (MAR) performance (green dashed line) and 95% confidence intervals (shaded area). <b>b.</b> Rank correlation comparison between MoE and attention-based feature importance across three TCGA cancer types (BRCA, COAD, STAD) with error bars indicating standard deviation. <b>c.</b> Ablation studies showing model performance sensitivity to hypergraph construction parameters: similarity threshold (left) and kernel parameter ratio (right), with 95% confidence intervals (shaded areas). (TIF)
Cited by 0Open access
Adrian-Catalin Florea, Razvan Andonie · International Journal of Computers Communications & Control · 2019
We introduce an improved version of Random Search (RS), used here for hyperparameter optimization of machine learning algorithms. Unlike the standard RS, which generates for each trial new values for all hyperparameters, we generate new values for each hyperparameter with a probability of change. The intuition behind our approach is that a value that already triggered a good result is a good candidate for the next step, and should be tested in new combinations of hyperparameter values. Within the same computational budget, our method yields better results than the standard RS. Our theoretical results prove this statement. We test our method on a variation of one of the most commonly used objective function for this class of problems (the Grievank function) and for the hyperparameter optimization of a deep learning CNN architecture. Our results can be generalized to any optimization problem dened on a discrete domain.
Cited by 41Open access
Chepuri Shri Krishna, Ashish Gupta, Swarnim Narayan et al. · arXiv (Cornell University) · 2020
Reinforcement Learning has yielded promising results for Neural Architecture Search (NAS). In this paper, we demonstrate how its performance can be improved by using a simplified Transformer block to model the policy network. The simplified Transformer uses a 2-stream attention-based mechanism to model hyper-parameter dependencies while avoiding layer normalization and position encoding. We posit that this parsimonious design balances model complexity against expressiveness, making it suitable for discovering optimal architectures in high-dimensional search spaces with limited exploration budgets. We demonstrate how the algorithm's performance can be further improved by a) using an actor-critic style algorithm instead of plain vanilla policy gradient and b) ensembling Transformer blocks with shared parameters, each block conditioned on a different auto-regressive factorization order. Our algorithm works well as both a NAS and generic hyper-parameter optimization (HPO) algorithm: it outperformed most algorithms on NAS-Bench-101, a public data-set for benchmarking NAS algorithms. In particular, it outperformed RL based methods that use alternate architectures to model the policy network, underlining the value of using attention-based networks in this setting. As a generic HPO algorithm, it outperformed Random Search in discovering more accurate multi-layer perceptron model architectures across 2 regression tasks. We have adhered to guidelines listed in Lindauer and Hutter while designing experiments and reporting results.
Cited by 2Open access
Krishna Khadka, Jaganmohan Chandrasekaran, Yu Lei et al. · 2024
In machine learning, hyperparameter optimization (HPO) is essential for effective model training and significantly impacts model performance. Hyperparameters are predefined model settings which fine-tune the model's behavior and are critical to modeling complex data patterns. Traditional HPO approaches such as Grid Search, Random Search, and Bayesian Optimization have been widely used in this field. However, as datasets grow and models increase in complexity, these approaches often require a significant amount of time and resources for HPO. This research introduces a novel approach using t-way testing---a combinatorial approach to software testing used for identifying faults with a test set that covers all t-way interactions---for HPO. T-way testing substantially narrows the search space and effectively covers parameter interactions. Our experimental results show that our approach reduces the number of necessary model evaluations and significantly cuts computational expenses while still outperforming traditional HPO approaches for the models studied in our experiments.
Cited by 9Open access
Somia A. Asklany (22566824), Doaa Mohammed (22655880), Ismail K. Youssef (22566821) et al. · Figshare · 2025
Hyperparameter optimization setup and final values.
Cited by 0Open access
H. Haddad, Thibault Falque, P. Talbot et al. · ArXiv.org · 2026
The performance of constraint programming solvers is highly sensitive to the choice of their hyperparameters. Manually finding the best solver configuration is a difficult, time-consuming task that typically requires expert knowledge. In this paper, we introduce probe and solve algorithm, a novel two-phase framework for automated hyperparameter optimization integrated into the CPMpy library. This approach partitions the available time budget into two phases: a probing phase that explores different sets of hyperparameters using configurable hyperparameter optimization methods, followed by a solving phase where the best configuration found is used to tackle the problem within the remaining time. We implement and compare two hyperparameter optimization methods within the probe and solve algorithm: Bayesian optimization and Hamming distance search. We evaluate the algorithm on two different constraint programming solvers, ACE and Choco, across 114 combinatorial problem instances, comparing their performance against the solver's default configurations. Results show that using Bayesian optimization, the algorithm outperforms the solver's default configurations, improving solution quality for ACE in 25.4% of instances and matching the default performance in 57.9%, and for Choco, achieving superior results in 38.6% of instances. It also consistently surpasses Hamming distance search within the same framework, confirming the advantage of model-based exploration over simple local search. Overall, the probe and solve algorithm offers a practical, resource-aware approach for tuning constraint solvers that yields robust improvements across diverse problem types.
Cited by 0Open access
Wesley Chorney, Haifeng Wang, Adrian Sescu · Information Sciences · 2025
Cited by 0
H. Haddad, Thibault Falque, P. Talbot et al. · arXiv (Cornell University) · 2026
The performance of constraint programming solvers is highly sensitive to the choice of their hyperparameters. Manually finding the best solver configuration is a difficult, time-consuming task that typically requires expert knowledge. In this paper, we introduce probe and solve algorithm, a novel two-phase framework for automated hyperparameter optimization integrated into the CPMpy library. This approach partitions the available time budget into two phases: a probing phase that explores different sets of hyperparameters using configurable hyperparameter optimization methods, followed by a solving phase where the best configuration found is used to tackle the problem within the remaining time. We implement and compare two hyperparameter optimization methods within the probe and solve algorithm: Bayesian optimization and Hamming distance search. We evaluate the algorithm on two different constraint programming solvers, ACE and Choco, across 114 combinatorial problem instances, comparing their performance against the solver's default configurations. Results show that using Bayesian optimization, the algorithm outperforms the solver's default configurations, improving solution quality for ACE in 25.4% of instances and matching the default performance in 57.9%, and for Choco, achieving superior results in 38.6% of instances. It also consistently surpasses Hamming distance search within the same framework, confirming the advantage of model-based exploration over simple local search. Overall, the probe and solve algorithm offers a practical, resource-aware approach for tuning constraint solvers that yields robust improvements across diverse problem types.
Cited by 0Open access
Joydeb Kumar Sana (14204182), Mohammad Zoynul Abedin (13488619), M. Sohel Rahman (12056885) et al. · Figshare · 2022
Cited by 0Open access
Franceschi L., Donini M., Frasconi P. et al. · Florence Research (University of Florence) · 2019
We study two procedures (reverse-mode and forward-mode) for computing the gradient of the validation error with respect to the hyperparameters of any iterative learning algorithm. These procedures mirror two ways of computing gradients for recurrent neural networks and have different trade-offs in terms of running time and space requirements. The reverse-mode procedure extends previous work by Maclaurin et al. (2015) and offers the opportunity to insert constraints on the hyperparameters in a natural way. The forward-mode procedure is suitable for real-time hyperparameter updates, which may significantly speedup the overall hyperparameter optimization process.
Cited by 0
Souparni Agnihotri · 2019
Cited by 5
M-Lipiec, JonathanMelcher · Zenodo (CERN European Organization for Nuclear Research) · 2024
Release as part of paper submission.
Cited by 0Open access
Maryam Tavakol, Sebastian Mair, Katharina Morik · Communications in computer and information science · 2020
Cited by 2
Debus, Charlotte, Rüttgers, Alexander · elib (German Aerospace Center) · 2020
Part I: Theory - Basics of Hyperparameter Optimization - Exhausive Searches - Surrogate-based Optimization, Sequential Model-based Optimization and Bayesian Inference - Evolutionary Strategies Part II: Hands-on programming session "Hyperparameter Optimization for Improving Neural Networks" - Manual hyperparameter optimization of a Keras regression model - Machine-assisted optimization of the Keras model using Talos - Good practice guidelines for hyperparameter tuning.
Cited by 0Open access
Ken Zheng · 2024
Cited by 1Open access