Skip to content
Free planNo credit card requiredStart writing
AI for Research home
AI for Research home
Dashboard
Sign inGet started

Search 250 million research papers

Read the abstract, see who wrote it and who cites it, and get the citation in APA, MLA, Chicago or Harvard.

About 10,000 papers for “hyperparameter optimization”

Hyperparameter optimization for large-scale machine learning

Palacios Cuesta, Aitor · UPM Digital Archive (Technical University of Madrid) · 2018

Hyperparameter optimization is a crucial task affecting the final performance of machine learning solutions. This thesis analyzes the properties of different hyperparameter optimization algorithms for machine learning problems with large datasets. In hyperparameter optimization, we repeatedly propose a hyperparameter configuration, train a machine learning model using that configuration and validate the performance of the model. In order to scale in these scenarios, this thesis focuses on the data-parallel approach to evaluate hyperparameter configurations. We utilize Apache Spark, a dataparallel processing system, to implement a selection of hyperparameter optimization algorithms. Moreover, this thesis proposes a novel hyperparameter optimization method, Bayesian Geometric Halving (BGH). It is designed to benefit from the characteristics of dataparallel systems and the properties of existing hyperparameter optimization algorithms. BGH enhances Bayesian methods by combining them with adaptive evaluation. At the same time, it has a time-eficient execution independently of the number of con_gurations evaluated. Various experiments compare the performance of several hyperparameter optimization algorithms. These experiments extend their theoretical comparison in order tounderstand under which conditions some algorithms perform better than the others. The experiments show that BGH improves the results of both Bayesian and adaptive evaluation methods. Indeed, one of the variants of BGH achieves the best result in every experiment where they are compared.

Cited by 0Open access

Enhancing Health Monitoring using Efficient Hyperparameter Optimization

Rahul Singhal · Journal of Artificial Intelligence and Capsule Networks · 2022

Nowadays, healthcare problems among elders have been increasing at an unprecedented rate, and every year, more than a quarter of the elderly people face weakening injuries such as unexpected falls, etc. resulting in broken bones and serious injuries in some cases. Sometimes, these injuries may go unnoticed, and the resulting health consequences can have a considerable negative impact on their quality of life. Constant surveillance by trained professionals is impossible owing to the expense and effort. The detection of physical activities by different sensors and recognition processes is a key topic of research in wireless systems, smartphones and mobile computing. Sensors document and keep track of the patient's movements, to report immediately when any irregularity is found, thus saving a variety of resources. Multiple types of sensors and devices are needed for activity identification of a person's various behaviours that record or sense human actions. This work intends to gather relevant insights from data gathered from sensors and use it to categorize various human actions with machine learning using appropriate feature selection and hyperparameter tuning, and then compare the implemented models based on their performance. Understanding human behaviour is very useful in the healthcare industry, particularly in the areas of rehabilitation, elder care assistance, and cognitive impairment.

Cited by 0Open access

Hyperparameter Optimization via Interacting with Probabilistic Circuits

Seng, Jonas, Ventola, Fabrizio, Yu, Zhongjie et al. · ArXiv.org · 2025

Despite the growing interest in designing truly interactive hyperparameter optimization (HPO) methods, to date, only a few allow to include human feedback. Existing interactive Bayesian optimization (BO) methods incorporate human beliefs by weighting the acquisition function with a user-defined prior distribution. However, in light of the non-trivial inner optimization of the acquisition function prevalent in BO, such weighting schemes do not always accurately reflect given user beliefs. We introduce a novel BO approach leveraging tractable probabilistic models named probabilistic circuits (PCs) as a surrogate model. PCs encode a tractable joint distribution over the hybrid hyperparameter space and evaluation scores. They enable exact conditional inference and sampling. Based on conditional sampling, we construct a novel selection policy that enables an acquisition function-free generation of candidate points (thereby eliminating the need for an additional inner-loop optimization) and ensures that user beliefs are reflected accurately in the selection policy. We provide a theoretical analysis and an extensive empirical evaluation, demonstrating that our method achieves state-of-the-art performance in standard HPO and outperforms interactive BO baselines in interactive HPO.

Cited by 0Open access

Hyperparameter Optimization Using Iterative Decision Tree (IDT)

Narith Saum, Satoshi Sugiura, Mongkut Piantanakulchai · IEEE Access · 2022

Machine learning and deep learning have gained a lot of attention from researchers because of their promising predictive performance and the availability of extensive high-dimensional data and high-performance computational hardware. However, the performance of these algorithms is susceptible to the choice of hyperparameters, while optimizing these hyperparameters is usually computationally expensive. For this reason, this study proposed a novel sequential search algorithm using decision tree regression as the surrogate function. In each iteration of this algorithm, several new combinations of hyperparameters were selected from a few best-performed leaves of the decision tree, called Iterative Decision Tree (IDT). Our approach could reduce the computational time from repetitive training the surrogate function compared to conventional sequential search algorithms and enable parallel computing. To confirm the effectiveness of the proposed algorithm, it was compared with six popular benchmark optimization algorithms, including Grid Search, Random Search, Bayesian Optimization, Random Forest, Tree-Structured Parzen Estimation, and Genetic Algorithm. The comparison was examined by optimizing three benchmark nonconvex functions, and hyperparameter tuning of two machine learning algorithms (Support Vector Machine and Random Forest) and two deep learning models (Autoencoder and Convolutional Neural Networks). As a result, the proposed algorithm achieved competitive performance with high stability in addition to the feature importance metrics.

Cited by 10Open access

Imputation and Hyperparameter Optimization in Cancer Diagnosis

Yi Liu, Wendy Wang, Haibo Wang · Journal of Engineering Research and Sciences · 2023

Cancer is one of the leading causes for death worldwide. Accurate and timely detection of cancer can save lives. As more machine learning algorithms and approaches have been applied in cancer diagnosis, there has been a need to analyze their performance. This study has compared the detection accuracy and speed of nineteen machine learning algorithms using a cervical cancer dataset. To make the approach general enough to detect various types of cancers, this study has intentionally excluded feature selection, a feature commonly applied in most studies for a specific dataset or a certain type of cancer. In addition, imputation and hyperparameter optimization have been employed to improve the algorithms’ performance. The results suggest that when both imputation and hyperparameter optimization are applied, the algorithms tend to perform better than when either of them is employed individually or when both are absent. The majority of the algorithms have shown improved accuracy in diagnosis, although with the trade-off of increased execution time. The findings from this study demonstrate the potential of machine learning in cancer diagnosis, especially the possibility of developing versatile systems that are able to detect various types of cancers with satisfactory performance.

Cited by 0Open access

Better and faster hyperparameter optimization with Dask

Scott Sievert, Tom Augspurger, Matthew Rocklin · Proceedings of the Python in Science Conferences · 2019

Nearly every machine learning model requires hyperparameters, parameters that the user must specify before training begins and influence model performance. Finding the optimal set of hyperparameters is often a time-and resource-consuming process. A recent breakthrough hyperparameter optimization algorithm, Hyperband finds high performing hyperparameters with minimal training via a principled early stopping scheme for random hyperparameter selection [LJD + 18]. This paper will provide an intuitive introduction to Hyperband and explain the implementation in Dask, a Python library that scales Python to larger datasets and more computational resources. The implementation makes adjustments to the Hyperband algorithm to exploit Dask's capabilities and parallel processing. In experiments, the Dask implementation of Hyperband rapidly finds high performing hyperparameters for deep learning models.

Cited by 12Open access

Sequential Policy Gradient for Adaptive Hyperparameter Optimization

Zheng Li, Jerry Cheng, Huanying Gu · ArXiv.org · 2025

Reinforcement learning is essential for neural architecture search and hyperparameter optimization, but the conventional approaches impede widespread use due to prohibitive time and computational costs. Inspired by DeepSeek-V3 multi-token prediction architecture, we propose Sequential Policy Gradient modeling (SPG), a novel trajectory generation paradigm for lightweight online hyperparameter optimization. In contrast to conventional policy gradient methods, SPG extends the base model with temporary modules, enabling it to generate state-action (padded) trajectories in a single forward pass. Our experiments demonstrate that models gain performance when retrained with SPG on their original datasets and also outperform standard transfer fine-tuning. We evaluate on five datasets spanning computer vision (ImageNet, COCO), natural language processing (GLUE, SQuAD), and audio (SUPERB) to assess the industrial applicability of SPG. The proposed method demonstrates consistent improvements across widely adopted models, achieving performance gains of $+0.2\sim7\%$, with significantly low computational costs. Fully reproducible code and pre-trained models: https://huggingface.co/UniversalAlgorithmic/SPG.

Cited by 0Open access

Surrogate Methods Applied to Hyperparameter Optimization Problem

José Ilmar Cruz Freire Neto, André Britto · 2022

Hyperparameters affects the performance of machine learning models. Hyperparameter optimization is an area that aims to find the best of them, but it deals with a considerable number of machine learning training, which can be slow. Thus, surrogates can be used to soften this slow process. This paper evaluates the performance of two surrogate methods, M1 and MARSAOP, applied to hyperparameter optimization. The surrogates are confronted with six hyperparameter optimization algorithms from the literature for classification and regression problems. Results indicate that the surrogate methods are faster than the traditional algorithms.

Cited by 0Open access

Weighted Random Search for CNN Hyperparameter Optimization

Andonie, Razvan, Florea, Adrian-Catalin · arXiv (Cornell University) · 2020

Nearly all model algorithms used in machine learning use two different sets of parameters: the training parameters and the meta-parameters (hyperparameters). While the training parameters are learned during the training phase, the values of the hyperparameters have to be specified before learning starts. For a given dataset, we would like to find the optimal combination of hyperparameter values, in a reasonable amount of time. This is a challenging task because of its computational complexity. In previous work [11], we introduced the Weighted Random Search (WRS) method, a combination of Random Search (RS) and probabilistic greedy heuristic. In the current paper, we compare the WRS method with several state-of-the art hyperparameter optimization methods with respect to Convolutional Neural Network (CNN) hyperparameter optimization. The criterion is the classification accuracy achieved within the same number of tested combinations of hyperparameter values. According to our experiments, the WRS algorithm outperforms the other methods.

Cited by 0Open access

An Improved AdaBoost Algorithm for Hyperparameter Optimization

Rongfang Gao, Zhanyu Liu · Journal of Physics Conference Series · 2020

Abstract AdaBoost algorithm is a typical Boosting algorithm, which belongs to a successful representative in the Boosting family. This algorithm can upgrade a weak classifier with a better classification effect than random classification to a strong classifier with high classification accuracy, where n_estimators represents the number of iterations of the base classifier. If the value is too large, it will easily cause the model to overfit, if it is too small, it is easy. The model is under-fitting, and the parameter setting is not set randomly, but according to the current status of the data set. Aiming at the problem that the number of iterations in the AdaBoost algorithm is uncertain, this paper introduces a Bayesian optimization algorithm for hyperparameter tuning, which makes the value of hyper parameter in AdaBoost algorithm suitable for the current data set, and finally obtains a hyperparameter optimization AdaBoost algorithm. The experiment result shows the method that adopt Bayesian optimization algorithm for hyperparameter optimization and apply the optimized hyperparameter value to the AdaBoost algorithm does not only improves the classification accuracy of the AdaBoost algorithm, but also avoids overfitting and underfitting of the model.

Cited by 16Open access

Using Large Language Models for Hyperparameter Optimization

Michael R. Zhang, Nishkrit Desai, Juhan Bae et al. · arXiv (Cornell University) · 2023

This paper explores the use of foundational large language models (LLMs) in hyperparameter optimization (HPO). Hyperparameters are critical in determining the effectiveness of machine learning models, yet their optimization often relies on manual approaches in limited-budget settings. By prompting LLMs with dataset and model descriptions, we develop a methodology where LLMs suggest hyperparameter configurations, which are iteratively refined based on model performance. Our empirical evaluations on standard benchmarks reveal that within constrained search budgets, LLMs can match or outperform traditional HPO methods like Bayesian optimization across different models on standard benchmarks. Furthermore, we propose to treat the code specifying our model as a hyperparameter, which the LLM outputs and affords greater flexibility than existing HPO approaches.

Cited by 7Open access

Hyperparameter Optimization for Multi-Objective Reinforcement Learning

Florian Felten, Д.Р. Гареев, El-Ghazali Talbi et al. · arXiv (Cornell University) · 2023

Reinforcement learning (RL) has emerged as a powerful approach for tackling complex problems. The recent introduction of multi-objective reinforcement learning (MORL) has further expanded the scope of RL by enabling agents to make trade-offs among multiple objectives. This advancement not only has broadened the range of problems that can be tackled but also created numerous opportunities for exploration and advancement. Yet, the effectiveness of RL agents heavily relies on appropriately setting their hyperparameters. In practice, this task often proves to be challenging, leading to unsuccessful deployments of these techniques in various instances. Hence, prior research has explored hyperparameter optimization in RL to address this concern. This paper presents an initial investigation into the challenge of hyperparameter optimization specifically for MORL. We formalize the problem, highlight its distinctive challenges, and propose a systematic methodology to address it. The proposed methodology is applied to a well-known environment using a state-of-the-art MORL algorithm, and preliminary results are reported. Our findings indicate that the proposed methodology can effectively provide hyperparameter configurations that significantly enhance the performance of MORL agents. Furthermore, this study identifies various future research opportunities to further advance the field of hyperparameter optimization for MORL.

Cited by 2Open access

Clinical BioBERT Hyperparameter Optimization using Genetic Algorithm

Navya Martin Kollapally, James Geller · arXiv (Cornell University) · 2023

Clinical factors account only for a small portion, about 10-30%, of the controllable factors that affect an individual's health outcomes. The remaining factors include where a person was born and raised, where he/she pursued their education, what their work and family environment is like, etc. These factors are collectively referred to as Social Determinants of Health (SDoH). The majority of SDoH data is recorded in unstructured clinical notes by physicians and practitioners. Recording SDoH data in a structured manner (in an EHR) could greatly benefit from a dedicated ontology of SDoH terms. Our research focuses on extracting sentences from clinical notes, making use of such an SDoH ontology (called SOHO) to provide appropriate concepts. We utilize recent advancements in Deep Learning to optimize the hyperparameters of a Clinical BioBERT model for SDoH text. A genetic algorithm-based hyperparameter tuning regimen was implemented to identify optimal parameter settings. To implement a complete classifier, we pipelined Clinical BioBERT with two subsequent linear layers and two dropout layers. The output predicts whether a text fragment describes an SDoH issue of the patient. We compared the AdamW, Adafactor, and LAMB optimizers. In our experiments, AdamW outperformed the others in terms of accuracy.

Cited by 0Open access

CNN Hyperparameter Optimization for Pulmonary Nodule Classification

Anthony Jatobá, Lucas Lima, Lucas Amorim et al. · 2020

Redes Neurais Convolucionais (RNCs) são uma técnica poderosa para sistemas de diagnóstico auxiliado por computador, mas a configuração manual de redes complexas é inviável. A otimização automática de hiper-parâmetros é uma abordagem promissora, mas não há consenso sobre a técnica mais adequada. Neste trabalho, comparamos busca direta, probabilística e otimização bayesiana na otimização de RNCs 2D e 3D para classificação de nódulos pulmonares. Foram obtidas AUC de 0,88, sensibilidade de 87,03% e especificidade de 78,66%. Nossos experimentos demonstram o fraco desempenho da busca em grade, enquanto mostram que técnicas simples, como a busca aleatória, pode ter desempenho comparável a abordagens probabilísticas

Cited by 3Open access