Lin, Ethan, Yu, Nathan, Chang, Jeromy · Scholar Commons (Santa Clara University) · 2025
Large Language Models (LLMs) are becoming increasingly popular in modern society. However, despite their popularity, the deployment of LLMs in real-world scenarios is extremely challenging due to substantial computational costs and memory constraints. Edge devices, like smartphones and IoT devices, lack resources needed to run these models locally, instead offloading computations for cloud computing. Cloud computing requires users to send their data over the internet leading to numerous privacy and security concerns. In some domains, such as health and finances, sending such sensitive information is not an option. Existing solutions to compress or increase inference speed include Small Language Models (SLMs), model compression techniques, and inference optimization strategies. However, all of these techniques require extensive human effort and manual tuning to find the optimal settings for increased speed without significant degradation of the quality of output. We propose an online hyperparameter finetuning method that autonomously discovers the optimal settings based on tangible metrics during inference. Our approach monitors performance metrics in real-time and dynamically adjusts any tunable parameter without human intervention. We demonstrate this framework on dynamic sparsity prediction, achieving 1.67÷ speedup while maintaining accuracy, but the method generalizes to any tunable parameters.
Cited by 0
Robert T. McGibbon, Carlos X. Hernández, Matthew P. Harrigan et al. · The Journal of Open Source Software · 2016
Cited by 40Open access
Pablo Martínez Martínez · 2017
Cited by 1
Tianyu Mu, Hongzhi Wang, Haoyun Tang et al. · 2024
In this era of exploding data volumes, more and more complex data analysis tasks are now accomplished by machine learning (ML) or deep learning (DL). Despite the powerful and flexible task processing capability, it also brings challenges such as how to reasonably design the optimal hyperparameters configuration and the huge consumption of time for a single validation. Existing Hyperrarameter Optimization (HPO) methods are gradually facing performance bottlenecks. However, in an era where progressively data-centric big data analytics methods are prevalent, such efficiency issues need to be urgently addressed in the design of intelligent DBMS. In this paper, we propose ShrinkHPO, an efficient and explainable-designed HPO approach with a major focus on (<tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$a$</tex>) efficient hyperparameter configuration search strategy, (b) asynchronous executing intervention, and (c) XAI (eXplainable AI) design. ShrinkHPO employs a hyperparameter weight estimation strategy named Shrink-search together with an asynchronous execution design to improve HPO efficiency. To the best of our knowledge, ShrinkHPO is the first HPO method that introduces explainable analysis and verification to ensure the judgment of “high significance” hyperparameters. We also conduct a series of experiments on data analysis tasks in the database domain (classification, regression, time series classification, CASH [1], etc.), collecting HPO results on the usual ML or DL models in each task. Compared to SOTA asynchronous and sequential HPO baselines, ShrinkHPO achieves top performance on both accuracy (RMSE for regression tasks) and time cost, accelerating from 20% to a maximum of 2.2 x.
Cited by 1
uzairgill179 · Zenodo (CERN European Organization for Nuclear Research) · 2026
First public release accompanying Scientific Reports submission
Cited by 0Open access
Oscar Rincón Cardeño, Silvana Montoya‐Noguera, Nicolás Guarín-Zapata et al. · Figshare · 2026
The top panel shows the optimization history, i.e., the evolution of the objective value across trials.<br>The bottom panels display the slice plots, illustrating the relationship between each hyperparameter (the activation function σ, the number of hidden layers 𝐿, the number of neurons per layer 𝑁, and the learning rate α of the Adam optimizer) and the achieved objective value.
Cited by 0Open access
Rincón Cardeño, Oscar, Montoya-Noguera, Silvana, Guarin-Zapata, Nicolas et al. · Figshare · 2026
The top panel shows the optimization history, i.e., the evolution of the objective value across trials.<br>The bottom panels display the slice plots, illustrating the relationship between each hyperparameter (the activation function σ, the number of hidden layers 𝐿, the number of neurons per layer 𝑁, and the learning rate α of the Adam optimizer) and the achieved objective value.
Cited by 0Open access
Lüken, Malte · Figshare · 2025
Raw outputs of the hyperparameter optimization from the paper:<br>"Generalization in neural posterior estimation: Case studies with the racing diffusion model".<br>Files:train_npe_trials.db: SQLite database with information over different Optuna trials.trials.csv: Optuna trials information in CSV format.
Cited by 0Open access
팀 주제정보서비스 · 2020
Cited by 0
Marius Lindauer, Katharina Eggensperger, Matthias Feurer et al. · arXiv (Cornell University) · 2021
Algorithm parameters, in particular hyperparameters of machine learning algorithms, can substantially impact their performance. To support users in determining well-performing hyperparameter configurations for their algorithms, datasets and applications at hand, SMAC3 offers a robust and flexible framework for Bayesian Optimization, which can improve performance within a few evaluations. It offers several facades and pre-sets for typical use cases, such as optimizing hyperparameters, solving low dimensional continuous (artificial) global optimization problems and configuring algorithms to perform well across multiple problem instances. The SMAC3 package is available under a permissive BSD-license at https://github.com/automl/SMAC3.
Cited by 83Open access
Herilalaina Rakotoarison, Steven Adriænsen, Neeratyoy Mallik et al. · arXiv (Cornell University) · 2024
With the increasing computational costs associated with deep learning, automated hyperparameter optimization methods, strongly relying on black-box Bayesian optimization (BO), face limitations. Freeze-thaw BO offers a promising grey-box alternative, strategically allocating scarce resources incrementally to different configurations. However, the frequent surrogate model updates inherent to this approach pose challenges for existing methods, requiring retraining or fine-tuning their neural network surrogates online, introducing overhead, instability, and hyper-hyperparameters. In this work, we propose FT-PFN, a novel surrogate for Freeze-thaw style BO. FT-PFN is a prior-data fitted network (PFN) that leverages the transformers' in-context learning ability to efficiently and reliably do Bayesian learning curve extrapolation in a single forward pass. Our empirical analysis across three benchmark suites shows that the predictions made by FT-PFN are more accurate and 10-100 times faster than those of the deep Gaussian process and deep ensemble surrogates used in previous work. Furthermore, we show that, when combined with our novel acquisition mechanism (MFPI-random), the resulting in-context freeze-thaw BO method (ifBO), yields new state-of-the-art performance in the same three families of deep learning HPO benchmarks considered in prior work.
Cited by 1Open access
Romain Égelé, Isabelle Guyon, Venkatram Vishwanath et al. · arXiv (Cornell University) · 2022
Bayesian optimization (BO) is a promising approach for hyperparameter optimization of deep neural networks (DNNs), where each model training can take minutes to hours. In BO, a computationally cheap surrogate model is employed to learn the relationship between parameter configurations and their performance such as accuracy. Parallel BO methods often adopt single manager/multiple workers strategies to evaluate multiple hyperparameter configurations simultaneously. Despite significant hyperparameter evaluation time, the overhead in such centralized schemes prevents these methods to scale on a large number of workers. We present an asynchronous-decentralized BO, wherein each worker runs a sequential BO and asynchronously communicates its results through shared storage. We scale our method without loss of computational efficiency with above 95% of worker's utilization to 1,920 parallel workers (full production queue of the Polaris supercomputer) and demonstrate improvement in model accuracy as well as faster convergence on the CANDLE benchmark from the Exascale computing project.
Cited by 2Open access
Binghui Si, Feng Liu, Yanxia Li · Buildings · 2023
Building energy optimization (BEO) is a promising technique to achieve energy efficient designs. The efficacy of optimization algorithms is imperative for the BEO technique and is significantly dependent on the algorithm hyperparameters. Currently, studies focusing on algorithm hyperparameters are scarce, and common agreement on how to set their values, especially for BEO problems, is still lacking. This study proposes a metamodel-based methodology for hyperparameter optimization of optimization algorithms applied in BEO. The aim is to maximize the algorithmic efficacy and avoid the failure of the BEO technique because of improper algorithm hyperparameter settings. The method consists of three consecutive steps: constructing the specific BEO problem, developing an ANN-trained metamodel of the problem, and optimizing algorithm hyperparameters with nondominated sorting genetic algorithm II (NSGA-II). To verify the validity, 15 benchmark BEO problems with different properties, i.e., five building models and three design variable categories, were constructed for numerical experiments. For each problem, the hyperparameters of four commonly used algorithms, i.e., the genetic algorithm (GA), the particle swarm optimization (PSO) algorithm, simulated annealing (SA), and the multi-objective genetic algorithm (MOGA), were optimized. Results demonstrated that the MOGA benefited the most from hyperparameter optimization in terms of the quality of the obtained optimum, while PSO benefited the most in terms of the computing time.
Cited by 10Open access
Aziz Jihadian Barid, Hadiyanto Hadiyanto · Journal of Emerging Science and Engineering · 2024
Air pollution, particularly the presence of Particulate Matter (PM) 2.5, poses significant health risks to humans, with industrial growth and urban vehicle emissions being major contributors. This study utilizes machine learning techniques, specifically K-Nearest Neighbors (KNN) and Support Vector Machine (SVM) algorithms, to predict PM2.5 levels. A dataset from Kaggle consisting of PM2.5 and other pollutant parameters is preprocessed and split into training and testing sets. The models are trained, evaluated, and compared using Mean Squared Error (MSE) and Root Mean Squared Error (RMSE) metrics. Additionally, hyperparameters are applied to optimize the models. Results show that SVM with hyperparameters performs better, indicating its potential for accurate air quality prediction. These findings can aid policymakers in implementing effective pollution control strategies.
Cited by 8Open access
Maximilian Michallik · mediaTUM (Technical University of Munich) · 2023
Cited by 0Open access
Zoltan Czako, Gheorghe Sebestyen, Anca Hângan · 2023
Anomaly detection is the technique of finding out-of-the-ordinary occurrences inside datasets. A multitude of anomaly detection methods was developed for this purpose in a wide variety of domains (e.g., economy, industrial processes, health, environmental monitoring, etc.), using as support more generic artificial intelligence and signal processing algorithms. The performance of these algorithms is strongly influenced by the initial dataset characteristics and by the hyperparameter settings that users choose for each algorithm. There is no a-priory best solution for a given domain or type of dataset; therefore, the researchers must test a multitude of methods with multiple hyperparameter settings. The research community does not yet have a multi-perspective assessment of the performance of these algorithms. Most of the work is focused on very specific problems or domains, and the algorithms are evaluated only from one or a few perspectives, using the ”default” settings, without hyperparameter optimization. In this article, we will discuss the findings obtained from an extensive anomaly detection benchmarking that covers 34 anomaly detection algorithms, tested on several relevant datasets from a wide area of domains. The results are presented from multiple perspectives. For each algorithm and dataset, a hyperparameter optimization approach is used in order to establish the optimal settings for each algorithm in each test case. This study discloses both the merits and limitations of the various techniques and proposes a taxonomy of anomaly detection algorithms based on the nature of the problem and the characteristics of the input data.
Cited by 0
Pfliegler, Dave · reposiTUm (TU Wien) · 2025
Das ALASPO Framework wendet Large Neighbourhood Search (LNS) auf Answer Set Programming (ASP) an und hat sich bereits bei der Lösung verschiedener Optimierungsprobleme bewährt. Um das Verhalten der LNS-Schleife zu steuern, bietet ALASPO eine Vielzahl von Konfigurationsoptionen, die in Kombination mit allen möglichen Parametern des zugrundeliegenden ASP-Solvers zu einem riesigen Konfigurationsraum führen. Um diese Problematik zu adressieren, integrieren wir das Framework für Bayessche Optimierung SMAC3 in ALASPO, um ein automatisches Tuning von ALASPO Konfigurationen zu ermöglichen. Weiters gruppieren wir die bereitgestellten Instanzen anhand ihrer Eigenschaften, so dass wir mehrere optimale Konfigurationen erhalten, eine für jede Gruppe ähnlicher Instanzen. Das Ergebnis des Tunings kann dann für zukünftige ALASPO Durchläufe mit bisher unbekannten Instanzen verwendet werden, um automatisch eine geeignete Konfiguration basierend auf den Eigenschaften der jeweiligen Instanz auszuwählen und anzuwenden. Wir evaluieren unsere Methodik anhand von vier Benchmark-Problemen und zeigen, dass unser Tuning bei einigen dieser Probleme zu signifikanten Verbesserungen gegenüber der Standardkonfiguration führen kann, während wir ebenso ihre Grenzen aufzeigen, wenn keine ausreichenden Verbesserungsmöglichkeiten vorhanden sind.
Cited by 0Open access
K. Saraswathi, Thangavel Renukadevi, Dileep Kumar et al. · AIP conference proceedings · 2025
Cited by 1
Minrui Zheng · 2021
Cited by 2
Plambeck, Swantje, Schmidt, Maximilian, Subias, Audine et al. · arXiv (Cornell University) · 2019
Hybrid systems, which combine both continuous and discrete behavior, are used in many fields, including robotics, biological systems, and control systems. However, due to their complexity, finding an accurate model is a challenge. This paper discusses the usage of symbolic regression to learn hybrid systems from data and specifically analyses learning parameters for a recent algorithm. Symbolic regression is a powerful tool that can automatically discover accurate and interpretable mathematical models in the form of symbolic expressions. Models generated by symbolic regression are a valuable tool for system identification and diagnosis, e.g., to predict future system behavior or detect anomalies. A major opportunity of our approach is the ability to detect transitions between different continuous behaviors of a system directly based on the dynamics. From a diagnosis perspective, this can advantageously be used to detect the system entering fault modes and identify their models. This paper presents a parameter study for a symbolic regression based identification algorithm.
Cited by 980Open access