Jonas El Gammal, Nils Schöneberg, Jesús Torrado et al. · arXiv (Cornell University) · 2022
We present the GPry algorithm for fast Bayesian inference of general (non-Gaussian) posteriors with a moderate number of parameters. GPry does not need any pre-training, special hardware such as GPUs, and is intended as a drop-in replacement for traditional Monte Carlo methods for Bayesian inference. Our algorithm is based on generating a Gaussian Process surrogate model of the log-posterior, aided by a Support Vector Machine classifier that excludes extreme or non-finite values. An active learning scheme allows us to reduce the number of required posterior evaluations by two orders of magnitude compared to traditional Monte Carlo inference. Our algorithm allows for parallel evaluations of the posterior at optimal locations, further reducing wall-clock times. We significantly improve performance using properties of the posterior in our active learning scheme and for the definition of the GP prior. In particular we account for the expected dynamical range of the posterior in different dimensionalities. We test our model against a number of synthetic and cosmological examples. GPry outperforms traditional Monte Carlo methods when the evaluation time of the likelihood (or the calculation of theoretical observables) is of the order of seconds; for evaluation times of over a minute it can perform inference in days that would take months using traditional methods. GPry is distributed as an open source Python package (pip install gpry) and can also be found at https://github.com/jonaselgammal/GPry.
Cited by 1Open access
Jonas El Gammal, Nils Schöneberg, Jesús Torrado et al. · Journal of Cosmology and Astroparticle Physics · 2023
Abstract We present the GPry algorithm for fast Bayesian inference of general (non-Gaussian) posteriors with a moderate number of parameters. GPry does not need any pre-training, special hardware such as GPUs, and is intended as a drop-in replacement for traditional Monte Carlo methods for Bayesian inference. Our algorithm is based on generating a Gaussian Process surrogate model of the log-posterior, aided by a Support Vector Machine classifier that excludes extreme or non-finite values. An active learning scheme allows us to reduce the number of required posterior evaluations by two orders of magnitude compared to traditional Monte Carlo inference. Our algorithm allows for parallel evaluations of the posterior at optimal locations, further reducing wall-clock times. We significantly improve performance using properties of the posterior in our active learning scheme and for the definition of the GP prior. In particular we account for the expected dynamical range of the posterior in different dimensionalities. We test our model against a number of synthetic and cosmological examples. GPry outperforms traditional Monte Carlo methods when the evaluation time of the likelihood (or the calculation of theoretical observables) is of the order of seconds; for evaluation times of over a minute it can perform inference in days that would take months using traditional methods. GPry is distributed as an open source Python package ( pip install gpry ) and can also be found at https://github.com/jonaselgammal/GPry .
Cited by 20Open access
Coker, Beau · Digital Access to Scholarship at Harvard (DASH) (Harvard University) · 2023
In many fields there is a need to estimate nonlinear functions and provide reliable uncertainty quantification. Gaussian processes (GPs) enable easy specification of prior assumptions and rigorous guarantees of posterior properties, but scale poorly with the number of observations. On the other hand, neural networks have achieved remarkable empirical success on large, high-dimensional datasets but lack dependable uncertainty estimation. Although Bayesian neural networks (BNNs) provide a conceptually elegant solution, it is fundamentally difficult to aggregate priors on large quantities of parameters into collectively meaningful properties of the model they define. This dissertation presents GPs and BNNs under a common framework, highlighting similarities and exposing failures of each. Chapter 1 focuses on the consequences of misspecifying a kernel, the key choice in a GP model, and illustrates how a BNN can overcome this misspecification. Next, chapter 2 explains how to encode a common property of a GP kernel (stationarity) in a BNN. Finally, chapter 3 proves a critical failure of overparameterized BNNs under approximate inference.
Cited by 0Open access
Zhao, Hu · RWTH Publications (RWTH Aachen) · 2021
Landslides are common natural hazards occurring around the world. They pose an ongoing threat to lives, properties, and environment. Driven by the practical need to predict hazards of future landslides and design mitigation strategies, various physics-based landslide run-out models have been developed in the past decades. To achieve reliable and transparent simulation-based risk assessment and mitigation design, comprehensive understanding of the various uncertainties associated with these models is required. However, advanced statistical methods that are capable of properly addressing the uncertainties are often not applicable due to the computational bottleneck resulting from the relatively long run time of a single simulation and the large number of necessary simulations. To address the research gap, new methodologies are developed and studied in this thesis. They make up a unified framework that allows us to systematically, routinely, and efficiently investigate both forward and inverse problems resulting from the various uncertainties. Chapter 1 introduces the background, frames the research gap, and motivates this study. Chapter 2 and 3 present theories of the two essential components of the unified framework, namely physics-based landslide run-out models and data-driven Gaussian process emulators. Chapter 4 presents a new methodology for efficient variance-based global sensitivity analyses of landslide run-out models. The methodology couples depth-averaged landslide run-out models, variance-based sensitivity analyses, robust multivariate Gaussian process emulation techniques, and an algorithm accounting for the emulator-uncertainty. Its feasibility and efficiency are validated by a case study based on the 2017 Bondo landslide event. The results show that it can recover common findings in the literature and provides further information on interactions between input variables along the full flow path. Chapter 5 presents a new methodology for efficient parameter calibration of landslide run-out models. It is developed by integrating depth-averaged landslide run-out models, Bayesian inference, Gaussian process emulation, and active learning. A case study using the new method is conducted based on the 2017 Bondo landslide event with synthetic observed data. The results show that the method is capable of correctly calibrating the rheological parameters and greatly improving the computational efficiency. Chapter 6 is devoted to uncertainty quantification of landslide run-out models. The focus is put on topographic uncertainty which is mostly overlooked in current practice. Two types of geostatistical methods are used to study the impact of topographic uncertainty on landslide run-out modeling based on the 2008 Yu Tung landslide event. It is found that topographic uncertainty significantly affects landslide run-out modeling, depending on how well the underlying flow path is represented. In addition, the close relation between the two geostatistical methods and Gaussian processes is revealed. Based on it, a new method that employs Karhunen–Loeve expansion to reduce the dimensionality of topographic uncertainty is proposed. It has great potentials to make Gaussian process emulation also applicable for high-dimensional topographic uncertainty and therefore allows us to treat topographic uncertainty within the unified framework. Chapter 7 provides concluding remarks and recommendations for future work.
Cited by 0Open access
Majdi Fanous, Omid Chatrabgoun, Mohsen Esmaeilbeigi et al. · Research Square · 2023
Abstract Uncertainty quantification (UQ) and propagation (UP) in complex multi-scale physical models are challenging problems due to the high-dimensionality of the inputs and the outputs for such models and the corresponding computational complexity. These challenges could be tackled using Gaussian process (GP) models by constructing a surrogate response surface. This response surface is an efficient approximation of the underlying expensive model which can be then used for UQ and UP. However, the standard GP model is hampered by various issues such as the curse of dimensionality, difficulties in capturing the non-Gaussian complex nature of the input/output relation, and modelling discontinuities of 1 model outputs. To overcome these issues, this work aims to use a non-linear GP model, known as deep Gaussian process for the UQ and UP of multi-scale models. Deep GP is an efficient approach for deep learning and modelling of high-dimensional complex systems and consists of several hidden layers connected with non-linear mappings. In this work, a modified variational approximation framework is presented to analytically approximate the posterior distribution of the model outputs using the model inputs. The proposed framework automatically selects the dimensionality of each hidden layer to prevent over-fitting. Consequently, the deep GP model is more flexible and can be used to propagate uncertainty obtained in each layer across the hierarchy. The suitability of this model is shown in UQ tasks by computing the error bars for the statistics of interest in a standard stochastic PDE based model.
Cited by 0Open access
Gracielle Antunes de Araújo, Flávio B. Gonçalves · ArXiv.org · 2026
In this work, we study scaling limits of shallow Bayesian neural networks (BNNs) via their connection to Gaussian processes (GPs), with an emphasis on statistical modeling, identifiability, and scalable inference. We first establish a general convergence result from BNNs to GPs by relaxing assumptions used in prior formulations, and we compare alternative parameterizations of the limiting GP model. Building on this theory, we propose a new covariance function defined as a convex mixture of components induced by four widely used activation functions, and we characterize key properties including positive definiteness and both strict and practical identifiability under different input designs. For computation, we develop a scalable maximum a posterior (MAP) training and prediction procedure using a Nyström approximation, and we show how the Nyström rank and anchor selection control the cost-accuracy trade-off. Experiments on controlled simulations and real-world tabular datasets demonstrate stable hyperparameter estimates and competitive predictive performance at realistic computational cost.
Cited by 0Open access
Gracielle Antunes de Araújo, Flávio B. Gonçalves · arXiv (Cornell University) · 2026
In this work, we study scaling limits of shallow Bayesian neural networks (BNNs) via their connection to Gaussian processes (GPs), with an emphasis on statistical modeling, identifiability, and scalable inference. We first establish a general convergence result from BNNs to GPs by relaxing assumptions used in prior formulations, and we compare alternative parameterizations of the limiting GP model. Building on this theory, we propose a new covariance function defined as a convex mixture of components induced by four widely used activation functions, and we characterize key properties including positive definiteness and both strict and practical identifiability under different input designs. For computation, we develop a scalable maximum a posterior (MAP) training and prediction procedure using a Nyström approximation, and we show how the Nyström rank and anchor selection control the cost-accuracy trade-off. Experiments on controlled simulations and real-world tabular datasets demonstrate stable hyperparameter estimates and competitive predictive performance at realistic computational cost.
Cited by 0Open access
Rare & Special e-Zone (The Hong Kong University of Science and Technology) · 2018
Elsevier’s Scopus, the largest abstract and citation database of peer-reviewed literature. Search and access research from the science, technology, medicine, social sciences and arts and humanities fields.
Cited by 0
Amirreza Kachabi, Sofia Altieri Correa, Naomi C. Chesler et al. · Computers in Biology and Medicine · 2025
Subject-specific modeling is a powerful tool in cardiovascular research, providing insights beyond the reach of current clinical diagnostics. Limitations in available clinical data require the incorporation of uncertainty into models to improve guidance for personalized treatments. However, for clinical relevance, such modeling must be computationally efficient. In this study, we used a one-dimensional (1D) fluid dynamics model informed by experimental data from a dog model of chronic thromboembolic pulmonary hypertension (CTEPH), incorporating measurements from multiple subjects under both baseline and CTEPH conditions. Surgical intervention can alleviate CTEPH, yet patients with microvascular disease (e.g., remodeling and narrowing of small vessels) often exhibit persistent pulmonary hypertension, highlighting the importance of assessing microvascular disease severity. Thus, each lung was modeled separately to account for the heterogeneous nature of CTEPH, allowing us to explore lung-specific microvascular narrowing and resistance. We compared inferred parameters between baseline and CTEPH and examined their correlation with clinical markers of disease severity. To accelerate model calibration, we employed Gaussian process (GP) emulators, enabling the estimation of microvascular parameters and their uncertainties within a clinically feasible timeframe. Our results demonstrated that CTEPH leads to heterogeneous microvascular adaptation, reflected in distinct parameter shifts. Notably, the changes in model parameters strongly correlated with disease severity, especially in the lung previously reported to have more advanced disease. This framework provides a rapid, uncertainty-aware method for evaluating microvascular dysfunction in CTEPH and may support more targeted treatment strategies within a timeframe suitable for clinical application.
Cited by 2Open access
Taylor, Benjamin M., Davies, Tilman M., Rowlingson, Barry S. et al. · Lancaster EPrints (Lancaster University) · 2015
Log-Gaussian Cox processes are an important class of models for spatial and spatiotemporal point-pattern data. Delivering robust Bayesian inference for this class of models presents a substantial challenge, since Markov chain Monte Carlo (MCMC) algorithms require careful tuning in order to work well. To address this issue, we describe recent advances in MCMC methods for these models and their implementation in the R package lgcp. Our suite of R functions provides an extensible framework for inferring covariate effects as well as the parameters of the latent field. We also present methods for Bayesian inference in two further classes of model based on the log-Gaussian Cox process. The first of these concerns the case where we wish to fit a point process model to data consisting of event-counts aggregated to a set of spatial regions: we demonstrate how this can be achieved using data-augmentation. The second concerns Bayesian inference for a class of marked-point processes specified via a multivariate log-Gaussian Cox process model. For both of these extensions, we give details of their implementation in R.
Cited by 2Open access
Takayuki Nakachi, Yitu Wang · 2024
In this paper, we propose a privacy-preserving Bayesian inference method for deep Gaussian processes (DGPs). By hierarchically combining Gaussian processes (GPs) in DGPs, a more expressive model can be constructed. The proposed method enables Bayesian inference for DGPs while keeping the input data scrambled through a random unitary transform (RUT). Our Bayesian inference on scrambled data performs equivalently to Bayesian inference on non-scrambled data. Furthermore, we propose an access control method for prediction by changing the private keys of the RUT for the training and new inputs. Finally, we demonstrate the effectiveness of our method through experiments with diabetes data from the medical analysis field.
Cited by 0
Daniel Turek, Mark Risser · 2019
Enables off-the-shelf functionality for fully Bayesian, nonstationary Gaussian process modeling. The approach to nonstationary modeling involves a closed-form, convolution-based covariance function with spatially-varying parameters; these parameter processes can be specified either deterministically (using covariates or basis functions) or stochastically (using approximate Gaussian processes). Stationary Gaussian processes are a special case of our methodology, and we furthermore implement approximate Gaussian process inference to account for very large spatial data sets (Finley, et al (2017) <<a href="https://doi.org/10.48550%2FarXiv.1702.00434" target="_top">doi:10.48550/arXiv.1702.00434</a>>). Bayesian inference is carried out using Markov chain Monte Carlo methods via the "nimble" package, and posterior prediction for the Gaussian process at unobserved locations is provided as a post-processing step.
Cited by 3Open access
Härkönen, Teemu · LUTPub (LUT University) · 2024
In this thesis, we introduce stochastic processes and Bayesian inference methods with a particular emphasis on spectroscopic applications. First, we apply sequential Monte Carlo samplers for estimating the modulating error function and line shape parameters that are relevant for coherent anti-Stokes Raman scattering spectroscopy. Second, we model posterior distributions of deconvoluted spectroscopic data as log-Gaussian Cox processes. Furthermore, we estimate the number of line shapes and their locations via sampling local maxima of the log-intensity of the log-Gaussian Cox process. Third, we propose nested sequential Monte Carlo samplers as an inference tool for mixtures of Gaussian process experts. The nested sequential Monte Carlo samplers allow for unbiased and more efficient inference in terms of Kullback–Leibler divergence while being thoroughly amenable to parallelisation. Lastly, we propose a new type of stochastic process, the log-Gaussian gamma process. We generate synthetic coherent anti-Stokes Raman scattering spectra with a combination of Gaussian and log-Gaussian gamma processes. We use the synthetic spectra to train a partially-Bayesian neural network for the inverse problem of estimating an underlying Raman spectrum from coherent anti-Stokes Raman scattering spectra. The Bayesian paradigm offers explicit and robust uncertainty quantification, an aspect that has seen little attention in spectroscopic analysis. All in all, we propose Bayesian statistical methods for spectroscopic analysis while extending existing statistics literature with the proposal of nested sequential Monte Carlo samplers for mixtures of Gaussian process experts and log-Gaussian gamma processes.
Cited by 0
Malem-Shinitski, Noa, Ojeda, César, Opper, Manfred · DepositOnce · 2022
Traditionally, Hawkes processes are used to model time-continuous point processes with history dependence. Here, we propose an extended model where the self-effects are of both excitatory and inhibitory types and follow a Gaussian Process. Whereas previous work either relies on a less flexible parameterization of the model, or requires a large amount of data, our formulation allows for both a flexible model and learning when data are scarce. We continue the line of work of Bayesian inference for Hawkes processes, and derive an inference algorithm by performing inference on an aggregated sum of Gaussian Processes. Approximate Bayesian inference is achieved via data augmentation, and we describe a mean-field variational inference approach to learn the model parameters. To demonstrate the flexibility of the model we apply our methodology on data from different domains and compare it to previously reported results.
Cited by 0Open access
Rui Meng · eScholarship (California Digital Library) · 2020
Temporal data modeling plays a vital role in various research including finance, environmental science and neuroscience. Understanding and interpreting the evolutionary system behind temporal data is of interest. This work mainly emphasizes efficient statistical models on temporal data via stochastic processes. In particular, we focus on statistical modeling via two flexible random processes: Markov processes and Gaussian processes. The first stage of the research involves a novel hidden Markov model based on Markov jump processes for cervical cancer screening test data. This model is able to model the heterogeneity of both individual and time. We provide an efficient and scalable expectation maximization based inference approach. To the best of our knowledge, our model is the first statistical model that is able to scale to a population-level dataset.Next, we consider an alternative stochastic process that is widely applied in temporal data modeling, the Gaussian process. Motivated by the kernel perspective of inducing-point based sparse Gaussian processes, we propose a general regularization framework of sparse Gaussian processes and extend it into latent variable models. We specifically consider variational inference under our regularization framework in various settings. We theoretically demonstrate that the variational inference under our regularization can be treated by maximizing a log likelihood lower bound on a corresponding empirical Bayesian model. Our framework is illustrated in various settings throughout both synthetic and real datasets. Building on our proposed regularization framework, we develop a hierarchical sparse latent Gaussian process model specifically for categorical data and then we extend our model to temporal data via dynamical priors. In particular, we propose efficient variational inference to make it applicable to large datasets. Moreover, our model provides a visualization way to summarize the dynamics of categorical data into a low-dimension manifold.The fourth project is motivated by electronic health record data and spatially varying coefficient linear coregionalization model. We propose a novel nonstationary multivariate Gaussian process model that allows it to model time dependent smoothness, scale and correlation across different dimensions. One special case of our model is emphasized due to its computational efficiency. It allows an efficient inference via Kronecker algebra. Moreover, we provide both Hamiltonian Monte Carlo inference as a fully Bayesian inference and Maximum a Posteriori based inference as an approximate Bayesian inference. Our posterior inference provides a promising way to understand the relation between cross-correlation of clinical variables and health status, which contributes to early disease detection.
Cited by 0Open access
Alina Bazarova, Jose Ignacio Robledo · JuSER (Forschungszentrum Jülich) · 2025
This course is the continuation of the course “Introduction to Bayesian Statistical Learning”. Although, participation in the latter is not strictly necessary to understand the material of this one, preliminary knowledge in Bayesian modelling, as well as in machine learning and artificial intelligence is a pre-requisite.The course consists of three parts. The first topic, normalizing flows, explores a class of generative models that facilitate likelihood-free inference. The second topic, diffusion models, introduces students to a powerful class of generative models that excel in modeling sequential data, as well as how they are related to Bayesian framework. The third topic, Gaussian processes, is a versatile tool for Bayesian inference and non-parametric modeling. Gaussian processes provide a flexible framework for modeling complex relationships between variables without assuming a specific functional form.The main topics are:Normalizing flowsDiffusion modelsGaussian ProcessesRunning models on a Supercomputer
Cited by 0
Räsänen, Elias · LUTPub (LUT University) · 2021
Gaussian processes provide a powerful Bayesian approach to many machine learning tasks. Unfortunately, their application has been limited by the cubic computational complexity of inference. Mixtures of Gaussian processes have been used to lower the computational costs and to enable inference on more complex data sets. In this thesis, we investigate a certain finite Gaussian process mixture model and its applicability to clustering and prediction tasks. We apply the mixture model on a multidimensional data set that contains multiple groups. We perform Bayesian inference on the model using Markov chain Monte Carlo. We find the predictive performance of the model satisfactory. Both the variances and the trends of the groups are estimated well, bar the issues caused by poor clustering. The model is unable to cluster some of the groups properly and we suggest improving the prior of the mixing proportions or incorporating more prior information as remedies for the issues in clustering.
Cited by 0Open access
Sébastien Marmin, Maurizio Filippone · arXiv (Cornell University) · 2018
Bayesian calibration of black-box computer models offers an established\nframework to obtain a posterior distribution over model parameters. Traditional\nBayesian calibration involves the emulation of the computer model and an\nadditive model discrepancy term using Gaussian processes; inference is then\ncarried out using MCMC. These choices pose computational and statistical\nchallenges and limitations, which we overcome by proposing the use of\napproximate Deep Gaussian processes and variational inference techniques. The\nresult is a practical and scalable framework for calibration, which obtains\ncompetitive performance compared to the state-of-the-art.\n
Cited by 2Open access
Semenova, Elizaveta · Zenodo (CERN European Organization for Nuclear Research) · 2024
This document contains lecture notes of the course "Bayesian Modelling and Probabilistic Programming with Numpyro and examples from Epidemiology". The course covers such topics as Bayesian inference, hierarchical modelling, Gaussian processes for spatial statistics and ordinary differential equations for disease transmission modelling. Readers will learn to construct probabilistic models and perform inference using a probabilistic programming language Numpyro in a fully Bayesian manner to characterise uncertainty of the modelled quantities. Although the course is primarily computational in nature, the models explored are grounded in epidemiological modelling practices.
Cited by 0Open access
Alexander Immer · arXiv (Cornell University) · 2020
In this thesis, we disentangle the generalized Gauss-Newton and approximate inference for Bayesian deep learning. The generalized Gauss-Newton method is an optimization method that is used in several popular Bayesian deep learning algorithms. Algorithms that combine the Gauss-Newton method with the Laplace and Gaussian variational approximation have recently led to state-of-the-art results in Bayesian deep learning. While the Laplace and Gaussian variational approximation have been studied extensively, their interplay with the Gauss-Newton method remains unclear. Recent criticism of priors and posterior approximations in Bayesian deep learning further urges the need for a deeper understanding of practical algorithms. The individual analysis of the Gauss-Newton method and Laplace and Gaussian variational approximations for neural networks provides both theoretical insight and new practical algorithms. We find that the Gauss-Newton method simplifies the underlying probabilistic model significantly. In particular, the combination of the Gauss-Newton method with approximate inference can be cast as inference in a linear or Gaussian process model. The Laplace and Gaussian variational approximation can subsequently provide a posterior approximation to these simplified models. This new disentangled understanding of recent Bayesian deep learning algorithms also leads to new methods: first, the connection to Gaussian processes enables new function-space inference algorithms. Second, we present a marginal likelihood approximation of the underlying probabilistic model to tune neural network hyperparameters. Finally, the identified underlying models lead to different methods to compute predictive distributions. In fact, we find that these prediction methods for Bayesian neural networks often work better than the default choice and solve a common issue with the Laplace approximation.
Cited by 2Open access