Xuechun Qiao, Yasen Wang · 2022
While sparse Bayesian learning (SBL) has attracted a great amount of attention in a wide range of areas, its practical utility is limited due to the high computational cost. To address this issue, we propose a recursive SBL algorithm in this paper, which reduces the computational complexity per iteration from $\mathcal{O}\left( {{n^3}} \right)$ to $\mathcal{O}\left( {{n^2}} \right)$. First, based on the Kalman filter, we derive the posterior distribution of model parameters recursively by dealing with the input-output data in turn. More specifically, the posterior distribution of model parameters of the previous iteration will be regarded as its prior distribution of the current iteration and then combined with the data likelihood to derive its posterior distribution of the current iteration, resulting in a Kalman filter like updating. Second, for unknown hyperparameters that were introduced for imposing the sparsity-promoting prior on model parameters, we also develop a computationally efficient method to estimate their values. Lastly, we demonstrate the effectiveness of our method on several numerical experiments. Compared with other methods, our method has a competitive performance with a very low computational cost.
Cited by 0
Lee, Minwoo, author, Anderson, Charles W., advisor, Ben-Hur, Asa, committee member et al. · Digital Collections of Colorado (Colorado State University) · 2017
SPARSE BAYESIAN REINFORCEMENT LEARNINGThis dissertation presents knowledge acquisition and retention methods for efficient and robust learning.We propose a framework for learning and memorizing, and we examine how we can use the memory for efficient machine learning.Temporal difference (TD) learning is a core part of reinforcement learning, and it requires function approximation.However, with function approximation, the most popular TD methods such as TD(), SARSA, and Q-learning lose stability and diverge especially when the complexity of the problem grows and the sampling distribution is biased.The biased samples cause function approximators such as neural networks to respond quickly to the new data by losing what was previously learned.Systematically selecting a most significant experience, our proposed approach gradually stores the snapshot memory.The memorized snapshots prevent forgetting important samples and increase learning stability.Our sparse Bayesian learning model maintains the sparse snapshot memory for efficiency in computation and memory.The Bayesian model extends and improves TD learning by utilizing the state information in hyperparameters for smart decision of action selection and filtering insignificant experience to maintain sparsity of snapshots for efficiency.The obtained memory can be used to further improve learning.First, the placement of the snapshot memories with a radial basis function kernel located at peaks of the value function approximation surface leads to an efficient way to search a continuous action space for practical application with fine motor control.Second, the memory is a knowledge representation for transfer learning.Transfer learning is a paradigm for knowledge generalization of machine learning and reinforcement learning.Transfer learning shortens the time for machine learning training by using the knowledge gained from similar tasks.The dissertation examines a practice approach that transfers the snapshots from non-goal-directive random movements to goal-driven reinforcement 4.3
Cited by 2Open access
David Wipf, Jason Palmer, Bhaskar Rao · 2003
Recently, relevance vector machines (RVM) have been fashioned from a sparse Bayesian learning (SBL) framework to perform supervised learning using a weight prior that encourages sparsity of representation. The methodology incorporates an additional set of hyperparameters governing the prior, one for each weight, and then adopts a specific approximation to the full marginalization over all weights and hyperparameters. Despite its empirical success however, no rigorous motivation for this particular approximation is currently available. To address this issue, we demonstrate that SBL can be recast as the application of a rigorous variational approximation to the full model by expressing the prior in a dual form. This formulation obviates the necessity of assuming any hyperpriors and leads to natural, intuitive explanations of why sparsity is achieved in practice. 1
Cited by 0
Cédric Archambeau, Shengbo Guo, Onno Zoeter · 2011
We propose a new sparse Bayesian model for multi-task regression and classification. The model is able to capture correlations between tasks, or more specifically a low-rank approximation of the covariance matrix, while being sparse in the features. We introduce a general family of group sparsity inducing priors based on matrix-variate Gaussian scale mixtures. We show the amount of sparsity can be learnt from the data by combining an approximate inference approach with type II maximum likelihood estimation of the hyperparameters. Empirical evaluations on data sets from biology and vision demonstrate the applicability of the model, where on both regression and classification tasks it achieves competitive predictive performance compared to previously proposed methods. 1
Cited by 0
Lin, Alexander, Song, Andrew H., Bilgic, Berkin et al. · arXiv (Cornell University) · 2021
Sparse Bayesian learning (SBL) is a powerful framework for tackling the sparse coding problem while also providing uncertainty quantification. The most popular inference algorithms for SBL exhibit prohibitively large computational costs for high-dimensional problems due to the need to maintain a large covariance matrix. To resolve this issue, we introduce a new method for accelerating SBL inference -- named covariance-free expectation maximization (CoFEM) -- that avoids explicit computation of the covariance matrix. CoFEM solves multiple linear systems to obtain unbiased estimates of the posterior statistics needed by SBL. This is accomplished by exploiting innovations from numerical linear algebra such as preconditioned conjugate gradient and a little-known diagonal estimation rule. For a large class of compressed sensing matrices, we provide theoretical justifications for why our method scales well in high-dimensional settings. Through simulations, we show that CoFEM can be up to thousands of times faster than existing baselines without sacrificing coding accuracy. Through applications to calcium imaging deconvolution and multi-contrast MRI reconstruction, we show that CoFEM enables SBL to tractably tackle high-dimensional sparse coding problems of practical interest.
Cited by 0Open access
Daniel D. Lee, Yuanqing Lin · 2006
endorsement of any of the University of Pennsylvania's products or services. Internal or personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by writing to pubs-permissions@ieee.org. By choosing to view this document, you agree to all provisions of the copyright laws protecting it. This paper is posted at ScholarlyCommons.
Cited by 0
Igor Fedorov, Bhaskar D. Rao · arXiv (Cornell University) · 2018
This paper addresses the problem of learning dictionaries for multimodal datasets, i.e. datasets collected from multiple data sources. We present an algorithm called multimodal sparse Bayesian dictionary learning (MSBDL). MSBDL leverages information from all available data modalities through a joint sparsity constraint. The underlying framework offers a considerable amount of flexibility to practitioners and addresses many of the shortcomings of existing multimodal dictionary learning approaches. In particular, the procedure includes the automatic tuning of hyperparameters and is unique in that it allows the dictionaries for each data modality to have different cardinality, a significant feature in cases when the dimensionality of data differs across modalities. MSBDL is scalable and can be used in supervised learning settings. Theoretical results relating to the convergence of MSBDL are presented and the numerical results provide evidence of the superior performance of MSBDL on synthetic and real datasets compared to existing methods.
Cited by 4Open access
Ritwik Giri, Bhaskar D. Rao · 2014 48th Asilomar Conference on Signals, Systems and Computers · 2014
In this article we study the sparse signal recovery problem in a Bayesian framework using a novel Bootstrapped Sparse Bayesian Learning method. Sparse Bayesian Learning (SBL) framework is an effective tool for pruning out the irrelevant features and ending up with a sparse representation. In SBL the choice of prior over the variances of the Gaussian Scale mixture has been an interesting area of research for some time now. This motivates us to use a more generalized maximum entropy density as the prior which results in a new variant of SBL. It has been shown to perform better than traditional SBL empirically and it also accelerates the pruning procedure. Because of this advantage, this variant of SBL can be claimed as more robust choice as it is less sensitive to the threshold for pruning. Theoretical justifications have also been provided to show that the proposed model actually promotes sparse point estimates.
Cited by 4
Jakob Möderl, Erik Leitinger, Bernard H. Fleury et al. · IEEE Transactions on Signal Processing · 2025
We propose a variational Bayesian (VB) implementation of block-sparse Bayesian learning (BSBL) to compute proxy probability density functions (PDFs) that approximate the posterior PDFs of the weights and associated hyperparameters in a block-sparse linear model, resulting in an iterative algorithm coined variational BSBL (VA-BSBL). The priors of the hyper-parameters are selected to belong to the family of generalized inverse Gaussian distributions. This family contains as special cases commonly used hyperpriors such as the Gamma and inverse Gamma distributions, as well as Jeffreyߣs improper distribution. <p xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Inspired by previous work on classical sparse Bayesian learning (SBL), we investigate the update stage in which the proxy PDFs of a single block of weights and of its associated hyperparameter are successively updated, while keeping the proxy PDFs of the other parameters fixed. This stage defines a nonlinear first-order recurrence relation for the mean of the proxy PDF of the hyperparameter. By iterating this relation “ad infinitum” we obtain a criterion that determines whether the so-generated sequence of hyperparameter means converges or diverges. Incorporating this criterion into the VA-BSBL algorithm yields a fast implementation, coined fast-BSBL (F-BSBL), which achieves a two-order-of-magnitude runtime improvement. <p xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">We further identify the range of the parameters of the generalized inverse Gaussian distribution which result in an inherent pruning procedure that switches off “weak” components in the model, which is necessary to obtain sparse results. Lastly, we show that expectation-maximization (EM)-based and VB-based implementations of BSBL are identical methods. Thus, we extend a well-known result from classical SBL to BSBL. Consequently, F-BSBL and BSBL using coordinate ascent to maximize the marginal likelihood coincide. These results provide a unified framework for interpreting existing BSBL methods.
Cited by 3Open access
Peter Gerstoft, Christoph F. Mecklenbräuker, Angeliki Xenaki et al. · Technical University of Denmark, DTU Orbit (Technical University of Denmark, DTU) · 2016
The directions of arrival (DOA) of plane waves are estimated from multisnapshot sensor array data using sparse Bayesian learning (SBL). The prior for the source amplitudes is assumed independent zero-mean complex Gaussian distributed with hyperparameters, the unknown variances (i.e., the source powers). For a complex Gaussian likelihood with hyperparameter, the unknown noise variance, the corresponding Gaussian posterior distribution is derived. The hyperparameters are automatically selected by maximizing the evidence and promoting sparse DOA estimates. The SBL scheme for DOA estimation is discussed and evaluated competitively against LASSO (l(1)-regularization), conventional beamforming, and MUSIC.
Cited by 239
Jakob Möderl · arXiv (Cornell University) · 2023
Fast Variational Block-Sparse Bayesian Learning This repository contains the matlab implementation of the fast variational block-sparse Bayesian learning algorithm (FV-BSBL) described in [1]. Repository Structure - fastBSBL/: Folder containing the algorithm described in [1] - fastBSBL/fastBSBL.m: main file of the algorithm - BSBL_public/: Folder containing the comparison algorithms BSBL-EM and BSBL-BO described in [2], copied from http://dsp.ucsd.edu/~zhilin/BSBL.html - SBL_MV_matlab/: Folder containing the DOA-SBL comparison algorithm decsribed in [3], copied from https://github.com/gerstoft/SBL - testFastBSBL.m: Demo script that generates a signal based on a Gaussian dictionary and blocks-sparse weights and applies the algorithms [1] and [2]. - application_DoA.m: Demo script how to apply the algorithm to Direction-of-Arrival (DOA) estimation and compares the performance to [3] - plotThresholding.m: Script to generate [1, Figure 1]. - ospa_dist.m: function to calculate the OSPA distance, needed for application_DoA.m - munkres.m: auxilary function required for ospa_dist.m References [1] J. Möderl, F. Pernkopf, K. Witrisal and E. Leitinger, "Fast Variational Block-Sparse Bayesian Learning", arXiv:2306.00442, 2023. [2] Z. Zhang and B. D. Rao, "Extension of SBL algorithms for the recovery of block sparse signals with intra-block correlation," IEEE Trans. Signal Process., vol. 61, no. 8, pp. 2009–2015, Apr. 2013. [3] P. Gerstoft, C. F. Mecklenbräuker, A. Xenaki, and S. Nannuru, "Multisnapshot sparse Bayesian learning for DOA,” IEEE Signal Process. Lett., vol. 23, no. 10, pp. 1469–1473, Oct. 2016.
Cited by 2Open access
Benyuan Liu, Hongqi Fan, Zaiqi Lu et al. · arXiv (Cornell University) · 2012
In this paper we propose a two-level hierarchical Bayesian model and an annealing schedule to re-enable the noise variance learning capability of the fast marginalized Sparse Bayesian Learning Algorithms. The performance such as NMSE and F-measure can be greatly improved due to the annealing technique. This algorithm tends to produce the most sparse solution under moderate SNR scenarios and can outperform most concurrent SBL algorithms while pertains small computational load.
Cited by 0Open access
Joaquin Qui nonero-Candela, Edward Snelson, Oliver Williams · 2007
Sparse Bayesian learning suffers from impractical, overconfident predictions where the uncertainty tends to be maximal around the observations. We propose an alternative treatment that breaks the rigidity of the implied prior through decorrelation, and consequently gives reasonable and intuitive error bars. The attractive computational efficiency is retained; learning leads to sparse solutions. An interesting by-product is the ability to model non-stationarity and input-dependent noise. 1 Sparse Bayesian learning Finite linear regression models are attractive for computational reasons and because they are easily interpreted. In these models, the regression function is simply a weighted linear sum of M basis functions φ1(x), . . . , φM(x):
Cited by 5
Santosh Nannuru, Kay L. Gemba, Peter Gerstoft · 2017
Sparse Bayesian learning (SBL) employs Gaussian priors with unknown parameters to solve an underdetermined system of linear equation. It provides comparable performance and is significantly faster than convex optimization techniques used in sparse processing. In this paper we extend SBL to process observations from multiple dictionaries when the sparse solutions have common support across dictionaries. Two solutions are presented, a multiple covariance formulation and a common covariance formulation. As an example, the multi-dictionary approach is used to estimate the direction-of-arrivals in presence of aliasing. Simulations and data from the SwellEx-96 experiment are used to demonstrate qualitatively the advantages of multi-dictionary SBL.
Cited by 14
Wen-hua Ju, David Madigan, Steven L. Scott · 2002
Figueiredo (2001) and Figueiredo and Jain (2001) described a particular sparseness- inducing Bayesian model for probit regression.
Cited by 0
Altena, Anique, Snellen, Mirjam, Luesutthiviboon, Salil et al. · Figshare · 2024
Over the past years, the use of acoustics for detection and localisation of hazardous drones has increased. A large number of studies focusing on this work use beamforming for this application. The aim of the research presented here is to investigate if a different technique can be used to improve the localisation range which can be achieved by beamforming. The technique used in this work is sparse Bayesian learning. Beamforming relies on a minimisation between measured and modelled data. Opposite to this, sparse Bayesian learning is a probabilistic parameter estimation technique. Both techniques are applied to real-world measurement data of fly-overs with 5 different quadcopters. The azimuth and elevation angles of the drones are estimated and the time interval in seconds is determined during which the drones are localised continuously. For both angles, sparse Bayesian learning clearly results in longer intervals. However, the exact improvement varies per drone. In azimuth direction, the intervals were 4.2% to 59.2% longer. In elevation direction, this was 1.2% to 40.4%.
Cited by 0Open access
Santosh Nannuru, Kay L. Gemba, Peter Gerstoft et al. · Signal Processing · 2019
Cited by 82Open access
Liwei Wang · OpenBU (Boston University) · 2023
2023
Cited by 0
Oisín Ruairi Boyle, Murat Üney, Xinping Yi et al. · SSRN Electronic Journal · 2024
Cited by 0Open access
Qiuyun Zhu · OpenBU (Boston University) · 2023
2023
Cited by 0