Ivana Tošić, Pascal Frossard · IEEE Signal Processing Magazine · 2011
We describe methods for learning dictionaries that are appropriate for the representation of given classes of signals and multisensor data. We further show that dimensionality reduction based on dictionary representation can be extended to address specific tasks such as data analy sis or classification when the learning includes a class separability criteria in the objective function. The benefits of dictionary learning clearly show that a proper understanding of causes underlying the sensed world is key to task-specific representation of relevant information in high-dimensional data sets.
Cited by 773Open access
Jean‐Luc Starck, Fionn Murtagh, Jalal Fadili · Cambridge University Press eBooks · 2015
A data set can be decomposed in many dictionaries, and we argue in this book that the “best” dictionary is the one providing the sparsest (most economical) representation. In practice, it is convenient to use dictionaries with a fast implicit transform (such as those described in detail in the previous chapters), which allows us to directly obtain the coefficients and reconstruct the signal from these coefficients using fast algorithms running in linear or almost linear time (unlike matrix-vector multiplications). We have also seen in Chapter 8 that fixed dictionaries can be gathered together in order to build a larger dictionary that can describe the data in a more versatile way. All these dictionaries are designed to handle specific contents and are restricted to signals and images that are of a certain type. For instance, Fourier represents stationary and periodic signals well, wavelets are good for analyzing isotropic objects of different scales, curvelets are designed for anisotropic and curvilinear features. Hence, the representation space that we use in our analysis can be seen as a prior we have on our data. Fixed dictionaries, though they have very fast implicit analysis and synthesis operators, which makes them attractive from a practical point of view, cannot guarantee sparse representations of new classes of signals of interest that present more complex patterns and features. What can one do if the data cannot be sufficiently sparsely represented by any of these fixed (or combined) existing dictionaries or if is not known the morphology of features contained in our data? Is there a way to make our data analysis more adaptive by optimizing for a dedicated dictionary? To answer these questions, a new field has emerged called Dictionary Learning (DL). Dictionary learning offers the possibility of learning an adaptive dictionary ɸ directly from the data (or from a set of exemplars that we believe represent the data well). DL is at the interface of machine learning and signal processing.
Cited by 0
Vishal M. Patel, Rama Chellappa · Springer briefs in electrical and computer engineering · 2013
Cited by 2
Angshul Majumdar · 2018
Cited by 0
Ali H. Sayed · Cambridge University Press eBooks · 2022
This extraordinary three-volume work, written in an engaging and rigorous style by a world authority in the field, provides an accessible, comprehensive introduction to the full spectrum of mathematical and statistical techniques underpinning contemporary methods in data-driven learning and inference. This final volume, Learning, builds on the foundational topics established in volume I to provide a thorough introduction to learning methods, addressing techniques such as least-squares methods, regularization, online learning, kernel methods, feedforward and recurrent neural networks, meta-learning, and adversarial attacks. A consistent structure and pedagogy is employed throughout this volume to reinforce student understanding, with over 350 end-of-chapter problems (including complete solutions for instructors), 280 figures, 100 solved examples, datasets and downloadable Matlab code. Supported by sister volumes Foundations and Inference, and unique in its scale and depth, this textbook sequence is ideal for early-career researchers and graduate students across many courses in signal processing, machine learning, data and inference.
Cited by 0
BARLA, ANNALISA, SALZO, SAVERIO, VERRI, ALESSANDRO · CINECA IRIS Institutial Research Information System (University of Genoa) · 2014
Cited by 0
Jordan Frecon, Lucas Anquetil, Yuan Liu et al. · HAL (Le Centre pour la Communication Scientifique Directe) · 2022
To bridge the gap between specific and universal attacks on deep classification networks, the present work frames the learning of multiple adversarial attacks as linear combinations of atoms from a dictionary of universal attacks. In order to learn such adversarial dictionary, a non-convex proximal splitting framework, termed as Adversarial Dictionary Learning (ADiL) is proposed. Numerical experiments evidence that the posteriori study of the dictionary atoms unveils the most common patterns to attack the classifier which, in turn, can be used to craft adversarial perturbations to new examples achieving great transferability on different deep network architectures.
Cited by 0Open access
Enrico Grimaldi, Claudio Battiloro, Paolo Di Lorenzo · IEEE Transactions on Signal Processing · 2025
The aim of this paper is to introduce a novel dictionary learning algorithm for sparse representation of signals defined over combinatorial topological spaces, specifically, regular cell complexes. Leveraging Hodge theory, we embed topology into the dictionary structure via concatenated sub-dictionaries, each as a polynomial of Hodge Laplacians, yielding localized spectral topological filter frames. The learning problem is cast to jointly infer the underlying cell complex and optimize the dictionary coefficients and the sparse signal representation. We efficiently solve the problem via iterative alternating algorithms. Numerical results on both synthetic and real data show the effectiveness of the proposed procedure in jointly learning the sparse representations and the underlying relational structure of topological signals.
Cited by 0
Simon Hawe, Matthias Seibert, Martin Kleinsteuber · arXiv (Cornell University) · 2013
Many techniques in computer vision, machine learning, and statistics rely on the fact that a signal of interest admits a sparse representation over some dictionary. Dictionaries are either available analytically, or can be learned from a suitable training set. While analytic dictionaries permit to capture the global structure of a signal and allow a fast implementation, learned dictionaries often perform better in applications as they are more adapted to the considered class of signals. In imagery, unfortunately, the numerical burden for (i) learning a dictionary and for (ii) employing the dictionary for reconstruction tasks only allows to deal with relatively small image patches that only capture local image information. The approach presented in this paper aims at overcoming these drawbacks by allowing a separable structure on the dictionary throughout the learning process. On the one hand, this permits larger patch-sizes for the learning phase, on the other hand, the dictionary is applied efficiently in reconstruction tasks. The learning procedure is based on optimizing over a product of spheres which updates the dictionary as a whole, thus enforces basic dictionary properties such as mutual coherence explicitly during the learning procedure. In the special case where no separable structure is enforced, our method competes with state-of-the-art dictionary learning methods like K-SVD.
Cited by 16Open access
Simon Hawe, Matthias Seibert, Martin Kleinsteuber · 2013
Many techniques in computer vision, machine learning, and statistics rely on the fact that a signal of interest admits a sparse representation over some dictionary. Dictionaries are either available analytically, or can be learned from a suitable training set. While analytic dictionaries permit to capture the global structure of a signal and allow a fast implementation, learned dictionaries often perform better in applications as they are more adapted to the considered class of signals. In imagery, unfortunately, the numerical burden for (i) learning a dictionary and for (ii) employing the dictionary for reconstruction tasks only allows to deal with relatively small image patches that only capture local image information. The approach presented in this paper aims at overcoming these drawbacks by allowing a separable structure on the dictionary throughout the learning process. On the one hand, this permits larger patch-sizes for the learning phase, on the other hand, the dictionary is applied efficiently in reconstruction tasks. The learning procedure is based on optimizing over a product of spheres which updates the dictionary as a whole, thus enforces basic dictionary properties such as mutual coherence explicitly during the learning procedure. In the special case where no separable structure is enforced, our method competes with state-of-the-art dictionary learning methods like K-SVD.
Cited by 100
Thème Cog, Julien Mairal, Francis Bach et al. · 2008
Cited by 0
Eva Grimaldi, Claudio Battiloro, Paolo Di Lorenzo · ArXiv.org · 2025
The aim of this paper is to introduce a novel dictionary learning algorithm for sparse representation of signals defined over combinatorial topological spaces, specifically, regular cell complexes. Leveraging Hodge theory, we embed topology into the dictionary structure via concatenated sub-dictionaries, each as a polynomial of Hodge Laplacians, yielding localized spectral topological filter frames. The learning problem is cast to jointly infer the underlying cell complex and optimize the dictionary coefficients and the sparse signal representation. We efficiently solve the problem via iterative alternating algorithms. Numerical results on both synthetic and real data show the effectiveness of the proposed procedure in jointly learning the sparse representations and the underlying relational structure of topological signals.
Cited by 0Open access
Amir Daneshmand, Gesualdo Scutari, Francisco Facchinei · 2016
The paper studies distributed Dictionary Learning (DL) problems where the learning task is distributed over a multi-agent network with time-varying (nonsymmetric) connectivity. This formulation is relevant, for instance, in Big Data scenarios where massive amounts of data are collected/stored in different spatial locations and it is unfeasible to aggregate and/or process all data in a fusion center, due to resource limitations, communication overhead or privacy considerations. We develop a general distributed algorithmic framework for the (nonconvex) DL problem and establish its asymptotic convergence. The new method hinges on Successive Convex Approximation (SCA) techniques coupled with i) a gradient tracking mechanism instrumental to locally estimate the missing global information; and ii) a consensus step, as a mechanism to distribute the computations among the agents. To the best of our knowledge, this is the first distributed algorithm with provable convergence for the DL problem and, more in general, bi-convex optimization problems over (time-varying) directed graphs.
Cited by 13
Bogdan Dumitrescu, Paul Irofti · 2018
Cited by 3
Sheng Li, Yun Fu · Advanced information and knowledge processing · 2017
Cited by 0
Mehrdad Yaghoobi, Thomas Blumensath, Michael Davies · 2009
Sparse modeling of signals has recently received a lot of attention. Often, a linear under-determined generative model for the signals of interest is proposed and a sparsity constraint imposed on the representation. When the generative model is not given, choosing an appropriate generative model is important, so that the given class of signals has approximate sparse representations. In this paper we introduce a new scheme for dictionary learning and impose an additional constraint to reduce the dictionary size. Small dictionaries are desired for coding applications and more likely to ldquoworkrdquo with suboptimal algorithms such as Basis Pursuit. Another benefit of small dictionaries is their faster implementation, e.g. a reduced number of multiplication/addition in each matrix vector multiplication, which is the bottleneck in sparse approximation algorithms.
Cited by 5
Amir Daneshmand, Scutari Gesualdo, Francisco Facchinei · arXiv (Cornell University) · 2016
The paper studies distributed Dictionary Learning (DL) problems where the learning task is distributed over a multi-agent network with time-varying (nonsymmetric) connectivity. This formulation is relevant, for instance, in big-data scenarios where massive amounts of data are collected/stored in different spatial locations and it is unfeasible to aggregate and/or process all the data in a fusion center, due to resource limitations, communication overhead or privacy considerations. We develop a general distributed algorithmic framework for the (nonconvex) DL problem and establish its asymptotic convergence. The new method hinges on Successive Convex Approximation (SCA) techniques coupled with i) a gradient tracking mechanism instrumental to locally estimate the missing global information; and ii) a consensus step, as a mechanism to distribute the computations among the agents. To the best of our knowledge, this is the first distributed algorithm with provable convergence for the DL problem and, more in general, bi-convex optimization problems over (time-varying) directed graphs.
Cited by 0Open access
Michaël Allouche, Emmanuel Gobet, Clara Lage et al. · SPIRE - Sciences Po Institutional REpository · 2022
Rating Migration Matrix is a crux to assess credit risks. Modeling andpredicting these matrices are then an issue of great importance for riskmanagers in any financial institution. As a challenger to usual parametric modeling approaches, we propose a new structured dictionarylearning model with auto-regressive regularization that is able to meetkey expectations and constraints: small amount of data, fast evolutionin time of these matrices, economic interpretability of the calibratedmodel. To show the model applicability, we present a numerical testwith both synthetic and real data and a comparison study with thewidely used parametric Gaussian Copula model: it turns out that ournew approach based on dictionary learning significantly outperformsthe Gaussian Copula model. The source code and the data are available at https://github.com/michael-allouche/dictionary-learning-RMM.git for the sake of reproducibility of our research.
Cited by 0
Schmitz, Morgan, Heitz, Matthieu, Bonneel, Nicolas et al. · HAL (Le Centre pour la Communication Scientifique Directe) · 2018
Cited by 0
Morgan A. Schmitz, Matthieu Heitz, Nicolas Bonneel et al. · RePEc: Research Papers in Economics · 2017
This paper introduces a new nonlinear dictionary learning method for histograms in the probability simplex. The method leverages optimal transport theory, in the sense that our aim is to reconstruct histograms using so-called displacement interpolations (a.k.a. Wasserstein barycenters) between dictionary atoms; such atoms are themselves synthetic histograms in the probability simplex. Our method simultaneously estimates such atoms, and, for each datapoint, the vector of weights that can optimally reconstruct it as an optimal transport barycenter of such atoms. Our method is computationally tractable thanks to the addition of an entropic regularization to the usual optimal transportation problem, leading to an approximation scheme that is efficient, parallel and simple to differentiate. Both atoms and weights are learned using a gradient-based descent method. Gradients are obtained by automatic differentiation of the generalized Sinkhorn iterations that yield barycenters with entropic smoothing. Because of its formulation relying on Wasserstein barycenters instead of the usual matrix product between dictionary and codes, our method allows for nonlinear relationships between atoms and the reconstruction of input data. We illustrate its application in several different image processing settings.
Cited by 1Open access