2012
Linear models have enjoyed great success in structured prediction in NLP. While a lot of progress has been made on efficient training with several loss functions, the problem of endowing learners with a mechanism for feature selection is still unsolved. Common approaches employ ad hoc filtering or L1regularization; both ignore the structure of the feature space, preventing practicioners from encoding structural prior knowledge. We fill this gap by adopting regularizers that promote structured sparsity, along with efficient algorithms to handle them. Experiments on three tasks (chunking, entity recognition, and dependency parsing) show gains in performance, compactness, and model interpretability. 1
Cited by 0
André F. T. Martins, Noah A. Smith, Pedro M. Q. Aguiar et al. · Empirical Methods in Natural Language Processing · 2018
Linear models have enjoyed great success in structured prediction in NLP. While a lot of progress has been made on efficient training with several loss functions, the problem of endowing learners with a mechanism for feature selection is still unsolved. Common approaches employ ad hoc filtering or <em>L</em><sub>1</sub>- regularization; both ignore the structure of the feature space, preventing practicioners from encoding structural prior knowledge. We fill this gap by adopting regularizers that promote structured sparsity, along with efficient algorithms to handle them. Experiments on three tasks (chunking, entity recognition, and dependency parsing) show gains in performance, compactness, and model interpretability.
Cited by 68Open access
Alessandro Rudi, Carlo Ciliberto, Gian Maria Marconi et al. · arXiv (Cornell University) · 2018
Structured prediction provides a general framework to deal with supervised problems where the outputs have semantically rich structure. While classical approaches consider finite, albeit potentially huge, output spaces, in this paper we discuss how structured prediction can be extended to a continuous scenario. Specifically, we study a structured prediction approach to manifold valued regression. We characterize a class of problems for which the considered approach is statistically consistent and study how geometric optimization can be used to compute the corresponding estimator. Promising experimental results on both simulated and real data complete our study.
Cited by 1Open access
Nowozin, Sebastian, Gehler, Peter, Jancsary, Jeremy et al. · MPG.PuRe (Max Planck Society) · 2014
The goal of structured prediction is to build machine learning models that predict relational information that itself has structure, such as being composed of multiple interrelated parts. These models, which reflect prior knowledge, task-specific relations, and constraints, are used in fields including computer vision, speech recognition, natural language processing, and computational biology. They can carry out such tasks as predicting a natural language sentence, or segmenting an image into meaningful components. These models are expressive and powerful, but exact computation is often intractable. A broad research effort in recent years has aimed at designing structured prediction models and approximate inference and learning procedures that are computationally efficient. This volume offers an overview of this recent research in order to make the work accessible to a broader research community. The chapters, by leading researchers in the field, cover a range of topics, including research trends, the linear programming relaxation approach, innovations in probabilistic modeling, recent theoretical progress, and resource-aware learning.
Cited by 0
David J. Weiss, Benjamin Taskar, Taskar, Ben · arXiv (Cornell University) · 2012
Structured prediction tasks pose a fundamental trade-off between the need for model complexity to increase predictive power and the limited computational resources for inference in the exponentially-sized output spaces such models require. We formulate and develop the Structured Prediction Cascade architecture: a sequence of increasingly complex models that progressively filter the space of possible outputs. The key principle of our approach is that each model in the cascade is optimized to accurately filter and refine the structured output state space of the next model, speeding up both learning and inference in the next layer of the cascade. We learn cascades by optimizing a novel convex loss function that controls the trade-off between the filtering efficiency and the accuracy of the cascade, and provide generalization bounds for both accuracy and efficiency. We also extend our approach to intractable models using tree-decomposition ensembles, and provide algorithms and theory for this setting. We evaluate our approach on several large-scale problems, achieving state-of-the-art performance in handwriting recognition and human pose recognition. We find that structured prediction cascades allow tremendous speedups and the use of previously intractable features and models in both settings.
Cited by 84Open access
Carlo Ciliberto, Francis Bach, Alessandro Rudi · arXiv (Cornell University) · 2018
Key to structured prediction is exploiting the problem structure to simplify the learning process. A major challenge arises when data exhibit a local structure (e.g., are made by "parts") that can be leveraged to better approximate the relation between (parts of) the input and (parts of) the output. Recent literature on signal processing, and in particular computer vision, has shown that capturing these aspects is indeed essential to achieve state-of-the-art performance. While such algorithms are typically derived on a case-by-case basis, in this work we propose the first theoretical framework to deal with part-based data from a general perspective. We derive a novel approach to deal with these problems and study its generalization properties within the setting of statistical learning theory. Our analysis is novel in that it explicitly quantifies the benefits of leveraging the part-based structure of the problem with respect to the learning rates of the proposed estimator.
Cited by 3Open access
Yue Zhang, Zhiyang Teng · Natural Language Processing · 2020
With a machine learning approach and less focus on linguistic details, this gentle introduction to natural language processing develops fundamental mathematical and deep learning models for NLP under a unified framework. NLP problems are systematically organised by their machine learning nature, including classification, sequence labelling, and sequence-to-sequence problems. Topics covered include statistical machine learning and deep learning models, text classification and structured prediction models, generative and discriminative models, supervised and unsupervised learning with latent variables, neural networks, and transition-based methods. Rich connections are drawn between concepts throughout the book, equipping students with the tools needed to establish a deep understanding of NLP solutions, adapt existing models, and confidently develop innovative models of their own. Featuring a host of examples, intuition, and end of chapter exercises, plus sample code available as an online resource, this textbook is an invaluable tool for the upper undergraduate and graduate student.
Cited by 0
Carlo Ciliberto, Francis Bach, Alessandro Rudi · arXiv (Cornell University) · 2018
Key to structured prediction is exploiting the problem structure to simplify\nthe learning process. A major challenge arises when data exhibit a local\nstructure (e.g., are made by "parts") that can be leveraged to better\napproximate the relation between (parts of) the input and (parts of) the\noutput. Recent literature on signal processing, and in particular computer\nvision, has shown that capturing these aspects is indeed essential to achieve\nstate-of-the-art performance. While such algorithms are typically derived on a\ncase-by-case basis, in this work we propose the first theoretical framework to\ndeal with part-based data from a general perspective. We derive a novel\napproach to deal with these problems and study its generalization properties\nwithin the setting of statistical learning theory. Our analysis is novel in\nthat it explicitly quantifies the benefits of leveraging the part-based\nstructure of the problem with respect to the learning rates of the proposed\nestimator.\n
Cited by 2Open access
UNH Communications and Public Affairs · University of New Hampshire Scholars Repository (University of New Hampshire at Manchester) · 2019
Cited by 0
Alessandro Rudi, Carlo Ciliberto, Gian Maria Marconi et al. · arXiv (Cornell University) · 2018
Structured prediction provides a general framework to deal with supervised problems where the outputs have semantically rich structure. While classical approaches consider finite, albeit potentially huge, output spaces, in this paper we discuss how structured prediction can be extended to a continuous scenario. Specifically, we study a structured prediction approach to manifold valued regression. We characterize a class of problems for which the considered approach is statistically consistent and study how geometric optimization can be used to compute the corresponding estimator. Promising experimental results on both simulated and real data complete our study.
Cited by 7Open access
Nowozin, Sebastian 1980- · 2014
The goal of structured prediction is to build machine learning models that predict relational information that itself has structure, such as being composed of multiple interrelated parts. These models, which reflect prior knowledge, task-specific relations, and constraints, are used in fields including computer vision, speech recognition, natural language processing, and computational biology. They can carry out such tasks as predicting a natural language sentence, or segmenting an image into meaningful components. These models are expressive and powerful, but exact computation is often intractable. A broad research effort in recent years has aimed at designing structured prediction models and approximate inference and learning procedures that are computationally efficient. This volume offers an overview of this recent research in order to make the work accessible to a broader research community.
Cited by 0
Hal Daumé, John Langford, Daniel Marcu · ArXiv.org · 2009
We present Searn, an algorithm for integrating search and learning to solve complex structured prediction problems such as those that occur in natural language, speech, computational biology, and vision. Searn is a meta-algorithm that transforms these complex problems into simple classification problems to which any binary classifier may be applied. Unlike current algorithms for structured learning that require decomposition of both the loss function and the feature functions over the predicted structure, Searn is able to learn prediction functions for any loss function and any class of features. Moreover, Searn comes with a strong, natural theoretical guarantee: good performance on the derived classification problems implies good performance on the structured prediction problem.
Cited by 73Open access
Tolga Bolukbasi, Kai-Wei Chang, Joseph Wang et al. · arXiv (Cornell University) · 2016
We study the problem of structured prediction under test-time budget constraints. We propose a novel approach applicable to a wide range of structured prediction problems in computer vision and natural language processing. Our approach seeks to adaptively generate computationally costly features during test-time in order to reduce the computational cost of prediction while maintaining prediction performance. We show that training the adaptive feature generation system can be reduced to a series of structured learning problems, resulting in efficient training using existing structured learning algorithms. This framework provides theoretical justification for several existing heuristic approaches found in literature. We evaluate our proposed adaptive system on two structured prediction tasks, optical character recognition (OCR) and dependency parsing and show strong performance in reduction of the feature costs without degrading accuracy.
Cited by 4Open access
Garoe Dorta, Sara Vicente, Lourdes Agapito et al. · arXiv (Cornell University) · 2018
This paper is the first work to propose a network to predict a structured uncertainty distribution for a synthesized image. Previous approaches have been mostly limited to predicting diagonal covariance matrices [15]. Our novel model learns to predict a full Gaussian covariance matrix for each reconstruction, which permits efficient sampling and likelihood evaluation. We demonstrate that our model can accurately reconstruct ground truth correlated residual distributions for synthetic datasets and generate plausible high frequency samples for real face images. We also illustrate the use of these predicted covariances for structure preserving image denoising.
Cited by 3Open access
Vicente Alabau Gonzalvo · 2014
Volia aprofitar estes lnies per a agrair a totes les persones que, d'una o altra forma, m'han oferit el seu suport i confiana per a que esta tesi s'haja pogut acabar.
Cited by 0Open access
David Belanger, Andrew McCallum · arXiv (Cornell University) · 2015
We introduce structured prediction energy networks (SPENs), a flexible framework for structured prediction. A deep architecture is used to define an energy function of candidate labels, and then predictions are produced by using back-propagation to iteratively optimize the energy with respect to the labels. This deep architecture captures dependencies between labels that would lead to intractable graphical models, and performs structure learning by automatically learning discriminative features of the structured output. One natural application of our technique is multi-label classification, which traditionally has required strict prior assumptions about the interactions between labels to ensure tractable learning and prediction. We are able to apply SPENs to multi-label problems with substantially larger label sets than previous applications of structured prediction, while modeling high-order interactions using minimal structural assumptions. Overall, deep learning provides remarkable tools for learning features of the inputs to a prediction problem, and this work extends these techniques to learning features of structured outputs. Our experiments provide impressive performance on a variety of benchmark multi-label classification tasks, demonstrate that our technique can be used to provide interpretable structure learning, and illuminate fundamental trade-offs between feed-forward and iterative structured prediction.
Cited by 15Open access
Er Grubb, Daniel Munoz, J. Andrew Bagnell · 2013
Structured prediction plays a central role in machine learning appli-cations from computational biology to computer vision. These models require significantly more computation than unstructured models, and, in many applications, algorithms may need to make predictions within a computational budget or in an anytime fashion. In this work we pro-pose an anytime technique for learning structured prediction that, at training time, incorporates both structural elements and feature compu-tation trade-offs that affect test-time inference. We apply our technique to the challenging problem of scene understanding in computer vision and demonstrate efficient and anytime predictions that gradually improve towards state-of-the-art classification performance as the allotted time in-creases. 1
Cited by 0
Hal Daumé, John Langford, Daniel Marcu · Machine Learning · 2009
Cited by 436Open access
Alabau Gonzalvo, Vicente · Tesis Doctorals en Xarxa (Consorci de Serveis Universitaris de Catalunya) · 2014
This thesis presents scientific contributions to the field of multimodal interac-\ntive structured prediction (MISP). The aim of MISP is to reduce the human\neffort required to supervise an automatic output, in an efficient and ergonomic\nway. Hence, this thesis focuses on the two aspects of MISP systems. The first\naspect, which refers to the interactive part of MISP, is the study of strate-\ngies for efficient human¿computer collaboration to produce error-free outputs.\nMultimodality, the second aspect, deals with other more ergonomic modalities\nof communication with the computer rather than keyboard and mouse.\nTo begin with, in sequential interaction the user is assumed to supervise the\noutput from left-to-right so that errors are corrected in sequential order. We\nstudy the problem under the decision theory framework and define an optimum\ndecoding algorithm. The optimum algorithm is compared to the usually ap-\nplied, standard approach. Experimental results on several tasks suggests that\nthe optimum algorithm is slightly better than the standard algorithm.\nIn contrast to sequential interaction, in active interaction it is the system that\ndecides what should be given to the user for supervision. On the one hand, user\nsupervision can be reduced if the user is required to supervise only the outputs\nthat the system expects to be erroneous. In this respect, we define a strategy\nthat retrieves first the outputs with highest expected error first. Moreover, we\nprove that this strategy is optimum under certain conditions, which is validated\nby experimental results. On the other hand, if the goal is to reduce the number\nof corrections, active interaction works by selecting elements, one by one, e.g.,\nwords of a given output to be supervised by the user. For this case, several\nstrategies are compared. Unlike the previous case, the strategy that performs\nbetter is to choose the element with highest confidence, which coincides with\nthe findings of the optimum algorithm for sequential interaction. However, this\nalso suggests that minimizing effort and supervision are contradictory goals.\nWith respect to the multimodality aspect, this thesis delves into techniques to\nmake multimodal systems more robust. To achieve that, multimodal systems\nare improved by providing contextual information of the application at hand.\nFirst, we study how to integrate e-pen interaction in a machine translation\ntask. We contribute to the state-of-the-art by leveraging the information from the source sentence. Several strategies are compared basically grouped into two\napproaches: inspired by word-based translation models and n-grams generated\nfrom a phrase-based system. The experiments show that the former outper-\nforms the latter for this task. Furthermore, the results present remarkable\nimprovements against not using contextual information. Second, similar ex-\nperiments are conducted on a speech-enabled interface for interactive machine\ntranslation. The improvements over the baseline are also noticeable. How-\never, in this case, phrase-based models perform much better than word-based\nmodels. We attribute that to the fact that acoustic models are poorer estima-\ntions than morphologic models and, thus, they benefit more from the language\nmodel. Finally, similar techniques are proposed for dictation of handwritten\ndocuments. The results show that speech and handwritten recognition can be\ncombined in an effective way.\nFinally, an evaluation with real users is carried out to compare an interactive\nmachine translation prototype with a post-editing prototype. The results of\nthe study reveal that users are very sensitive to the usability aspects of the\nuser interface. Therefore, usability is a crucial aspect to consider in an human\nevaluation that can hinder the real benefits of the technology being evaluated.\nHopefully, once usability problems are fixed, the evaluation indicates that users\nare more favorable to work with the interactive machine translation system than\nto the post-editing system.
Cited by 0
Pedro M. Q. Aguiar, Mário A. T. Figueiredo · 2014
Structured prediction (SP) problems are characterized by strong interdependence among the output variables, usually with sequential, graphical, or combinatorial structure [17, 7]. Obtaining good predictors often requires a large effort in feature/kernel design and tuning (usually done via cross-validation). Because discriminative training of structured predictors can be quite slow, specially in
Cited by 0