Kristian Woodsend, Jacek Gondzio · 2007
Support Vector Machines (SVMs) are powerful machine learning techniques for classification and regression, and they offer state-of-the-art performance. The training of an SVM is computationally expensive and relies on optimization. The core of the approach is a dense convex quadratic optimization problem (QP), min z 1 2 zT Y KY z − e T z s.t. y T z = 0 (1) 0 ≤ z ≤ τe, where K ∈ R n×n is the matrix resulting from applying a (possibly nonlinear) kernel function to all points of the data set, y ∈ {−1, 1} n the vector of classification labels, Y = diag(y), z ∈ R n the support vector variables, τ> 0 a scalar parameter of the problem, and e ∈ R n the vector of all ones. Due to the dense Hessian matrix, a general-purpose QP solver needs O(n 3) flops to solve such a problem. This complexity result makes applying SVMs to large-scale data sets challenging, and
Cited by 0
Fawzia Abujalala, Hajer Hajer, Ahmed M M. Abushaala · مجلة البحوث الأكاديمية · 2018
Machine Learning (ML) is a very important field that enable researchers to develop many computer-based applications that can be used to facilitate employer's carrier in many scientific area. Support vector machine (SVM) is one of the most popular supervised learning algorithms in ML that are used to classify data. In this paper, researchers have trained a support vector machine classifier over linearly and nonlinearly separable data set. Over the nonlinearly separable dataset to types of kernels has been used, polynomial kernel and radial-basis function (RBF) kernel. The polynomial kernel reached the peak accuracy of 96% at degree of 10 and no additional capacity control. However, RBF kernel reached the peak accuracy of 96% at RBF sigma of 1 and misclassification error of 10. Moreover, cross validation technique has been used to improve and measure the performance of the nonlinear classifier
Cited by 0Open access
Tang Faming, Chen Mian‐yun, Zhongdong Wang · Journal of Systems Engineering and Electronics · 2006
Support vector machine has become an increasingly popular tool for machine learning tasks involving classification, regression or novelty detection. Training a support vector machine requires the solution of a very large quadratic programming problem. Traditional optimization methods cannot be directly applied due to memory restrictions. Up to now, several approaches exist for circumventing the above shortcomings and work well. Another learning algorithm, particle swarm optimization, for training SVM is introduted. The method is tested on UCI datasets.
Cited by 9
Kristian Woodsend, Jacek Gondzio · Springer optimization and its applications · 2008
Cited by 8
Bryan Catanzaro, Narayanan Sundaram, Kurt Keutzer · 2008
Recent developments in programmable, highly parallel Graphics Processing Units (GPUs) have enabled high performance implementations of machine learning algorithms. We describe a solver for Support Vector Machine training running on a GPU, using the Sequential Minimal Optimization algorithm and an adaptive first and second order working set selection heuristic, which achieves speedups of 9-35 × over LIBSVM running on a traditional processor. We also present a GPU-based system for SVM classification which achieves speedups of 81-138 × over LIBSVM (5-24 × over our own CPU based SVM classifier). 1.
Cited by 0
Hong Gunn Chew, Cheng‐Chew Lim, R.E. Bogner · Adelaide Research & Scholarship (AR&S) (University of Adelaide) · 2001
Cited by 2
Bauckhage, Christian, Schneider, Helen, Wulff, Benjamin et al. · Open MIND · 2022
We explore the merits of training of support vector machines for binary classification by means of solving systems of ordinary differential equations. We thus assume a continuous time perspective on a machine learning problem which may be of interest for implementations on (re)emerging hardware platforms such as analog- or quantum computers.
Cited by 0Open access
Ruchi Shah, Shaoshuai Zhang, Ying Lin et al. · 2019
Kernel Support Vector Machine (SVM) is a popular machine learning model for classification and regression. A significant challenge of large scale Kernel SVM is the size of the Gram matrix (n × n), which cannot be stored or processed efficiently when training data-set is large (e.g. n in the millions). This paper proposes a novel SVM training algorithm and its parallelization strategy that can efficiently train on data-sets with millions of samples on thousands of processors. It consists of an accurate, fast, and scalable low rank matrix approximation based on random projection, and a primal-dual interior point method to solve the approximated optimization problem. We demonstrate that xSVM is fast, scalable, and accurate on large scale data-sets and computing nodes. Compared to state-of-the-art distributed Kernel L1-SVM system xSVM is consistently several times faster, with comparable accuracy to the exact model trained by LIBSVM.
Cited by 6
Christian Bauckhage, Helen Schneider, Benjamin Wulff et al. · arXiv (Cornell University) · 2022
We explore the merits of training of support vector machines for binary classification by means of solving systems of ordinary differential equations. We thus assume a continuous time perspective on a machine learning problem which may be of interest for implementations on (re)emerging hardware platforms such as analog- or quantum computers.
Cited by 0Open access
2008
Training support vector machines involves a huge optimization problem and many specially designed algorithms have been proposed. In this paper, we proposed an algorithm called ClusterSVM that accelerates the training process by exploiting the distributional properties of the training data, that is, the natural clustering of the training data and the overall layout of these clusters relative to the decision boundary of support vector machines. The proposed algorithm first partitions the training data into several pair-wise disjoint clusters. Then, the representatives of these clusters are used to train an initial support vector machine, based on which we can approximately identify the support vectors and non-support vectors. After replacing the cluster containing only non-support vectors with its representative, the number of training data can be significantly reduced, thereby speeding up the training process. The proposed ClusterSVM has been tested against the popular training algorithm SMO on both the artificial data and the real data, and a significant speedup was observed. The complexity of ClusterSVM scales with the square of the number of support vectors and, after a further improvement, it is expected that it will scale with square of the number of non-boundary support vectors.
Cited by 0
Kristian Woodsend, Jacek Gondzio · 2009
Support Vector Machines are a powerful machine learning technology, but the training process involves a dense quadratic optimization problem and is computationally challenging. A parallel implementation of Support Vector Machine training has been developed, using a combination of MPI and OpenMP. Using an interior point method for the optimization and a reformulation that avoids the dense Hessian matrix, the structure of the augmented system matrix is exploited to partition data and computations amongst parallel processors efficiently. The new implementation has been applied to solve problems from the PASCAL Challenge on Large Scale Learning. We show that our approach is competitive, and is able to solve problems in the Challenge many times faster than other parallel approaches. We also demonstrate that the hybrid version performs more efficiently than the version using pure MPI. 1
Cited by 1
Yen-Ju Lin · OakTrust (Texas A&M University Libraries) · 2016
Machine learning algorithms allow us to reason about and analyze large amounts of data. The support vector machine (SVM) is one popular learning algorithm, which has been applied to a broad range of applications. To this end, hardware-based SVM processors are very appealing due to their improved runtime and energy efficiency. \n\nThis research proposes an FPGA-based parallel support vector machine processor, which is capable of processing multi-dimensional data sets. The proposed FPGA SVM is based upon the cascade SVM algorithm, which is leveraged to allow efficient parallel processing of data on the FPGA platform, leading to significant processing efficiency.
Cited by 0Open access
Rafet Sifa, Daniel Paurat, Daniel Trabold et al. · Lecture notes in computer science · 2018
Cited by 5
Nello Cristianini, Colin Campbell, John Shawe‐Taylor · 1999
Abstract. Support Vector Machines nd maximal margin hyperplanes in a high dimensional feature space, represented as a sparse linear com-bination of training points. Theoretical results exist which guarantee a high generalization performance when the margin is large or when the representation is very sparse. Multiplicative-Updating algorithms are a new tool for perceptron learning which are guaranteed to converge rapidly when the target concept is sparse. In this paper we present a Multiplicative-Updating algorithm for training Support Vector Machines which combines the generalization power provided by VC theory with the convergence properties of multiplicative algorithms. 1.
Cited by 14
Jianxiong Dong, Adam Krzyżak, Ching Y. Suen · Lecture notes in computer science · 2007
Cited by 61
Woodsend, Kristian, Gondzio, Jacek · Edinburgh Research Explorer (University of Edinburgh) · 2009
Support vector machines are a powerful machine learning technology, but the training process involves a dense quadratic optimization problem and is computationally challenging. A parallel implementation of linear Support Vector Machine training has been developed, using a combination of MPI and OpenMP. Using an interior point method for the optimization and a reformulation that avoids the dense Hessian matrix, the structure of the augmented system matrix is exploited to partition data and computations amongst parallel processors efficiently. The new implementation has been applied to solve problems from the PASCAL Challenge on Large-scale Learning. We show that our approach is competitive, and is able to solve problems in the Challenge many times faster than other parallel approaches. We also demonstrate that the hybrid version performs more efficiently than the version using pure MPI.
Cited by 0Open access
Kristian Woodsend, Jacek Gondzio · 2007
Support vector machine training can be represented as a large quadratic program. We present an efficient and numerically stable algorithm for this problem using interior point methods, which requires only O(n) operations per iteration. Through exploiting the separability of the Hessian, we provide a unified approach, from an optimization perspective, to 1-norm classification, 2-norm classification and ɛ-insensitive regression. Numerical experiments indicate that, in contrast to existing decomposition methods, the algorithm is largely unaffected by noisy data, for both linear and non-linear kernels, and they show our implementation outperforming all known implementations by a large margin. We discuss the effect of using multiple correctors, and monitoring the angle of the normal to the hyperplane to determine termination.
Cited by 0
Alba Sloin, David Burshtein · IEEE Transactions on Signal Processing · 2007
We present a discriminative training algorithm, that uses support vector machines (SVMs), to improve the classification of discrete and continuous output probability hidden Markov models (HMMs). The algorithm uses a set of maximum-likelihood (ML) trained HMM models as a baseline system, and an SVM training scheme to rescore the results of the baseline HMMs. It turns out that the rescoring model can be represented as an unnormalized HMM. We describe two algorithms for training the unnormalized HMM models for both the discrete and continuous cases. One of the algorithms results in a single set of unnormalized HMMs that can be used in the standard recognition procedure (the Viterbi recognizer), as if they were plain HMMs. We use a toy problem and an isolated noisy digit recognition task to compare our new method to standard ML training. Our experiments show that SVM rescoring of hidden Markov models typically reduces the error rate significantly compared to standard ML training.
Cited by 99
Tiago Lino Bello, Luiz Carlos Matioli, Lucas Garcia Pedroso et al. · Greater South Information System · 2023
This paper introduces a filtering strategy for addressing optimization problems arising in binary Support Vector Machine classification. The training optimization problem aims to solve the dual formulation which involves a quadratic objective function subjected to a linear and box constraints. Our approach employs a Filter algorithm with Sequential Quadratic Programming iterations that minimize the quadratic Lagrangian approximations. Notably, we utilize the exact Hessian matrix in our numerical experiments to seek the desired classification function. Moreover, we present a Filter algorithm combined with the Augmented Lagrangian method aiming to accelerate the algorithm convergence. To substantiate our method's effectiveness, we conduct numerical experiments through MATLAB, comparing outcomes with alternative methodologies detailed in existing literature. Numerical experiments shows that the Filter--SQP combined with Augmented Lagrangian method is competitive and efficient method compared with an interior-point based solver and LIBSVM software in relation of classification metrics and CPU-time.
Cited by 0Open access
Elma Sibonghanoy Groenewald, Sarath Babu Dodda, Coenrad Adolph Groenewald et al. · Algorithms for intelligent systems · 2024
Cited by 0