Skip to content
Free planNo credit card requiredStart writing
AI for Research home
AI for Research home
Dashboard
Sign inGet started

Search 250 million research papers

Read the abstract, see who wrote it and who cites it, and get the citation in APA, MLA, Chicago or Harvard.

About 10,000 papers for “Generalization error”

Fit scores and generalization error.

Stefan Pinkert (14013), Jörg Schultz (2714), Jörg Reichardt (211509) · Figshare · 2015

<p>A Comparison of highest fit scores (4) and (5) for the full HPRD dataset with 32,331 interactions. Aggregating nodes into cohesive groups (diagonal image graphs) cannot improve the score beyond a certain limit, while non-diagonal image graphs are able to capture more and more structure as the image graph gets larger and larger. For comparison, the analysis was repeated on a randomized (RND) version of the original network. Standard deviation is smaller than symbol size. The fit scores we obtain on the real data show that the structure we find is far from random. B After removing a test-set of links from the network, we optimized the assignment of nodes into classes according to (2) using only the remaining links and keeping the image graphs fixed to those found in the runs that lead to figure A. With the assignment of nodes into classes for this training set of links, we computed the score on the test set of links. The figure shows average and standard deviation over 10 repetitions of this experiment. C p-values of Student's t-test for a statistically significant difference in the means of the test scores of panel B. For higher numbers of classes and thus larger differences in the fit scores of diagonal and non-diagonal image graphs, all differences become significant at the level.</p>

Cited by 0Open access

Inference for the Generalization Error

Claude Nadeau, Yoshua Bengio · 1999

In order to compare learning algorithms, experimental results reported in the machine learning literature often use statistical tests of signicance to support the claim that a new learning algorithm generalizes better. Such tests should take into account the variability due to the choice of training set and not only that due to the test examples, as is often the case. This could lead to gross underestimation of the variance of the cross-validation estimator, and to the wrong conclusion that the new algorithm is signicantly better when it is not. We perform a theoretical investigation of the variance of a cross-validation estimator of the generalization error that takes into account the variability due to the randomness of the training set as well as test examples. Our analysis shows that all the variance estimators that are based only on the results of the cross-validation experiment must be biased. This analysis allows us to propose new estimators of this variance. We show, via simulations, that tests of hypothesis about the generalization error using those new variance estimators have better properties than tests involving variance estimators currently in use and listed in (Dietterich, 1998). In particular, the new tests have correct size and good power. That is, the new tests do not reject the null hypothesis too often when the hypothesis is true, but they tend to frequently reject the null hypothesis when the latter is false.

Cited by 0

Generalization error of spectral algorithms

Maksim Velikanov, Maxim S. Panov, Dmitry Yarotsky · arXiv (Cornell University) · 2024

The asymptotically precise estimation of the generalization of kernel methods has recently received attention due to the parallels between neural networks and their associated kernels. However, prior works derive such estimates for training by kernel ridge regression (KRR), whereas neural networks are typically trained with gradient descent (GD). In the present work, we consider the training of kernels with a family of $\textit{spectral algorithms}$ specified by profile $h(λ)$, and including KRR and GD as special cases. Then, we derive the generalization error as a functional of learning profile $h(λ)$ for two data models: high-dimensional Gaussian and low-dimensional translation-invariant model. Under power-law assumptions on the spectrum of the kernel and target, we use our framework to (i) give full loss asymptotics for both noisy and noiseless observations (ii) show that the loss localizes on certain spectral scales, giving a new perspective on the KRR saturation phenomenon (iii) conjecture, and demonstrate for the considered data models, the universality of the loss w.r.t. non-spectral details of the problem, but only in case of noisy observation.

Cited by 0Open access

Generalization error calculated as RMSECV.

Bjørn S. Dissing (351791), Michael E. Nielsen (351792), Bjarne K. Ersbøll (351793) et al. · Figshare · 2015

<p>To select a proper model, a leave one out cross validation scheme has been used, where the sum of squared errors (RMSECV) are shown here as a function of components included in the model. The lowest error is indicated with a vertical line, corresponding to a total of 7 components.</p>

Cited by 0Open access

SGD Implicitly Regularizes Generalization Error

Daniel A. Roberts · arXiv (Cornell University) · 2021

We derive a simple and model-independent formula for the change in the generalization gap due to a gradient descent update. We then compare the change in the test error for stochastic gradient descent to the change in test error from an equivalent number of gradient descent updates and show explicitly that stochastic gradient descent acts to regularize generalization error by decorrelating nearby updates. These calculations depends on the details of the model only through the mean and covariance of the gradient distribution, which may be readily measured for particular models of interest. We discuss further improvements to these calculations and comment on possible implications for stochastic optimization.

Cited by 7Open access

On the Training Error and Generalization Error of

Neural Network Regression, Katsuyuki Hagiwara · 2007

In this article, we analyzed the expected training error and the expected generalization error for neural networks in unidentifiable case, in which a set of output data is assumed to be a Gaussian noise sequence. Firstly, the results on the bounds of the expected training error and the expected generalization error for a general neural networks are reviewed. Secondly, we gave the order of the expected errors for a special Gaussian unit by combining the notion of #--covering and the extreme value theory.

Cited by 0

Generalization Error of Invariant Classifiers

Jure Sokolić, Raja Giryes, Guillermo Sapiro et al. · arXiv (Cornell University) · 2016

This paper studies the generalization error of invariant classifiers. In particular, we consider the common scenario where the classification task is invariant to certain transformations of the input, and that the classifier is constructed (or learned) to be invariant to these transformations. Our approach relies on factoring the input space into a product of a base space and a set of transformations. We show that whereas the generalization error of a non-invariant classifier is proportional to the complexity of the input space, the generalization error of an invariant classifier is proportional to the complexity of the base space. We also derive a set of sufficient conditions on the geometry of the base space and the set of transformations that ensure that the complexity of the base space is much smaller than the complexity of the input space. Our analysis applies to general classifiers such as convolutional neural networks. We demonstrate the implications of the developed theory for such classifiers with experiments on the MNIST and CIFAR-10 datasets.

Cited by 19Open access

Generalization Error in Deep Learning

Daniel Jakubovitz, Raja Giryes, Miguel R. D. Rodrigues · arXiv (Cornell University) · 2018

Deep learning models have lately shown great performance in various fields such as computer vision, speech recognition, speech translation, and natural language processing. However, alongside their state-of-the-art performance, it is still generally unclear what is the source of their generalization ability. Thus, an important question is what makes deep neural networks able to generalize well from the training set to new data. In this article, we provide an overview of the existing theory and bounds for the characterization of the generalization error of deep neural networks, combining both classical and more recent theoretical and empirical results.

Cited by 2Open access

Constructing CIs for 'the' Generalization Error

Fischer, Sebastian, Schulz-Kümpel, Hannah · Zenodo (CERN European Organization for Nuclear Research) · 2024

This data contains information that is too large to be included in our GitHub repository: https://github.com/slds-lmu/paper_2023_ci_for_ge.After downloading the data you need to first extract/decompress it.The files are:* results - Result datasets that need to be moved into the the GitHub repository in order to reproduce the analysis from the paper. Both the raw and processed datasets are contained.* figures-granular - additional figures on a very granular levelThe folders also contain READMEs describing the datasets and figures.If you want to work with the datasets (all in .rds format) but don't want to work in R, you can simply load them in R (using readRDS(<path>)) and then convert them e.g. to csv using the write.csv function.

Cited by 0Open access

Constructing CIs for 'the' Generalization Error

Fischer, Sebastian, Schulz-Kümpel, Hannah · Zenodo (CERN European Organization for Nuclear Research) · 2024

This data contains information that is too large to be included in our GitHub repository: https://github.com/slds-lmu/paper_2023_ci_for_ge.After downloading the data you need to first extract/decompress it.The files are:* results - Result datasets that need to be moved into the the GitHub repository in order to reproduce the analysis from the paper. Both the raw and processed datasets are contained.* figures-granular - additional figures on a very granular levelThe folders also contain READMEs describing the datasets and figures.If you want to work with the datasets (all in .rds format) but don't want to work in R, you can simply load them in R (using readRDS(<path>)) and then convert them e.g. to csv using the write.csv function.

Cited by 0Open access

mlr3inferr: Inference on the Generalization Error

Sebastian Fischer, Hannah Kümpel · 2025

An 'mlr3' extension that provides various resampling-based confidence interval (CI) methods for estimating the generalization error. These CI methods are implemented as 'mlr3' measures, enabling the evaluation of individual algorithms on specific tasks as well as the comparison of different learning algorithms.

Cited by 1Open access

A Generalization Error for Q-Learning.

Susan A. Murphy · PubMed · 2005

Planning problems that involve learning a policy from a single training set of finite horizon trajectories arise in both social science and medical fields. We consider Q-learning with function approximation for this setting and derive an upper bound on the generalization error. This upper bound is in terms of quantities minimized by a Q-learning algorithm, the complexity of the approximation space and an approximation term due to the mismatch between Q-learning and the goal of learning a policy that maximizes the value function.

Cited by 122Open access

Process-Oriented Estimation of Generalization Error

2008

Methods to avoid overfitting fall into two broad categories: data-oriented (using separate data for validation) and representation-oriented (penalizing complexity in the model). Both have limitations that are hard to overcome. We argue that fully adequate model evaluation is only possible if the search process by which models are obtained is also taken into account. To this end, we recently proposed a method for process-oriented evaluation (P0E), and successfully applied it to rule induction [Domingos, 1998b]. However, for the sake of simplicity this treatment made a number of rather artificial assumptions. In this paper the assumptions are removed, and a simple formula for error estimation is obtained. Empirical trials show the new, better-founded form of POE to be as accurate as the previous one, while further reducing theory sizes. 1

Cited by 0