Skip to content
Free planNo credit card requiredStart writing
AI for Research home
AI for Research home
Dashboard
Sign inGet started

Search 250 million research papers

Read the abstract, see who wrote it and who cites it, and get the citation in APA, MLA, Chicago or Harvard.

About 10,000 papers for “Multi-task learning”

Multi-task Learning

Khalid M. Mosalam, Yuqing Gao · Synthesis lectures on mechanical engineering · 2024

Cited by 1

Multi-Task Learning with Multi-Task Optimization

Lu Bai, Abhishek Gupta, Yew-Soon Ong · arXiv (Cornell University) · 2024

Multi-task learning solves multiple correlated tasks. However, conflicts may exist between them. In such circumstances, a single solution can rarely optimize all the tasks, leading to performance trade-offs. To arrive at a set of optimized yet well-distributed models that collectively embody different trade-offs in one algorithmic pass, this paper proposes to view Pareto multi-task learning through the lens of multi-task optimization. Multi-task learning is first cast as a multi-objective optimization problem, which is then decomposed into a diverse set of unconstrained scalar-valued subproblems. These subproblems are solved jointly using a novel multi-task gradient descent method, whose uniqueness lies in the iterative transfer of model parameters among the subproblems during the course of optimization. A theorem proving faster convergence through the inclusion of such transfers is presented. We investigate the proposed multi-task learning with multi-task optimization for solving various problem settings including image classification, scene understanding, and multi-target regression. Comprehensive experiments confirm that the proposed method significantly advances the state-of-the-art in discovering sets of Pareto-optimized models. Notably, on the large image dataset we tested on, namely NYUv2, the hypervolume convergence achieved by our method was found to be nearly two times faster than the next-best among the state-of-the-art.

Cited by 0Open access

Learning Sparse Task Relations in Multi-Task Learning

Yu Zhang, Qiang Yang · Proceedings of the AAAI Conference on Artificial Intelligence · 2017

In multi-task learning, when the number of tasks is large, pairwise task relations exhibit sparse patterns since usually a task cannot be helpful to all of the other tasks and moreover, sparse task relations can reduce the risk of overfitting compared with the dense ones. In this paper, we focus on learning sparse task relations. Based on a regularization framework which can learn task relations among multiple tasks, we propose a SParse covAriance based mulTi-taSk (SPATS) model to learn a sparse covariance by using the ℓl regularization. The resulting objective function of the SPATS method is convex, which allows us to devise an alternating method to solve it. Moreover, some theoretical properties of the proposed model are studied. Experiments on synthetic and real-world datasets demonstrate the effectiveness of the proposed method.

Cited by 59Open access

Task-Aware Federated Multi-Task Learning

Lei Li, Haochen Yang, Jiacheng Guo et al. · 2025

Federated Multi-Task Learning (FMTL) enables collaborative training of multiple tasks across decentralized clients, but faces two key challenges in practice: negative transfer among tasks and scalability under resource constraints. Task differences can cause gradient conflicts that degrade overall performance, while limited computation and storage on edge devices make it difficult to maintain accuracy with low overhead. Existing methods address these issues either by adopting multi-backbone architectures, which split tasks to reduce interference but incur substantial parameter and computation costs, or by performing naive global averaging, which ignores inter-task differences and fails to effectively mitigate negative transfer. To overcome these limitations, we propose Task-Aware Federated Multi-Task Learning (TA-FMTL), a single-backbone framework that balances accuracy and efficiency. TA-FMTL integrates two lightweight components: a min–max task-difficulty weighting strategy that dynamically allocates more updates to harder tasks for balanced optimization, and a variance-aware reputation aggregation that down-weights clients with high overall loss or unstable cross-task performance. This design enables robust coordination across heterogeneous tasks without task splitting. Experiments on the Taskonomy benchmark show that TA-FMTL consistently achieves better or comparable accuracy to state-of-the-art MAS variants while reducing parameters by up to 77.2% and FLOPs by 37.3% in challenging 5-task and 9-task settings, demonstrating its scalability and practicality for real-world FMTL under heterogeneous and resource-limited conditions.

Cited by 0Open access

Task Variance Regularized Multi-Task Learning

Yuren Mao, Zekai Wang, Weiwei Liu et al. · IEEE Transactions on Knowledge and Data Engineering · 2022

Multi-task Learning (MTL), which involves the simultaneous learning of multiple tasks, can achieve better performance than learning each task independently. It has achieved great success in various applications, ranging from Computer Vision (CV) to Natural Language Processing (NLP). In MTL, the losses of the including tasks are jointly optimized. However, it is common for these tasks to be competing. When the tasks are competing, minimizing the losses of some tasks increases the losses of others, which accordingly increases the task variance (variance between the task-specific loss); furthermore, it induces under-fitting in some tasks and over-fitting in others, which degenerates the generalization performance of an MTL model. To address this issue, it is necessary to control the task variance; thus, task variance regularization is a natural choice. While intuitive, task variance regularization remains unexplored in MTL. Accordingly, to fill this gap, we study the generalization error bound of MTL through the lens of task variance and propose the task variance matters the generalization performance of MTL. Furthermore, this paper investigates how the task variance might be effectively regularized, and consequently proposes a multi-task learning method based on adversarial multi-armed bandit. The proposed method, dubbed BanditMTL, regularizes the task variance by means of a mirror gradient ascent-descent algorithm. Adopting BanditMTL both in CV and NLP applications is found to achieve state-of-the-art performance. The results of extensive experiments back up our theoretical analysis and validate the superiority of our proposals.

Cited by 5

Hierarchical Multi-task Learning

Malakouti, Salim · D-Scholarship@Pitt (University of Pittsburgh) · 2023

Traditionally, machine learning research has adopted methods that were designed to learn one or a set of machine learning tasks independently. However, motivated by our brain's learning mechanism to transfer knowledge from past and other related experiences, recent research has developed and studied methods incorporating target task relationships in the learning algorithms. The area of machine learning in which multiple target tasks are solved simultaneously while exploiting their similarities and underlying structures is known as multi-task learning. Multi-task learning methods (MTL) have proven effective in learning improved machine learning models by facilitating the transfer of knowledge through simultaneously learning a set of target tasks. However, the success of existing multi-task learning methods depends on the extent of the similarity between the target tasks. When tasks are not sufficiently similar, the negative transfer that impacts the quality of the learned models may occur. Therefore, new techniques were adopted that took advantage of task clusters, task-task relatedness, or an asymmetric knowledge transfer. However, none of these techniques are adequate when applied to a large number of heterogeneous tasks organized in a complex hierarchical structure. The abundance of such hierarchies in many domains, including health-care, document classification, and image classification, motivates the development of a new class of multi-task learning methods that can take advantage of these complex hierarchical task relationships. In this thesis, we explore and develop supervised multi-task learning methods that leverage existing task hierarchies to guide the transfer of knowledge between related tasks and evaluate these methods in the context of healthcare applications. First, we propose a simple, yet flexible, approach for learning low-dimensional representations of patients' electronic health records data that are able to overcome challenges related to learning of the models for multiple target tasks from such data. Second, we propose new hierarchical multi-task learning methods that enable the transfer of knowledge in the form of parameter transfer. Third, we study and present new feature-based hierarchical multi-task learning methods that utilize feature transfer instead of parameter transfer solutions to further improve the performance of the models. Finally, we discuss the open questions and problems and provide ideas for future research directions.

Cited by 0Open access

Asynchronous Multi-Task Learning

İnci M. Baytaş, Ming Yan, Anil K. Jain et al. · arXiv (Cornell University) · 2016

Many real-world machine learning applications involve several learning tasks which are inter-related. For example, in healthcare domain, we need to learn a predictive model of a certain disease for many hospitals. The models for each hospital may be different because of the inherent differences in the distributions of the patient populations. However, the models are also closely related because of the nature of the learning tasks modeling the same disease. By simultaneously learning all the tasks, multi-task learning (MTL) paradigm performs inductive knowledge transfer among tasks to improve the generalization performance. When datasets for the learning tasks are stored at different locations, it may not always be feasible to transfer the data to provide a data-centralized computing environment due to various practical issues such as high data volume and privacy. In this paper, we propose a principled MTL framework for distributed and asynchronous optimization to address the aforementioned challenges. In our framework, gradient update does not wait for collecting the gradient information from all the tasks. Therefore, the proposed method is very efficient when the communication delay is too high for some task nodes. We show that many regularized MTL formulations can benefit from this framework, including the low-rank MTL for shared subspace learning. Empirical studies on both synthetic and real-world datasets demonstrate the efficiency and effectiveness of the proposed framework.

Cited by 0Open access

Variational Multi-Task Learning

Jiayi Shen, Xiantong Zhen, Marcel Worring et al. · 2021

Multi-task learning aims to improve the overall performance of a set of tasks by leveraging their relatedness. When training data is limited using priors is pivotal, but currently this is done in ad-hoc ways. In this paper, we develop variational multi-task learning - VMTL, a general probabilistic inference framework for simultaneously learning multiple related tasks. We cast multi-task learning as a variational Bayesian inference problem, which enables task relatedness to be explored in a principled way by specifying priors. We introduce Gumbel-softmax priors to condition the prior of each task on related tasks. Each prior is represented as a mixture of variational posteriors of other related tasks and the mixing weights are learned in a data-driven manner for each individual task. The posteriors over representations and classifiers are inferred jointly for all tasks and individual tasks are able to improve their performance by using the shared inductive bias. Experimental results demonstrate that VMTL is able to tackle challenging multi-task learning with limited training data well, and it achieves state-of-the-art performance on four benchmarks, consistently surpassing previous methods.

Cited by 0

Boosted multi-task learning

Olivier Chapelle, Pannagadatta K. Shivaswamy, Srinivas Vadrevu et al. · Machine Learning · 2010

Cited by 74Open access

Regularized multi-task learning

Charles A. Micchelli, Massimiliano Pontil · 2004

This paper provides a foundation for multi–task learning using reproducing kernel Hilbert spaces of vector–valued functions. In this setting, the kernel is a matrix–valued function. Some explicit examples will be described which go beyond our earlier results in [7]. In particular, we characterize classes of matrix– valued kernels which are linear and are of the dot product or the translation invariant type. We discuss how these kernels can be used to model relations between the tasks and present linear multi–task learning algorithms. Finally, we present a novel proof of the representer theorem for a minimizer of a regularization functional which is based on the notion of minimal norm interpolation. 1

Cited by 0

Multi-task statistical learning

Matthieu Solnon, ARLOT, Sylvain, BACH, Francis · OpenGrey (Institut de l'Information Scientifique et Technique) · 2013

Cette thèse a pour objet la construction, la calibration et l'étude d'estimateurs multi-tâches, dans un cadre fréquentiste non paramétrique et non asymptotique. Nous nous plaçons dans le cadre de la régression ridge ànoyau et y étendons les méthodes existantes de régression multi-tâches. La question clef est la calibration d'un paramètre de régularisation matriciel, qui encode la similarité entre les tâches. Nous proposons une méthode de calibration de ce paramètre, fondée sur l'estimation de la matrice de covariance du bruit entre les tâches. Nous donnons ensuite pour l'estimateur obtenu des garanties d'optimalité, via une inégalité oracle, puis vérifions son comportement sur des exemples simulés. Nous obtenons par ailleurs un encadrement précis des risques des estimateurs oracles multi-tâches et mono-tâche dans certains cas. Cela nous permet de dégager plusieurs situations intéressantes, où l'oracle multi-tâches est plus efficace que l'oracle mono-tâche, ou vice versa. Cela nous permet aussi de nous assurer que l'inégalité oracle force l'estimateur multi-tâches à avoir un risque inférieur à l'estimateur mono-tâche dans les cas étudiés. Le comportement des oracles multi-tâches et mono-tâche est vérifié sur des exemples simulés.

Cited by 0Open access

Pareto Multi-Task Learning

Xi Lin, Hui‐Ling Zhen, Zhenhua Li et al. · arXiv (Cornell University) · 2019

Multi-task learning is a powerful method for solving multiple correlated tasks simultaneously. However, it is often impossible to find one single solution to optimize all the tasks, since different tasks might conflict with each other. Recently, a novel method is proposed to find one single Pareto optimal solution with good trade-off among different tasks by casting multi-task learning as multiobjective optimization. In this paper, we generalize this idea and propose a novel Pareto multi-task learning algorithm (Pareto MTL) to find a set of well-distributed Pareto solutions which can represent different trade-offs among different tasks. The proposed algorithm first formulates a multi-task learning problem as a multiobjective optimization problem, and then decomposes the multiobjective optimization problem into a set of constrained subproblems with different trade-off preferences. By solving these subproblems in parallel, Pareto MTL can find a set of well-representative Pareto optimal solutions with different trade-off among all tasks. Practitioners can easily select their preferred solution from these Pareto solutions, or use different trade-off solutions for different situations. Experimental results confirm that the proposed algorithm can generate well-representative solutions and outperform some state-of-the-art algorithms on many multi-task learning applications.

Cited by 52Open access

Multi-Stage Multi-Task Feature Learning

Pinghua Gong, Jieping Ye, Changshui Zhang · arXiv (Cornell University) · 2012

Multi-task sparse feature learning aims to improve the generalization performance by exploiting the shared features among tasks. It has been successfully applied to many applications including computer vision and biomedical informatics. Most of the existing multi-task sparse feature learning algorithms are formulated as a convex sparse regularization problem, which is usually suboptimal, due to its looseness for approximating an $\ell_0$-type regularizer. In this paper, we propose a non-convex formulation for multi-task sparse feature learning based on a novel non-convex regularizer. To solve the non-convex optimization problem, we propose a Multi-Stage Multi-Task Feature Learning (MSMTFL) algorithm; we also provide intuitive interpretations, detailed convergence and reproducibility analysis for the proposed algorithm. Moreover, we present a detailed theoretical analysis showing that MSMTFL achieves a better parameter estimation error bound than the convex formulation. Empirical studies on both synthetic and real-world data sets demonstrate the effectiveness of MSMTFL in comparison with the state of the art multi-task sparse feature learning algorithms.

Cited by 126Open access

Semisoft Task Clustering for Multi-Task Learning

Zhang Yuzhao, Yifan Sun · Advances in physics research/Advances in Physics Research · 2024

Multi-task learning (MTL) aims to improve the performance of multiple related prediction tasks by leveraging useful information from them.Due to their flexibility and ability to reduce unknown coefficients substantially, the task-clustering-based MTL approaches have attracted considerable attention.Motivated by the idea of semisoft clustering of data, we propose a semisoft task clustering approach, which can simultaneously reveal the task cluster structure for both pure and mixed tasks as well as select the relevant features.The main assumption behind our approach is that each cluster has some pure tasks, and each mixed task can be represented by a linear combination of pure tasks in different clusters.To solve the resulting non-convex constrained optimization problem, we design an efficient three-step algorithm.The experimental results based on synthetic and real-world datasets validate the effectiveness and efficiency of the proposed approach.Finally, we extend the proposed approach to a robust task clustering problem.

Cited by 0Open access