article · International Journal of Innovative Research in Engineering & Multidisciplinary Physical Sciences · 2025
Optimal Utilization of Computing Resources with Data Parallelism and Task Parallelism in building AI models
Cited by 0Open access
Abstract
This article focuses on techniques for the most effective utilization of computing infrastructure for training and tuning the AI models in a distributed fashion used across various industries for making informed decisions. The objective is to investigate the potential benefits of data parallelism and task parallelism to optimize computing infrastructure which is in the form spark clusters on databricks. In addition the paper aims to illustrate how data parallelism and task parallelism can be achieved by setting up Ray cluster on top of databricks spark cluster to train and tune the Time series forecasting models to forecast sales of various products at stores in retail industry. The paper also discusses the significance of data parallelism and task parallelism in driving down the AI model building compute cost and time. The paper aims to provide the Machine Learning Engineers, Organization IT sponsors and all other stake holders actionable insights towards adoption of Ray cluster on top of databricks spark cluster. It presents detailed analysis of Time series data forecasting using Ray + Spark clusters combination facilitated by databricks. The paper aims to aid the organizations in enhancing the efficiency of computing resources, reducing the operational costs and speeding up the process of forecast model building using the solution presented.
Cite this paper
Malthummeda, V. (2025). Optimal Utilization of Computing Resources with Data Parallelism and Task Parallelism in building AI models. International Journal of Innovative Research in Engineering & Multidisciplinary Physical Sciences, 13(4). https://doi.org/10.37082/ijirmps.v13.i4.232655
Malthummeda, Vamshi. “Optimal Utilization of Computing Resources with Data Parallelism and Task Parallelism in building AI models.” International Journal of Innovative Research in Engineering & Multidisciplinary Physical Sciences, vol. 13, no. 4, 2025, https://doi.org/10.37082/ijirmps.v13.i4.232655.
Malthummeda, Vamshi. 2025. “Optimal Utilization of Computing Resources with Data Parallelism and Task Parallelism in building AI models.” International Journal of Innovative Research in Engineering & Multidisciplinary Physical Sciences 13 (4). https://doi.org/10.37082/ijirmps.v13.i4.232655.
Malthummeda, V. (2025) ‘Optimal Utilization of Computing Resources with Data Parallelism and Task Parallelism in building AI models’, International Journal of Innovative Research in Engineering & Multidisciplinary Physical Sciences, 13(4). Available at: https://doi.org/10.37082/ijirmps.v13.i4.232655.
@article{malthummeda2025optimal,
title = {{Optimal Utilization of Computing Resources with Data Parallelism and Task Parallelism in building AI models}},
author = {Malthummeda, Vamshi},
journal = {International Journal of Innovative Research in Engineering & Multidisciplinary Physical Sciences},
volume = {13},
number = {4},
year = {2025},
doi = {10.37082/ijirmps.v13.i4.232655}
}Made from the index record. Check names, capitalisation and page numbers against the paper. Need another style? AI for Research has over 10,000.