Javier Cabezas Rodríguez · 2015
Multi-GPU systems are widely used in High Performance Computing environments to accelerate scientific computations. This trend is expected to continue as integrated GPUs will be introduced to processors used in multi-socket servers and servers will pack a higher number of GPUs per node. GPUs are currently connected to the system through the PCI Express interconnect, which provides limited bandwidth (compared to the bandwidth of the memory in GPUs) and it often becomes a bottleneck for performance scalability. Current programming models present GPUs as isolated devices with their own memory, even if they share the host memory with the CPU. Programmers explicitly manage allocations in all GPU memories and use primitives to communicate data between GPUs. Furthermore, programmers are required to use mechanisms such as command queues and inter-GPU synchronization. This explicit model harms the maintainability of the code and introduces new sources for potential errors. The first proposal of this thesis is the HPE model. HPE builds a simple, consistent programming interface based on three major features. (1) All device address spaces are combined with the host address space to form a Unified Virtual Address Space. (2) Programs are provided with an Asymmetric Distributed Shared Memory system for all the GPUs in the system. It allows to allocate memory objects that can be accessed by any GPU or CPU. (3) Every CPU thread can request a data exchange between any two GPUs, through simple memory copy calls. Such a simple interface allows HPE to provide always the optimal implementation; eliminating the need for application code to handle different system topologies. Experimental results show improvements on real applications that range from 5% in compute-bound benchmarks to 2.6x in communication-bound benchmarks. HPE transparently implements sophisticated communication schemes that can deliver up to a 2.9x speedup in I/O device transfers. The second proposal of this thesis is a shared memory programming model that exploits the new GPU capabilities for remote memory accesses to remove the need for explicit communication between GPUs. This model turns a multi-GPU system into a shared memory system with NUMA characteristics. In order to validate the viability of the model we also perform an exhaustive performance analysis of remote memory accesses over PCIe. We show that the unique characteristics of the GPU execution model and memory hierarchy help to hide the costs of remote memory accesses. Results show that PCI Express 3.0 is able to hide the costs of up to a 10% of remote memory accesses depending on the access pattern, while caching of remote memory accesses can have a large performance impact on kernel performance. Finally, we introduce AMGE, a programming interface, compiler support and runtime system that automatically executes computations that are programmed for a single GPU across all the GPUs in the system. The programming interface provides a data type for multidimensional arrays that allows for robust, transparent distribution of arrays across all GPU memories. The compiler extracts the dimensionality information from the type of each array, and is able to determine the access pattern in each dimension of the array. The runtime system uses the compiler-provided information to automatically choose the best computation and data distribution configuration to minimize inter-GPU communication and memory footprint. This model effectively frees programmers from the task of decomposing and distributing computation and data to exploit several GPUs. AMGE achieves almost linear speedups for a wide range of dense computation benchmarks on a real 4-GPU system with an interconnect with moderate bandwidth. We show that irregular computations can also benefit from AMGE, too. Los sistemas multi-GPU son muy comúnmente utilizados en entornos de computación de altas prestaciones para acelerar cálculos científicos. Esta tendencia continuará con la introducción de GPUs integradas en los procesadores de los servidores procesador y con una mayor densidad de GPUs por nodo. Las GPUs actualmente se contectan al sistema a través de una interconexión PCI Express, que provee un ancho de banda reducido (comparado con las memorias de las GPUs) y habitualmente se convierte en el cuello de botella para escalar el rendimiento. Los modelos de programación actuales exponen las GPUs como dispositivos aislados con su propia memoria, incluso si comparten la memoria física con la CPU. Los programadores manejan diferentes reservas en todas las memorias de GPU y usan primitivas para comunicar datos entre GPUs. Además, los programadores deben utilizar mecanismos como colas de comandos y sincronicación entre GPUs. Este modelo explícito empeora la programabilidad del código e introduce nuevas fuentes de errores potenciales. La primera propuesta de esta tesis es el modelo HPE. HPE construye una interfaz de programaci ón consistente basada en tres características principales. (1) Todos los espacios de direcciones de los dispositivos son combinados para formar un espacio de direcciones unificado. (2) Los programas usan un sistema asimétrico distribuido de memoria compartida para todas las GPUs del sistema, que permite declarar objetos de memoria que pueden ser accedidos por cualquier GPU o CPU. (3) Cada hilo de ejecución de la CPU puede lanzar un intercambio de datos entre dos GPUs a través de simples llamadas de copia de memoria. Esta interfaz simplificada permite a HPE usar la implementaci ón óptima; sinque la aplicación contemple diferentes topologías de sistema. Los resultados experimentales muestran mejoras en aplicaciones reales que van desde un 5% en aplicaciones limitadas por el cómputo a 2.6x aplicaciones imitadas por la comunicación. HPE implementa sofisticados esquemas de transferencia para dispositivos de E/S que proporcionan mejoras de rendimiento de 2.9x. La segunda propuesta de esta tesis es un modelo de programación basado en memoria compartida que aprovecha las nuevas capacidades acceso remoto de memoria de las GPUs para eliminar la comunicación explícita entre memorias de GPU. Este modelo convierte un sistema multi-GPU en un sistema de memoria compartida con características NUMA. Para validar la viabilidad del modelo realizamos un anlásis exhaustivo del rendimiento los accessos de memoria remotos sobre PCIe. Los resultados muestran que PCI Express 3.0 elimina los costes de hasta un 10% de accesos remotos, dependiendo en el patrón de acceso, mientras que guardar los accesos remotos en memorias cache tiene un gran inpacto en el rendimiento de las computaciones. Finalmente, presentamos AMGE, una interfaz de programación con soporte de compilación y un sistema que ejecuta, de forma automática, computaciones programadas para una única GPU en todas las GPUs del sistema. La interfaz de programación proporciona un tipo de datos para arreglos multidimensionales que permite una distribuci ón transparente y robusta de los datos en todas las memorias de GPU. El compilador extrae la información sobre la dimensionalidad de cada arreglo y puede determinar el patrón de acceso en cada dimensión de forma individual. El sistema utiliza, en tiempo de ejecución, la información del compilador para elegir la mejor descomposición de la computación y los datos para minimizar la comunicación entre GPUs y el uso de memoria. AMGE consigue mejoras de rendimiento que crecen de forma lineal con el número de GPUs para un amplio abanico de computaciones densas en un sistema real con 4 GPUs. También mostramos que las computaciones con patrones irregulares también se pueden beneficiar de AMGE.
Cited by 1Open access
Hammad Mazhar, Toby Heyn, Daniel Melanz et al. · ASME Press eBooks · 2014
This chapter discusses a multi-physics simulation engine, called Chrono, that relies heavily on parallel computing. Chrono aims at simulating the dynamics of systems containing rigid bodies, flexible (compliant) bodies, and fluid-rigid body interaction. To this end, it provides five modules: equation formulation (modeling), equation solution (simulation), collision detection support, domain decomposition for parallel computing, and post-processing analysis with emphasis on high quality rendering/visualization. For each component we point out how parallel central processing unit (CPU) and/or graphics processing unit (GPU) computing have been leveraged to allow for the physical simulation of problems with millions of degrees of freedom such as (1) rigid multi-body dynamics, (2) flexible body dynamics with friction and contact, and (3) fluid-structure interaction problems. Simulation and visualization of these physical phenomena becomes even more difficult at large scales, such as in granular dynamics.
Cited by 0
Andrea Borsic, Elia A. Attardo, Ryan J. Halter · Physiological Measurement · 2012
Image reconstruction in soft-field tomography is based on an inverse problem formulation, where a forward model is fitted to the data. In medical applications, where the anatomy presents complex shapes, it is common to use finite element models (FEMs) to represent the volume of interest and solve a partial differential equation that models the physics of the system. Over the last decade, there has been a shifting interest from 2D modeling to 3D modeling, as the underlying physics of most problems are 3D. Although the increased computational power of modern computers allows working with much larger FEM models, the computational time required to reconstruct 3D images on a fine 3D FEM model can be significant, on the order of hours. For example, in electrical impedance tomography (EIT) applications using a dense 3D FEM mesh with half a million elements, a single reconstruction iteration takes approximately 15-20 min with optimized routines running on a modern multi-core PC. It is desirable to accelerate image reconstruction to enable researchers to more easily and rapidly explore data and reconstruction parameters. Furthermore, providing high-speed reconstructions is essential for some promising clinical application of EIT. For 3D problems, 70% of the computing time is spent building the Jacobian matrix, and 25% of the time in forward solving. In this work, we focus on accelerating the Jacobian computation by using single and multiple GPUs. First, we discuss an optimized implementation on a modern multi-core PC architecture and show how computing time is bounded by the CPU-to-memory bandwidth; this factor limits the rate at which data can be fetched by the CPU. Gains associated with the use of multiple CPU cores are minimal, since data operands cannot be fetched fast enough to saturate the processing power of even a single CPU core. GPUs have much faster memory bandwidths compared to CPUs and better parallelism. We are able to obtain acceleration factors of 20 times on a single NVIDIA S1070 GPU, and of 50 times on four GPUs, bringing the Jacobian computing time for a fine 3D mesh from 12 min to 14 s. We regard this as an important step toward gaining interactive reconstruction times in 3D imaging, particularly when coupled in the future with acceleration of the forward problem. While we demonstrate results for EIT, these results apply to any soft-field imaging modality where the Jacobian matrix is computed with the adjoint method.
Cited by 15
Naw Safrin Sattar, Hao Lu, Feiyi Wang et al. · 2024
Community detection is a fundamental operation in graph mining, and by uncovering hidden structures and patterns within complex systems it helps solve fundamental problems pertaining to social networks, such as information diffusion, epidemics, and recommender systems. Scaling graph algorithms for massive networks becomes challenging on modern distributed-memory multi-GPU (Graphics Processing Unit) systems due to limitations such as irregular memory access patterns, load imbalances, higher communication-computation ratios, and cross-platform support. We present a novel algorithm HiPDPL-GPU (Distributed Parallel Louvain) to address these challenges. We conduct experiments involving different partitioning techniques to achieve an optimized performance of HiPDPL-GPU on the two largest supercomputers: Frontier and Summit. Remarkably, HiPDPL-GPU processes a graph with 4.2 billion edges in less than 3 minutes using 1024 GPUs. Qualitatively, the performance of HiPDPL-GPU is similar or better compared to other state-of-the-art CPU- and GPU-based implementations. While prior GPU implementations have predominantly employed CUDA, our first-of-its-kind implementation for community detection is cross-platform, accommodating both AMD and NVIDIA GPUs.
Cited by 6
Johansell Villalobos, Josef Ruzicka, Silvio Rizzi · ArXiv.org · 2024
Scientific computing in the exascale era demands increased computational power to solve complex problems across various domains. With the rise of heterogeneous computing architectures the need for vendor-agnostic, performance portability frameworks has been highlighted. Libraries like Kokkos have become essential for enabling high-performance computing applications to execute efficiently across different hardware platforms with minimal code changes. In this direction, this paper presents preliminary time-to-solution results for two representative scientific computing applications: an N-body simulation and a structured grid simulation. Both applications used a distributed memory approach and hardware acceleration through four performance portability frameworks: Kokkos, OpenMP, RAJA, and OCCA. Experiments conducted on a single node of the Polaris supercomputer using four NVIDIA A100 GPUs revealed significant performance variability among frameworks. OCCA demonstrated faster execution times for small-scale validation problems, likely due to JIT compilation, however its lack of optimized reduction algorithms may limit scalability for larger simulations while using its out of the box API. OpenMP performed poorly in the structured grid simulation most likely due to inefficiencies in inter-node data synchronization and communication. These findings highlight the need for further optimization to maximize each framework’s capabilities. Future work will focus on enhancing reduction algorithms, data communication, memory management, as wells as performing scalability studies, and a comprehensive statistical analysis to evaluate and compare framework performance.
Cited by 0Open access
Wang, Zeyu, Juhász, Zoltán · Zenodo (CERN European Organization for Nuclear Research) · 2023
Implementation strategies for EEG processing on multi-GPU computing systems Zeyu Wang, Zoltan Juhasz, Department of Electrical Engineering and Information SystemsUniversity of Pannonia, {zeyu.wang, juhasz.zoltan}@mik.uni-pannon.hu 8200 Veszprém, Egyetem u. 10. Abstract: Most EEG processing algorithms are very time-consuming due to the large amount of EEG data to be processed and the complexity of the algorithms required. Multi-GPU systems are ideal candidates to accelerate EEG processing tasks thanks to their high compute performance and support for massively parallel execution. However, developing multi-GPU programs is a difficult task and various overheads can reduce the final program performance. In this paper, we investigate the performance effects of various memory access implementation patterns for multi-GPU EEG algorithms and recommend strategies with the best performance to be used for large-scale EEG processing.
Cited by 0Open access
Sun Hongdi, Minzheng Jia, Gao Zhu · 2024
Generally, the single GPU computing method is utilized for the conventional radix sort algorithm based on GPU parallel computing. Nevertheless, as the data scale grows, the single GPU sorting algorithm is gradually demonstrating its performance bottlenecks. In the paper, an efficient radix sort algorithm based on multi-GPU parallel computing is proposed, which implements a strategy of using different bucket classifications on multiple GPUs to improve the sorting performance and efficiency of large-scale datasets. With the multi-GPU parallel computing, more buckets may be used for data classification in one traversal, effectively reducing data sorting times, lowering time complexity, and improving sorting speed and throughput. The experiment shows that the algorithm has significantly improved the operational efficiency, demonstrating good application prospects. Meanwhile, the algorithm herein also presents good scalability, which can adapt to the constantly growing data scale in the future.
Cited by 3
袁方, 赵瑞莲, 郭俊霞 et al. · 2016
Cited by 0
Marlon Arce Acuna, 青木尊之, Takayuki Aoki · Tokyo Tech Research Repository (Tokyo Institute of Technology) · 2010
Cited by 0
Zhuolong Jiang, Zinuo Cai, Hongyu Zhao et al. · IEEE Transactions on Computers · 2025
With the rapid growth of artificial intelligence and large-scale model computing, the demand for GPUs in datacenters continues to increase, especially for large-scale training and inference tasks. Heterogeneous multi-GPU systems, which integrate GPUs with varying types and computational capabilities, have become critical computing resources. This leads to two main challenges. First, due to the differences in GPU performance and power consumption, task scheduling involves a complex multi-objective optimization to balance energy efficiency and performance. More importantly, the lack of coordinated mechanisms for multi-task sharing and energy-efficient resource management across heterogeneous GPUs can result in GPU overload or underutilization, leading to wasted resources and potential system risks. <p xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">To address these challenges, we propose HESHARE, an energy-aware and efficient heterogeneous GPU framework for datacenters. First, we design an energy-aware task scheduling strategy that optimizes task allocation across different GPUs to achieve a balance between energy consumption and performance. Second, we introduce a GPU sharing optimization mechanism that adaptively configures MPS and DVFS settings for each GPU, enhancing resource utilization, reducing overall energy consumption, and ensuring task performance. Compared to the state-of-the-art framework, we reduce average energy costs by 26% and improve job completion time by 31%, achieving a balance between energy efficiency and performance.
Cited by 3
Cristina Ocaña · 2017
Cited by 0
Teng Li, Vikram K. Narayana, Tarek El‐Ghazawi · 2012
The HPC field is witnessing a widespread adoption of GPUs as accelerators for traditional homogeneous HPC systems. One of the prevalent parallel programming models is the SPMD paradigm, which has been adapted for GPU-based parallel processing. Since each process executes the same program under SPMD, every process mapped to a CPU core also needs the GPU availability. Therefore SPMD demands a symmetric CPU/GPU distribution. However, since modern HPC systems feature a large number of CPU cores that outnumber the number of GPUs, computing resources are generally underutilized with SPMD. Our previous efforts have focused on GPU virtualization that enables efficient sharing of GPU among multiple CPU processes. Nevertheless, a formal method to evaluate and choose the appropriate GPU sharing approach is still lacking. In this paper, based on SPMD GPU kernel profiles, we propose different multi-process GPU sharing scenarios under virtualization. We introduce an analytical model that captures these sharing scenarios and provides a theoretical performance gain estimation. Benchmarks validate our analyses and achievable performance gains. While our analytical study provides a suitable theoretical foundation for GPU sharing, the experimental results demonstrate that GPU virtualization affords significant performance improvements over the non-virtualized solutions for all proposed sharing scenarios.
Cited by 5
Kenia Picos, Víctor H. Diaz-Ramirez, Juan J. Tapia · Research in Computing Science · 2014
This work presents a real-time video processing algorithm for 3D scenes using a graphics processor. The processing is based on parallel computing using concurrent kernels. The proposed algorithm processes individual pixels of each pair of input stereo images to obtain an anaglyph image for each frame. To reduce the computational time, a concurrent kernel implementation using POSIX threads and CUDA streams is utilized. Also, an asynchronous streams execution is used to increase overlapping of the video processing implementation. The obtained results are presented and discussed in terms of speedup and execution time.
Cited by 1Open access
Marcos Lupión, N.C. Cruz, Juan F. Sanjuan et al. · The Journal of Supercomputing · 2022
Cited by 9Open access
Anju Mongandampulath Akathoott, Benila Virgin Jerald Xavier, Martin Burtscher · 2025
Computing a maximal independent set (MIS) of a graph is an important problem in many scientific applications.Several parallel algorithms exist to perform this computation quickly.Though the state-of-the-art GPU implementation is very efficient, it cannot process graphs that do not fit in the global memory of a single GPU.We propose MG-MIS, a multi-GPU algorithm that addresses this problem.It distributes the computation across the GPUs in a compute node and uses novel techniques to minimize inter-GPU communication.Our results show that, for graphs that require more than 32 GB memory, MG-MIS outperforms the state-of-the-art single-GPU code with UVM by a geometric mean of 17.73 on a system with 4 V100 GPUs, each with 32 GB global memory.For another set of graphs that require more than 12 GB memory, MG-MIS outperforms the same single-GPU code by 22.88 on a system with 2 RTX 3080 GPUs, each with a global memory of 12 GB.On average, the size of the MIS computed by MG-MIS is 2.6% smaller than that produced by the state-of-the-art single-GPU code.
Cited by 1Open access
Zhou, Faguo, Li, Shunde, Xue, Rong et al. · ArXiv.org · 2025
Three-dimensional neutron transport calculations using the Method of Characteristics (MOC) are highly regarded for their exceptional computational efficiency, precision, and stability. Nevertheless, when dealing with extensive-scale computations, the computational demands are substantial, leading to prolonged computation times. To address this challenge while considering GPU memory limitations, this study transplants the real-time generation and characteristic line computation techniques onto the GPU platform. Empirical evidence emphasizes that the GPU-optimized approach maintains a heightened level of precision in computation results and produces a significant acceleration effect. Furthermore, to fully harness the computational capabilities of GPUs, a dual approach involving characteristic line preloading and load balancing mechanisms is adopted, further enhancing computational efficiency. The resulting increase in computational efficiency, compared to traditional methods, reaches an impressive 300 to 400-fold improvement.
Cited by 0Open access
Πέτρος Αναστασιάδης · 2024
Οι πράξεις γραμμικής άλγεβρας εμφανίζονται συχνά σε εφαρμογές υψηλής απόδοσης (HPC), καθιστώντας την απόδοσή τους κρίσιμη για την επίτευξη βέλτιστης κλιμάκωσης. Καθώς πολλές σύγχρονες συστάδες HPC περιλαμβάνουν κόμβους με πολλαπλούς επεξεργαστές γραφικών (GPUs), οι πράξεις BLAS συχνά εκφορτώνονται σε GPUs, καθιστώντας απαραίτητη τη χρήση βελτιστοποιημένων βιβλιοθηκών για τη διασφάλιση της απόδοσης. Ωστόσο, η βελτιστοποίηση των BLAS σε πολλαπλές GPUs εισάγει πολλές προκλήσεις παρόμοιες με εκείνες του κατανεμημένου υπολογισμού, όπως την αποσύνθεση δεδομένων, την δρομολόγηση υποπροβλημάτων και την επικοινωνία μεταξύ GPU με διακριτές μνήμες. Αυτή η πολυπλοκότητα καθιστά την βελτιστοποίηση των BLAS πολύ περίπλοκη, οδηγώντας σε πτώση απόδοσης ή μεμονωμένες λύσεις που λειτουργούν μονο σε μερικά σύστήματα. Για να αντιμετωπίσουμε αυτά τα ζητήματα, προτείνουμε μια προσέγγιση αυτόματης βελτιστοποίησης με τη βοήθεια μοντελοποίησης: εισάγουμε διάφορα μοντέλα απόδοσης για τις BLAS και τα ενσωματώνουμε στο PARALiA, μια ολοκληρωμένη βιβλιοθήκη BLAS. Το PARALiA χρησιμοποιεί μοντέλα επίδοσης για την δυναμική αυτόματη βελτιστοποίηση της εκτέλεσης των BLAS, προσαρμόζοντας κρίσιμες παραμέτρους απόδοσης για κάθε συγκεκριμένο πρόβλημα και σύστημα κατά την εκτέλεση. Αυτή η αυτόματη βελτιστοποίηση συνδυάζεται με έναν δρομολογητή εργασιών, οδηγώντας σε αποδοτική κατανομή δεδομένων και απόδοση πόρων. Το PARALiA παρέχει κορυφαία απόδοση και ενεργειακή αποδοτικότητα και ενσωματώνει την ικανότητα προσαρμογής σε ετερογενή συστήματα και σενάρια μέσω αποφάσεων βασισμένων σε μοντέλα. Τέλος, εστιάζουμε στον πυρήνα GEMM, επεκτείνοντας το PARALiA με έναν προσαρμοσμένο στατικό δρομολογητή που ενσωματώνει νέες βελτιστοποιήσεις στον αλγόριθμο και την επικοινωνία της GEMM βασισμένες σε μοντέλα (PARALiA-GEMMex), ο οποίος παρέχει σημαντικά υψηλότερη απόδοση από τις προηγούμενες βιβλιοθήκες.
Cited by 0
Youngjin Jo, Seungjae Lee, Jaebum Cho et al. · 2018
Cited by 0
渡辺勢也, Seiya Watanabe, 青木尊之 et al. · Tokyo Tech Research Repository (Tokyo Institute of Technology) · 2020
Cited by 0
Soichiro Ikuno, Susumu Nakata, Yuta Hirokawa et al. · Journal of Physics Conference Series · 2015
High performance computing of Meshless Time Domain Method (MTDM) on multi-GPU using the supercomputer HA-PACS (Highly Accelerated Parallel Advanced system for Computational Sciences) at University of Tsukuba is investigated. Generally, the finite difference time domain (FDTD) method is adopted for the numerical simulation of the electromagnetic wave propagation phenomena. However, the numerical domain must be divided into rectangle meshes, and it is difficult to adopt the problem in a complexed domain to the method. On the other hand, MTDM can be easily adept to the problem because MTDM does not requires meshes. In the present study, we implement MTDM on multi-GPU cluster to speedup the method, and numerically investigate the performance of the method on multi-GPU cluster. To reduce the computation time, the communication time between the decomposed domain is hided below the perfect matched layer (PML) calculation procedure. The results of computation show that speedup of MTDM on 128 GPUs is 173 times faster than that of single CPU calculation.
Cited by 2Open access