Zhou, Faguo, Li, Shunde, Xue, Rong et al. · arXiv (Cornell University) · 2025
Three-dimensional neutron transport calculations using the Method of Characteristics (MOC) are highly regarded for their exceptional computational efficiency, precision, and stability. Nevertheless, when dealing with extensive-scale computations, the computational demands are substantial, leading to prolonged computation times. To address this challenge while considering GPU memory limitations, this study transplants the real-time generation and characteristic line computation techniques onto the GPU platform. Empirical evidence emphasizes that the GPU-optimized approach maintains a heightened level of precision in computation results and produces a significant acceleration effect. Furthermore, to fully harness the computational capabilities of GPUs, a dual approach involving characteristic line preloading and load balancing mechanisms is adopted, further enhancing computational efficiency. The resulting increase in computational efficiency, compared to traditional methods, reaches an impressive 300 to 400-fold improvement.
Cited by 0Open access
Fernando Alexandre, Ricardo Manuel Fernandes Marques, Hervé Paulino · 2014
An emerging trend in the field of Graphics Processing Unit (GPU) computing is the harnessing of multiple devices to cope with scalability and performance requirements. However, multi-GPU execution adds new challenges to the already complex world of General Purpose computing on GPUs (GPGPU), such as the efficient problem decomposition, and dealing with device heterogeneity. To this extent, we propose the use of the Marrow algorithmic skeleton framework (ASkF) to abstract most of the details intrinsic to the programming of such platforms. To the best of our knowledge, Marrow is the first ASkF to support skeleton nesting on single and (now) multiple GPU systems. In this paper we present how it can transparently distribute the execution of skeleton compositions among a set of, possibly, heterogeneous devices. An experimental evaluation assesses the proposal's effectiveness, from a scalability and performance perspective, with good results.
Cited by 11
Björn Meyer, Christian Plessl, Jens Förstner · 2011
We propose an approach for high-performance scientific computing that separates the description of algorithms from the generation of code for parallel hardware architectures like Multi-Core CPUs, GPUs or FPGAs. This way, a scientist can focus on his domain of expertise by describing his algorithms generically without the need to have knowledge of specific hardware architectures, programming languages, APIs or tool flows. We present our prototype implementation that allows for transforming generic descriptions of algorithms with intensive array-type data access to highly optimized code for GPU and multi GPU cluster systems. We evaluate the approach for an example from the domain of computational nanophotonics and show that our current tool flow is able to generate efficient code that achieves speedups of up to 15.3x for a single GPU and even 35.9x for a multi GPU setup compared to a reference CPU implementation.
Cited by 3
Hanming Yang, Chenqi Zou, Daniel Shigueo Morikawa et al. · Computer Methods in Applied Mechanics and Engineering · 2026
Cited by 0
Simon McIntosh‐Smith · 2011
Cited by 3
Lorenzo Pichetti, Elia Cunegatti, Danilo Orlando et al. · SSRN Electronic Journal · 2025
Cited by 0Open access
Jeremy J. Williams, Jordy Trilaksono, Stefan Costea et al. · arXiv (Cornell University) · 2026
Particle-in-Cell (PIC) Monte Carlo (MC) simulations are central to plasma physics but face increasing challenges on heterogeneous HPC systems due to excessive data movement, synchronization overheads, and inefficient utilization of multiple accelerators. In this work, we present a portable, multi-GPU hybrid MPI+OpenMP implementation of BIT1 that enables scalable execution on both Nvidia and AMD accelerators through OpenMP target tasks with explicit dependencies to overlap computation and communication across devices. Portability is achieved through persistent device-resident memory, an optimized contiguous one-dimensional data layout, and a transition from unified to pinned host memory to improve large data-transfer efficiency, together with GPU Direct Memory Access (DMA) and runtime interoperability for direct device-pointer access. Standardized and scalable I/O is provided using openPMD and ADIOS2, supporting high-performance file I/O, in-memory data streaming, and in-situ analysis and visualization. Performance results on pre-exascale and exascale systems, including Frontier (OLCF-5) for up to 16,000 GPUs, demonstrate significant improvements in run time, scalability, and resource utilization for large-scale PIC MC simulations.
Cited by 0Open access
Jeremy J. Williams, Jordy Trilaksono, Stefan Costea et al. · ArXiv.org · 2026
Particle-in-Cell (PIC) Monte Carlo (MC) simulations are central to plasma physics but face increasing challenges on heterogeneous HPC systems due to excessive data movement, synchronization overheads, and inefficient utilization of multiple accelerators. In this work, we present a portable, multi-GPU hybrid MPI+OpenMP implementation of BIT1 that enables scalable execution on both Nvidia and AMD accelerators through OpenMP target tasks with explicit dependencies to overlap computation and communication across devices. Portability is achieved through persistent device-resident memory, an optimized contiguous one-dimensional data layout, and a transition from unified to pinned host memory to improve large data-transfer efficiency, together with GPU Direct Memory Access (DMA) and runtime interoperability for direct device-pointer access. Standardized and scalable I/O is provided using openPMD and ADIOS2, supporting high-performance file I/O, in-memory data streaming, and in-situ analysis and visualization. Performance results on pre-exascale and exascale systems, including Frontier (OLCF-5) for up to 16,000 GPUs, demonstrate significant improvements in run time, scalability, and resource utilization for large-scale PIC MC simulations.
Cited by 0Open access
Nathan Ellingwood, Youbing Yin, Matthew R. Smith et al. · 2010
Results and Conclusions Methods In this work we introduce a novel Diffeomorphic Multi-level Transform Composite method (DMTC) for accelerated computation of nonrigid masspreserving registration of lung images with large deformation on Graphics Processing Units (GPUs). The proposed method dramatically reduces computational time when compared to its single- and multi-threaded CPU counterparts, with the speedup factor ranging from 6 to 112. The significance of this work is two-fold. First, the DMTC achieves computation and memory efficiencies on GPUs, and together with the mass-preserving measure improves the accuracy of registration. Second, the improved computational efficiency is essential in analyzing data for population-based studies and translational science. This GPU implementation can be easily adapted for use with other non-mass-preserving similarity measures.
Cited by 0
Ao Zhang, Ser Gi Hong, S. C. Jeong et al. · Nuclear Engineering and Technology · 2025
Discrete ordinates (S N ) method with unstructured meshes is highly appropriate for high-fidelity modeling and simulation of radiation shielding problems with complicated geometries. However, the large number of unknowns resulting from discretization of the transport equation on the spatial, angular, and energy variables, necessitates the use of parallel computing to achieve efficient solutions. In this work, a GPU-based S N transport sweep algorithm combined with a GPU-parallel multi-group Krylov subspace solver has been proposed and implemented in the STRAUM (S N Transport for Radiation Analysis with Unstructured Meshes) code. A group chunk decomposition method within the framework of the multi-group Krylov subspace solver has been applied to STRAUM to leverage multi-GPU parallel computing. For the Kobayashi-like and reactor pressure vessel problems, STRAUM typically runs faster by factors of 100 ∼ 200 on a single NVIDIA GeForce RTX 4090 GPU and by factors of 70 ∼ 120 on a single NVIDIA GeForce RTX 3080 Ti GPU than on a single AMD Ryzen 9 7900X CPU core. For the simulations on dual-GPU systems, the group chunk decomposition method achieves parallel computing efficiencies greater than 90% without degradation in convergence except for cases using very coarse angular divisions. Besides, this method reduces per-GPU memory usage by more than 40% and enables STRAUM to effectively simulate problems with up to ten billion unknowns using two RTX 4090 GPUs.
Cited by 2Open access
Yizhi Huang, Yanlong Yin, Yan Liu et al. · 2021
This paper presents a heterogeneous collaborative computing framework for SGD-based Matrix Factorization, named HCC-MF. HCC-MF can train the feature matrix efficiently using multiple CPUs and GPUs. It performs collaborative computing with data parallelism, where a server CPU is in charge of management and synchronization and other heterogeneous worker CPUs and worker GPUs performs calculation with their data assignments. HCC-MF adopts two data partition strategies, “data partition with heterogeneous load balance” and “data partition with hidden synchronization.” We build a time cost model to guide the data distribution among multiple workers and we design several communication optimization techniques with consideration of datasets’ and processors’ characteristics. Experimental results indicate that HCC-MF can utilize more than 88% of the platform’s computing power, yielding a speedup of 2.9 compared with advanced SGD-based MF, CuMF_SGD, on large-scale data sets.
Cited by 3
Yifan Sun · 2020
GPUs have been accelerating a wide range of algorithms and applications with their massively parallel computing capabilities. Today, as GPU programs work under the close supervision of a CPU, computing platforms are usually equipped with both CPUs and GPUs. Most existing GPU programs underutilize the computing capabilities of the CPU since the CPU is frequently idle while GPUs are executing. Additionally, existing work primarily focuses on using a single GPU, thus failing to utilize the computing power of multi-GPU systems. In this dissertation, we propose a new heterogeneous computing paradigm--Collaborative Heterogeneous Computing. Collaborative Heterogeneous Computing leverages fine-grained CPU-GPU communication mechanisms in computation with the use of CPUs and multiple GPUs simultaneously. Collaborative Heterogeneous Computing can be divided into CPU-GPU collaborative computing and Multi-GPU collaborative computing. For CPU-GPU collaborative computing, we first categorize 7 CPU-GPU collaborative execution patterns. With the guidance of the summarized CPU-GPU execution patterns, we implement Hetero-Mark, a benchmark suite that enables users to explore CPU-GPU collaborative execution patterns. We also design and develop Multi2Sim-HSA, an HSA system emulator that emulates behaviors of CPUs and GPUs in CPU-GPU collaborative execution applications. In addition, we observe that CPU-GPU communication over the PCIe network causes congestion and can become a major performance bottleneck. To address this issue, we propose a priority-based PCIe scheduling algorithm that can improve device utilization and satisfy Quality of Service (QoS) requirements. For multi-GPU collaborative computing, we develop MGPUMark, a benchmark suite that explores multi-GPU collaborative execution and fine-grained inter-GPU communication. Benchmarks in MGPUMark enable exploration of inter-GPU communication patterns. We also design and develop MGPUSim, a high-performance, high-flexibility, and high-accuracy GPU simulator. MGPUSim can faithfully model behaviors of GPUs in multi-GPU collaborative computing. We identify that the main performance bottlenecks occur due to long-latency, low-bandwidth, inter-GPU interconnects. Thus, reducing inter-GPU communication traffic can improve multi-GPU system performance. We develop the Locality APIs, a set of GPU programming APIs that enables programmers to place the computing threads and associated data on a multi-GPU system to reduce inter-GPU traffic. We also introduce Progressive Page-Splitting Migration (PASI), a hardware-based solution that allows GPUs to adjust data placement to avoid inter-GPU traffic.--Author's abstract
Cited by 1Open access
Shayan Mohammadian, Ankit S. Kumar, Chongmin Song · SSRN Electronic Journal · 2024
Cited by 0Open access
Chen Zhang, Qiang Zhang, Zhuoshan Zhou et al. · 2026
Tensor parallelism (TP) in large-scale LLM inference and training introduces frequent collective operations that dominate inter-GPU communication. While in-switch computing, exemplified by NVLink SHARP (NVLS), accelerates collective operations by reducing redundant data transfer, its communication-centric design philosophy introduces the mismatch between its communication mode and the memory semantic requirement of LLM's computation kernel. Such a mismatch isolates the compute and communication phases, resulting in underutilized resources and limited overlap in multi-GPU systems. To address the limitation, we propose CAIS, the first ComputeAware In-Switch computing framework that aligns communication modes with computation's memory semantics requirement. CAIS consists of three integral techniques: (1) compute-aware ISA and microarchitecture extension to enable compute-aware in-switch computing. (2) merging-aware TB (Thread Block) coordination to improve the temporal alignment for efficient request merging. (3) graph-level dataflow optimizer to achieve a tight cross-kernel overlap. Evaluations on LLM workloads show that CAIS achieves <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$1.38 \times$</tex> average end-to-end training speedup over the SOTA NVLS-enabled solution, and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$1.61 \times$</tex> over T3, the SOTA compute-communicate overlap solutions but do not leverage NVLS, demonstrating its effectiveness in accelerating TP on multi-GPU systems.
Cited by 0
Seong-Hyeon Kang, Jungkyun Shin, Sangmin Kim et al. · Electronic Imaging · 2025
In a lens-based camera, light from an object is projected onto an image sensor through a lens. This projection involves various optical effects, including the point spread function (PSF), distortion, and relative illumination (RI), which can be modeled using lens design software such as Zemax and Code V. To evaluate lens performance, a simulation system can be implemented to apply these effects to a scene image. In this paper, we propose a lens-based multispectral image simulation system capable of handling images with 200 million pixels, matching the resolution of the latest mobile devices. Our approach optimizes the spatially variant PSF (SVPSF) algorithm and leverages a distributed multi-GPU system for massive parallel computing. The system processes 200M-pixel multispectral 2D images across 31 wavelengths ranging from 400 nm to 700 nm. We compare edge spread function, RI, and distortion results with Zemax, achieving approximately 99% similarity. The entire computation for a 200M-pixel, 31-wavelength image is completed in under two hours. This efficient and accurate simulation system enables pre-evaluation of lens performance before manufacturing, making it a valuable tool for optical design and analysis.
Cited by 0Open access
Ao Zhang, Ser Gi Hong, S. C. Jeong et al. · SSRN Electronic Journal · 2024
Cited by 0Open access
Shayan Mohammadian, Ankit S. Kumar, Chongmin Song · Computer Methods in Applied Mechanics and Engineering · 2025
This paper proposes a high-performance computing framework for large-scale elastodynamic analysis utilizing Graphics Processor Units (GPUs). The study adopts an octree algorithm for automatic mesh generation. The scaled boundary finite element method (SBFEM) is employed with the octree mesh, eliminating hanging nodes between octree cells with different sizes. This approach significantly reduces the computational cost and memory requirement by exploiting the limited number of master cells in a balanced octree grid, and is advantageous for GPU computation. The parallelization is achieved through mesh-partitioning techniques and message-passing-interface (MPI) directives, complemented by the NVIDIA Collective Communication Library (NCCL) for optimal point-to-point communication between GPUs in high-performance computing (HPC) facilities. The HPC framework is implemented for both explicit and implicit dynamic analysis. The preconditioned conjugate gradient method is employed for the equation solution in the implicit analysis. Numerical examples are presented for validation of the implementation and for demonstrating the capabilities of the GPU implementation. An image-based 3D model representing a portion of the Moon’s complex surface is simulated with a layered structure comprising of approximately 440 million degrees of freedom. Using the explicit solver, a speed-up of 865 is achieved on a single computational node equipped with eight NVIDIA A100 GPUs in parallel. A 3D virtual city comprising of approximately 61 million degrees of freedom is modelled using the implicit solver.
Cited by 7Open access
HU Yahui, ZHANG Chenkang, WANG Yuelin et al. · SHILAP Revista de lepidopterología · 2025
To address issues like coarse-grained resource allocation, lack of a unified vGPU view, and insufficient cross-cluster network awareness in multi-cluster GPU scheduling for large-scale deep learning tasks, a computing-network state-aware multi-cluster GPU scheduling platform was designed. Adopting a centralized architecture, fine-grained global resource orchestration was achieved through real-time perception and collaborative scheduling of cross-cluster computing resources and network states. Firstly, a multi-dimensional metric system covering device, cluster, vGPU, and network layers was built to collect real-time key data such as core utilization, memory, and bandwidth. Then a node-level vGPU orchestration module was designed, breaking the "job-to-cluster" limitation to realize precise "job-to-node" scheduling and improve GPU sharing efficiency. Experiments show that the platform enables real-time collection and visualization of multi-cluster vGPU and network information, and exhibits efficient resource management capabilities verified by DDPG reinforcement learning and the BestFit algorithm.
Cited by 0Open access
小山田, 徹 · Hosei University Repository (Hosei University) · 2017
The GPU (Graphics Processing Unit) has high computing power.The technology called GPGPU which uses GPU for general purpose computation is used in various fields.In this research, the theoretical model of multi GPU with plural GPUs construct, analyze and implement algorithms that operate on multi GPU.\nKeyword: multi GPU, theoretical model
Cited by 0Open access
Amirhossein Sojoodi, Ali Farazdaghi, Hamed Sharifian et al. · 2025
The concept of GPU-aware multi-path communication and heterogeneous computing is pivotal in enhancing HPC cluster performance. Heterogeneous computing utilizes various computing units to boost distributed applications, while multi-path communication frameworks optimize data transfer and reduce latency. This study proposes a collaborative Allreduce collective communication framework that utilizes heterogeneous computing and multi-path communication for multi-GPU systems. Leveraging GPU parallelism and high memory bandwidth, our approach optimizes both data transfer and computing power of the host system. We provide a proof-of-concept implementation and evaluation on a single-node multi-GPU system, comparing its performance to NCCL, UCC, and other algorithms. Our results show that our framework can improve Allreduce performance for large message sizes by up to 1.45x compared to NCCL. Our approach can be extended to various topologies and systems, with more extensive evaluations planned for the future.
Cited by 0