Skip to content
Free planNo credit card requiredStart writing
AI for Research home
AI for Research home
Dashboard
Sign inGet started

Search 250 million research papers

Read the abstract, see who wrote it and who cites it, and get the citation in APA, MLA, Chicago or Harvard.

About 2,319 papers for “GPU clusters”

Rail: ReArranging Inter-GPU Links for GPU-Centric Clusters

Haixin Nan, Jun Xu, P. Cao et al. · IEEE Transactions on Networking · 2026

In modern GPU-centric clusters, large-scale AI training relies on two distinct communication domains: a high-bandwidth intra-node domain using proprietary interconnects (e.g., NVLink), and a scale-out inter-node network domain (e.g., RDMA). We observe that the widely-used ring algorithm, often create a significant load imbalance across these domains. This leads to the counter-intuitive scenario where the expensive, high-bandwidth intra-node domain becomes a performance bottleneck, while the inter-node network remains underutilized. This inefficiency is further exacerbated by the disparity in bandwidth provisioning: inter-node network bandwidth is generally more cost-effective and accessible, whereas intra-node bandwidth is often proprietary and more costly to scale. To address this fundamental imbalance, we propose RAIL, aimed at resolving the intra-node bottleneck by strategically rearranging inter-GPU communication paths. This rebalancing ensures that traffic loads are appropriately matched with the distinct transmission capabilities of each domain, thereby maximizing overall communication performance. RAIL incorporates a Load Distributing Strategy (LDS) that can accurately partition physical nodes into logical nodes based on the a transmission capabilities of both domains, shifting excess traffic from the overloaded intra-node domain to the underutilized network domain. Additionally, the Intra-Rail Strategy (IRS) leverages topological characteristics to ensure optimal communication paths through the network domain between logical nodes. Our evaluation demonstrates that RAIL effectively mitigates congestion and achieves a 30.7% average increase in collective communication bus bandwidth compared to the widely-used NCCL solution.

Cited by 0

GPU-centered parallel model on heterogeneous multi-GPU clusters

Feng Wang · 2012

On the multi-GPU cluster platform, it is difficult to use the full compute power of the CPUs. One of the reasons is that the traditional parallel models based on the homogeneous platform is not suitable to the heterogeneous platform. We research and develop the GPU-centered parallel model to control the CPUs using more fine granularity. This model decreases the idle time of the CPUs introduced by the work load unbalance significantly. Our experiments show that this model can achieve higher performance than the traditional node-centered parallel model for some real applications. The efficiency of the LINPACK benchmark using GPU-centered parallel model is 5.34% higher than the node-centered parallel model.

Cited by 1

Data Parallel Skeletons for GPU Clusters and Multi-GPU Systems

Ernsting Steffen, Kuchen Herbert · Advances in parallel computing · 2012

CUDA made general-purpose computing for Graphics Processing Units (GPU) popular. But still GPU programming is error-prone and there are a lot of peculiarities to be considered for developing efficient GPU accelerated applications. Algorithmic skeletons encapsulate typical parallel programming patterns. They have emerged to be an efficient approach to simplifying the development of parallel and distributed applications. In this paper, we present an extension of our skeleton library Muesli in terms of GPU accelerated data parallel skeletons for multi-GPU systems and GPU clusters using CUDA. Besides the computation on the GPU, these skeletons also fully hide data transfer between different GPU devices as well as network transfer between different compute nodes. Experimental results demonstrate competitive performance results for some example applications.

Cited by 10

Framework for Heterogeneous CPU/GPU Clusters

Snucl An Opencl, Jaejin Lee · 2016

6aP`p M6aP`p M6aP`p M6aP`p BS06Ee aP`p EeBQ!mWF<1hN.i[W`pV2lT.X]O4fW?tJ(*+XGR'V,lK]_nVCklK>On O#J3#/aP`p?l`p'WEerj08lLaP`p)?l`pW

Cited by 0

Seismic wave modelling on GPU clusters

Anders Solberg Pedersen · Duo Research Archive (University of Oslo) · 2019

Vi utvikler et program for å simulere seismiske bølger på GPU-klynger ved hjelp av CUDA og MPI. Ved å bruke spektralelementmetoden til å løse den elastiske bølgelikningen i 3D kan problemet paralleliseres og løses effektivt. Implementasjonen ble tested med opp til 16 Nvidia P100 GPUer og skaleringsoppførselen viser en 14% økning i kjøretid mellom 1 og 16 GPUer. Den numeriske nøyaktigheten undersøkes ved å sammenlikne med den analytiske løsningen og med annen implementasjon i et isotropisk homogent medium med en uniform mesh. Begge sammenlikningene bekrefter at implementasjoner er korrekt.

Cited by 0Open access

Computing data cubes over GPU clusters.

Lucas Henrique Moreira Silva · 2018

The data cube is a fundamental relational operator for decision support systems, thus very important for analytics. Unfortunately, a full data cube with all of its tuples has exponential complexity in terms of runtime and memory consumption as the dimensions increase linearly, so algorithms to reduce query response times continue under development. The problem stated in this work is: how can we reduce complex multidimensional queries response times from high dimensional data cubes? The problem is aggravated if recurrent updates occur and if there is a huge volume of high dimensional data to be managed. The hypothesis of this work is that clusters of CPU-GPU devices can speedup queries from high dimensional holistic data cubes that are updated constantly. The alternative solution presented in this work, named JCL-GPU-Cubing, partitions the base relation into multiple independent sub- cubes. These multiple sub-cubes represent a partial data cube to reduce the exponentiality and they are used to perform queries in CPU or in CPU-GPU computer architectures efficiently. The experimental evaluations using complex multidimensional queries demonstrated that the CPU cluster version scaled well when the base relation increased and the CPU-GPU version outperformed the CPU only version in certain scenarios.

Cited by 1

Distributed GPGPU on Cloud GPU Clusters

Schuchardt · Digital Library of the University of Innsbruck (University of Innsbruck) · 2018

At this point, I would like to thank my three supervisors for their support and patience.Since I wrote this thesis as part of a term abroad, it was necessary to start and finish it at the LFU in Innsbruck and working on a predefined, separate part at JAIST in Japan.Compared with other theses, this added a lot of additional administrative tasks, made communications more difficult and prolonged the processing time.

Cited by 0Open access

Distributed Transactional Memory for GPU Clusters

Samuel Irving · Civil War Book Review · 2019

In this dissertation, we explore multiple designs for a Distributed Transactional Memory framework for GPU clusters. Using Transactional Memory, we relieve the programmer of many concerns including 1) how to move data between many discrete memory spaces; 2) how to ensure data correctness when shared objects may be accessed by multiple devices; 3) how to prevent catastrophic warp divergence caused by atomic operations; 4) how to prevent catastrophic warp divergence caused by long-latency off-device communications; and 5) how to ensure Atomicity, Consistency, Isolation, Durability for programs with irregular memory accesses. Each of these concerns individually can be daunting to programmers who lack expert knowledge of the GPUs architectural quirks including the use of SIMD, weak memory model, and lack of direct access to a NIC. The goal of this work is to significantly reduce the programming effort required to realize performant GPU applications despite workload characteristics that are not favorable to the underlying architecture. Using our automatic concurrency control system, CUDA-DTM, programmers can convert some traditional applications to GPU applications in an afternoon that would have otherwise taken months to develop and debug. We analyze the performance and workload flexibility of CUDA-DTM, the first ever Distributed Transactional Memory framework written in CUDA for large scale GPU clusters. Transactional Memory has become an attractive concurrency control scheme for GPU applications with irregular memory access patterns due to its ability to avoid serializing threads while still maintaining programmability and preventing deadlocks. CUDA-DTM extends existing GPU Software Transactional Memory model to allow individual threads across many GPUs to initiate access to a Partitioned Global Address Space using a proposed scheme for GPU-to-GPU communication using CUDA-Aware MPI. CUDA-DTM allows programmers to treat individual GPU threads as though they were as flexible and independent as CPU threads, using a run-time system that automatically resolves conflicts, prevents warp-divergence, facilitates data movement between host and device memory spaces, and preserves data integrity. While CUDA-DTM will ensure applications run correctly and to completion without deadlocks or live-locks, there is no free lunch and programmers must be aware of the underlying architecture as well as locality of PGAS memory accesses to achieve the "100x" speedup that is so enticingly advertised by existing GPU literature. In fact, achieving the best-performing implementation for a given workload would likely require replacing each of the CUDA-DTM components with an application-specific design. However, using CUDA-DTM, programmers can rapidly test the suitability of GPU clusters

Cited by 0Open access

Large-scale FFT on GPU clusters

Yifeng Chen, Xiang Cui, Hong Mei · 2010

A GPU cluster is a cluster equipped with GPU devices. Excellent acceleration is achievable for computation-intensive tasks (e. g. matrix multiplication and LINPACK) and bandwidth-intensive tasks with data locality (e. g. finite-difference simulation). Bandwidth-intensive tasks such as large-scale FFTs without data locality are harder to accelerate, as the bottleneck often lies with the PCI between main memory and GPU device memory or the communication network between workstation nodes. That means optimizing the performance of FFT for a single GPU device will not improve the overall performance. This paper uses large-scale FFT as an example to show how to achieve substantial speedups for these more challenging tasks on a GPU cluster. Three GPU-related factors lead to better performance: firstly the use of GPU devices improves the sustained memory bandwidth for processing large-size data; secondly GPU device memory allows larger subtasks to be processed in whole and hence reduces repeated data transfers between memory and processors; and finally some costly main-memory operations such as matrix transposition can be significantly sped up by GPUs if necessary data adjustment is performed during data transfers. This technique of manipulating array dimensions during data transfer is the main technical contribution of this paper. These factors (as well as the improved communication library in our implementation) attribute to 24.3x speedup with respect to FFTW and 7x speedup with respect to Intel MKL for 4096 3D single-precision FFT on a 16-node cluster with 32 GPUs. Around 5x speedup with respect to both standard libraries are achieved for double precision.

Cited by 85

Distributed Transactional Memory for GPU Clusters

Irving, Samuel Oridge · Civil War Book Review · 2019

In this dissertation, we explore multiple designs for a Distributed Transactional Memory framework for GPU clusters. Using Transactional Memory, we relieve the programmer of many concerns including 1) how to move data between many discrete memory spaces; 2) how to ensure data correctness when shared objects may be accessed by multiple devices; 3) how to prevent catastrophic warp divergence caused by atomic operations; 4) how to prevent catastrophic warp divergence caused by long-latency off-device communications; and 5) how to ensure Atomicity, Consistency, Isolation, Durability for programs with irregular memory accesses. Each of these concerns individually can be daunting to programmers who lack expert knowledge of the GPUs architectural quirks including the use of SIMD, weak memory model, and lack of direct access to a NIC. The goal of this work is to significantly reduce the programming effort required to realize performant GPU applications despite workload characteristics that are not favorable to the underlying architecture. Using our automatic concurrency control system, CUDA-DTM, programmers can convert some traditional applications to GPU applications in an afternoon that would have otherwise taken months to develop and debug.\nWe analyze the performance and workload flexibility of CUDA-DTM, the first ever Distributed Transactional Memory framework written in CUDA for large scale GPU clusters. Transactional Memory has become an attractive concurrency control scheme for GPU applications with irregular memory access patterns due to its ability to avoid serializing threads while still maintaining programmability and preventing deadlocks. CUDA-DTM extends existing GPU Software Transactional Memory model to allow individual threads across many GPUs to initiate access to a Partitioned Global Address Space using a proposed scheme for GPU-to-GPU communication using CUDA-Aware MPI.\nCUDA-DTM allows programmers to treat individual GPU threads as though they were as flexible and independent as CPU threads, using a run-time system that automatically resolves conflicts, prevents warp-divergence, facilitates data movement between host and device memory spaces, and preserves data integrity. While CUDA-DTM will ensure applications run correctly and to completion without deadlocks or live-locks, there is no free lunch and programmers must be aware of the underlying architecture as well as locality of PGAS memory accesses to achieve the "100x" speedup that is so enticingly advertised by existing GPU literature. In fact, achieving the best-performing implementation for a given workload would likely require replacing each of the CUDA-DTM components with an application-specific design. However, using CUDA-DTM, programmers can rapidly test the suitability of GPU clusters for a given workload and study the performance of a working application to decide whether it is worth investing the significant programming effort required to write a distributed CUDA application.\nTo reduce CUDA-DTM's sensitivity to memory access partition locality, we relax CUDA-DTMS consistency model using BifurKTM (BKTM), a read-optimized Transactional Memory model, and a distributed shared memory system that allows GPU threads to access read-only copies of shared data elements with a bounded staleness. BKTM uses K-Opaque Approximate Consistency to reduce contention, allowing shared objects to be read and written by transactions simultaneously. BKTM uses a combination of Control Flow and Data Flow DTM: transactions that modify shared memory use Control Flow to avoid cache maintenance overheads and allow communications to be removed from the critical path; read-only transactions instead use the Data Flow model in which shared objects are loaded in advance into a local cache, allowing read-only transactions to hide long latencies from remote communications.

Cited by 0Open access

Productive programming of GPU clusters with OmpSs

Bueno Hedo, Javier, Planas, Judit, Duran González, Alejandro et al. · RECERCAT (Consorci de Serveis Universitaris de Catalunya)

Clusters of GPUs are emerging as a new computational\nscenario. Programming them requires the use of\nhybrid models that increase the complexity of the applications,\nreducing the productivity of programmers.\nWe present the implementation of OmpSs for clusters of\nGPUs, which supports asynchrony and heterogeneity for task\nparallelism. It is based on annotating a serial application with\ndirectives that are translated by the compiler. With it, the same\nprogram that runs sequentially in a node with a single GPU\ncan run in parallel in multiple GPUs either local (single node)\nor remote (cluster of GPUs). Besides performing a task-based\nparallelization, the runtime system moves the data as needed\nbetween the different nodes and GPUs minimizing the impact\nof communication by using affinity scheduling, caching, and\nby overlapping communication with the computational task.\nWe show several applicactions programmed with OmpSs\nand their performance with multiple GPUs in a local node\nand in remote nodes. The results show good tradeoff between\nperformance and effort from the programmer.

Cited by 2Open access

Hwu, “Gpu clusters for high-performance computing

Volodymyr V. Kindratenko, Jeremy J. Enos, Guochun Shi et al. · 2014

Abstract—Large-scale GPU clusters are gaining popularity in the scientific computing community. However, their deployment and production use are associated with a number of new challenges. In this paper, we present our efforts to address some of the challenges with building and running GPU clusters in HPC environments. We touch upon such issues as balanced cluster architecture, resource sharing in a cluster environment, programming models, and applications for GPU clusters. I.

Cited by 0

Parallel rendering on hybrid multi-GPU clusters

Eilemann, Stefan, Bilgili, Ahmet, Abdellah, Marwan et al. · Zurich Open Repository and Archive (University of Zurich) · 2012

Achieving efficient scalable parallel rendering for interactive visualization applications on medium-sized graphics clusters remains a challenging problem. Framerates of up to 60hz require a carefully designed and fine-tuned parallel rendering implementation that fits all required operations into the 16ms time budget available for each rendered frame. Furthermore, modern commodity hardware embraces more and more a NUMA architecture, where multiple processor sockets each have their locally attached memory and where auxiliary devices such as GPUs and network interfaces are directly attached to one of the processors. Such so called fat NUMA processing and graphics nodes are increasingly used to build cost-effective hybrid shared/distributed memory visualization clusters. In this paper we present a thorough analysis of the asynchronous parallelization of the rendering stages and we derive and implement important optimizations to achieve highly interactive framerates on such hybrid multi-GPU clusters. We use both a benchmark program and a real-world scientific application used to visualize, navigate and interact with simulations of cortical neuron circuit models.

Cited by 0Open access