Samuel Irving · Civil War Book Review · 2019
In this dissertation, we explore multiple designs for a Distributed Transactional Memory framework for GPU clusters. Using Transactional Memory, we relieve the programmer of many concerns including 1) how to move data between many discrete memory spaces; 2) how to ensure data correctness when shared objects may be accessed by multiple devices; 3) how to prevent catastrophic warp divergence caused by atomic operations; 4) how to prevent catastrophic warp divergence caused by long-latency off-device communications; and 5) how to ensure Atomicity, Consistency, Isolation, Durability for programs with irregular memory accesses. Each of these concerns individually can be daunting to programmers who lack expert knowledge of the GPUs architectural quirks including the use of SIMD, weak memory model, and lack of direct access to a NIC. The goal of this work is to significantly reduce the programming effort required to realize performant GPU applications despite workload characteristics that are not favorable to the underlying architecture. Using our automatic concurrency control system, CUDA-DTM, programmers can convert some traditional applications to GPU applications in an afternoon that would have otherwise taken months to develop and debug. We analyze the performance and workload flexibility of CUDA-DTM, the first ever Distributed Transactional Memory framework written in CUDA for large scale GPU clusters. Transactional Memory has become an attractive concurrency control scheme for GPU applications with irregular memory access patterns due to its ability to avoid serializing threads while still maintaining programmability and preventing deadlocks. CUDA-DTM extends existing GPU Software Transactional Memory model to allow individual threads across many GPUs to initiate access to a Partitioned Global Address Space using a proposed scheme for GPU-to-GPU communication using CUDA-Aware MPI. CUDA-DTM allows programmers to treat individual GPU threads as though they were as flexible and independent as CPU threads, using a run-time system that automatically resolves conflicts, prevents warp-divergence, facilitates data movement between host and device memory spaces, and preserves data integrity. While CUDA-DTM will ensure applications run correctly and to completion without deadlocks or live-locks, there is no free lunch and programmers must be aware of the underlying architecture as well as locality of PGAS memory accesses to achieve the "100x" speedup that is so enticingly advertised by existing GPU literature. In fact, achieving the best-performing implementation for a given workload would likely require replacing each of the CUDA-DTM components with an application-specific design. However, using CUDA-DTM, programmers can rapidly test the suitability of GPU clusters
Cited by 0Open access
Irving, Samuel Oridge · Civil War Book Review · 2019
In this dissertation, we explore multiple designs for a Distributed Transactional Memory framework for GPU clusters. Using Transactional Memory, we relieve the programmer of many concerns including 1) how to move data between many discrete memory spaces; 2) how to ensure data correctness when shared objects may be accessed by multiple devices; 3) how to prevent catastrophic warp divergence caused by atomic operations; 4) how to prevent catastrophic warp divergence caused by long-latency off-device communications; and 5) how to ensure Atomicity, Consistency, Isolation, Durability for programs with irregular memory accesses. Each of these concerns individually can be daunting to programmers who lack expert knowledge of the GPUs architectural quirks including the use of SIMD, weak memory model, and lack of direct access to a NIC. The goal of this work is to significantly reduce the programming effort required to realize performant GPU applications despite workload characteristics that are not favorable to the underlying architecture. Using our automatic concurrency control system, CUDA-DTM, programmers can convert some traditional applications to GPU applications in an afternoon that would have otherwise taken months to develop and debug.\nWe analyze the performance and workload flexibility of CUDA-DTM, the first ever Distributed Transactional Memory framework written in CUDA for large scale GPU clusters. Transactional Memory has become an attractive concurrency control scheme for GPU applications with irregular memory access patterns due to its ability to avoid serializing threads while still maintaining programmability and preventing deadlocks. CUDA-DTM extends existing GPU Software Transactional Memory model to allow individual threads across many GPUs to initiate access to a Partitioned Global Address Space using a proposed scheme for GPU-to-GPU communication using CUDA-Aware MPI.\nCUDA-DTM allows programmers to treat individual GPU threads as though they were as flexible and independent as CPU threads, using a run-time system that automatically resolves conflicts, prevents warp-divergence, facilitates data movement between host and device memory spaces, and preserves data integrity. While CUDA-DTM will ensure applications run correctly and to completion without deadlocks or live-locks, there is no free lunch and programmers must be aware of the underlying architecture as well as locality of PGAS memory accesses to achieve the "100x" speedup that is so enticingly advertised by existing GPU literature. In fact, achieving the best-performing implementation for a given workload would likely require replacing each of the CUDA-DTM components with an application-specific design. However, using CUDA-DTM, programmers can rapidly test the suitability of GPU clusters for a given workload and study the performance of a working application to decide whether it is worth investing the significant programming effort required to write a distributed CUDA application.\nTo reduce CUDA-DTM's sensitivity to memory access partition locality, we relax CUDA-DTMS consistency model using BifurKTM (BKTM), a read-optimized Transactional Memory model, and a distributed shared memory system that allows GPU threads to access read-only copies of shared data elements with a bounded staleness. BKTM uses K-Opaque Approximate Consistency to reduce contention, allowing shared objects to be read and written by transactions simultaneously. BKTM uses a combination of Control Flow and Data Flow DTM: transactions that modify shared memory use Control Flow to avoid cache maintenance overheads and allow communications to be removed from the critical path; read-only transactions instead use the Data Flow model in which shared objects are loaded in advance into a local cache, allowing read-only transactions to hide long latencies from remote communications.
Cited by 0Open access
Jorge González‐Domínguez, Verónica Bolón‐Canedo, Borja Freire et al. · Information Sciences · 2019
Cited by 15Open access
Christian Osendorfer, Jie Tao, Carsten Trinitis et al. · 2004
This work describes ViSMI, a software distributed shared memory system for cluster systems connected via InfiniBand. ViSMI implements a kind of home-based lazy release consistency protocol, which uses a multiple-writer coherence scheme to alleviate the traffic introduced by false sharing. For further performance gain, InfiniBand features and optimized page invalidation mechanisms are applied in order to reduce synchronization overhead. First experimental results show that ViSMI introduces good performance comparable to similar software DSMs.
Cited by 6
Phyllis E. Crandall, J. Leichtl · 2002
The availability of multiprocessing workstations offers the opportunity to simultaneously apply both shared and distributed memory parallel processing techniques to a single application. It has been assumed that using the data sharing formalisms native to the specific machine would yield performance superior to those obtained with a more general message passing system. We compared the performance of pure message passing methods with a combination of message passing and shared memory data sharing techniques and found that the benefit of the added complexity of dual level programming is negligible or non-existent.
Cited by 0
Samuel Irving, Sui Chen, Lu Peng et al. · Lecture notes in computer science · 2019
Cited by 4
Kinson Chan · 2013
Within a decade, multicore processors emerged and revolutionised the world of computing. Nowadays, even a low-end computer comes with a multi-core processor and is capable running multiple threads simultaneously. It becomes impossible to make the best computation power out from a computer with a single-threaded program. Meanwhile, writing multi-threaded software is daunting to a lot of programmers as the threads share data and involve complicated synchronisation techniques such as locks and conditions. Software transactional memory is a promising alternative model that programmers simply need to understand transactional consistency and segment code into transactions. Programming becomes exciting again, without races, deadlocks and other issues that are common in lock-based paradigms. \n \nTo pursue high throughput, performance-oriented computers have several multicore processors per each. A processor’s cache is not directly accessible by the cores in other processors, leading to non-uniform latency when the threads share data. These computers no longer behave like the classical symmetric multiprocessor computers. Although old programs continue to work, they do not necessary benefit from the added cores and caches. Most software transactional memory implementations fall into this category. They rely on a centralised and shared meta-variable (like logical clock) in order to provide the single-lock atomicity. On a computer with two or more multicore processors, the single and shared meta-variable gets regularly updated by different processors. This leads to a tremendous amount of cache contentions. Much time is spent on inter-processor cache invalidations rather than useful computations. \n \nNevertheless, as computers with four processors or more are exponentially complex and expensive, people would desire solving sophisticated problems with several smaller computers whenever possible. Supporting software transactional consistency across multiple computers is a rarely explored research area. Although we have similar mature research topics such as distributed shared memory and distributed relational database, they have remarkably different characteristics so that most of the implementation techniques and tricks are not applicable to the new system. There are several existing distributed software transactional memory systems, but we feel there is much room for improvement. One crucial area is the conflict detection mechanism. Some of these systems make use of broadcast messages to commit transactions, which are certainly not scalable for large-scale clusters. Others use directories to direct messages to the relevant nodes only, but they also keep visible reader lists for invalidation per node. Updating a shared reader lists involves cache invalidations on processors. Reading shared data on such systems are more expensive compared to the conventional low-cost invisible reader validation systems. \n \nIn this research, we aim to have a distributed software transactional memory system, with distributed clock validation for conflict detection purpose. As preparation, we first investigate some issues such as concurrency control and conflict detection in single-node systems. Finally, we combine the techniques with a tailor-made cache coherence protocol that is differentiated from typical distributed shared memory.
Cited by 0Open access
J. Bader, Edgar Blumenthal, Marten Eckardt et al. · 2025
Cited by 0
M. Doggett, S. Laine, W. Hunt (editors et al. · 2015
Data sets of immense size are regularly generated on large scale computing resources. Even among more tra-ditional methods for acquisition of volume data, such as MRI and CT scanners, data which is too large to be effectively visualized on standard workstations is now commonplace. One solution to this problem is to employ a ‘visualization cluster, ’ a small to medium scale cluster dedicated to performing visualization and analysis of massive data sets generated on larger scale supercomputers. These clusters are designed to fit a different need than traditional supercomputers, and therefore their design mandates different hardware choices, such as increased memory, and more recently, graphics processing units (GPUs). While there has been much previous work on distributed memory visualization as well as GPU visualization, there is a relative dearth of algorithms which effectively use GPUs at a large scale in a distributed memory environment. In this work, we study a common visualization technique in a GPU-accelerated, distributed memory setting, and present performance characteristics when scaling to extremely large data sets. 1.
Cited by 0
Darshan Thaker, Vipin Chaudhary · 2003
Distributed shared memory systems have become popular as a means of utilizing clusters of com-puters for solving large applications. We have de-veloped a high-performance DSM at Wayne State University. To improve the performance of our DSM, we have developed a memory hierarchy sim-ulator that allows us to compare various techniques very quickly and with much less effort. This pa-per describes our simulator, DSMSim. We show that the simulator’s performance closely matches the real system and demonstrate potential perfor-mance gains of up to 60 percent after adding op-timization features to the simulator. The simula-tor also accepts the same code as the Software Dis-tributed Shared Memory. 1
Cited by 0
Arindam Banerji, Dinesh C. Kulkarni, John Tracey et al. · 2002
In order to exploit the latest advances in hardware technology, application developers need high-performance, easy-to-use cooperation tools that spun interconnections of standard hardware. Distributed shared memory has been proposed as such a cooperation tool, but performance problems have limited its usefulness. This paper argues that a new approach to distributed shared memory implementation can make it an effective tool in its own right and a foundation for other tools. It describes a prototype implementation that allows sharing of memory resources in workstation cluster. The prototype is based on an innovative, low-overhead messaging protocol which utilizes the high bandwidth of the underlying hardware, while adding very little latency overhead. Finally, the interface exported by this software is designed to function effectively as a substrate for a variety of cooperation tools.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">></ETX>
Cited by 8
Jan Meizner, M. Malawski · ArXiv.org · 2025
Paper presents and evaluates various mechanisms for remote access to memory in distributed systems based on two distinct HPC clusters. We are comparing solutions based on the shared storage and MPI (over Infiniband and Slingshot) to the local memory access. This paper also mentions medical use-cases that would mostly benefit from the described solution. We have found out that results for remote access esp. backed by MPI are similar to local memory access.
Cited by 0Open access
Emmanuel Cecchet · ACM SIGOPS Operating Systems Review · 2000
No abstract available.
Cited by 0
Upadhayaya, Niraj · OpenGrey (Institut de l'Information Scientifique et Technique) · 2006
Cited by 0Open access
Cyril Perrin, Emmanuel Cecchet · 2000
Web caches have become the standard method to ensure an acceptable quality for Web access. For a cache implemented on a single dedicated computer, the throughput is quickly limited by the network interface. SMP computers also have to face this limitation. In contrast, cluster-based Web caches allow the throughput to be increased by simply adding nodes (and therefore network interfaces) to the cluster. Using a Distributed Shared Memory (DSM), request-distribution algorithms are easier to design, since the locality of data is handled by the DSM. A memory mapped network, such as Scalable Coherent Interface (SCI), provides high bandwidth and low latency for efficient software DSM implementation. We show that efficient Web caches can actually be developed on a SCI-based cluster. Using a simple waiting-queue based model, we simulate the cluster behaviour using various kinds of workloads following a Zipf-like distribution. Both the bandwidth and the number of transactions per second appear to incr...
Cited by 0
Sutmann, G. · JuSER (Forschungszentrum Jülich) · 2001
Cited by 0
Johannes Gutleber · 2007
This diploma thesis demonstrates that it is possible to build a distributed shared memory system for clusters of heterogeneous workstations. Distributed or virtual shared memory provides a high level of programming abstraction when it comes to build distributed or parallel applications. It is a promising technology and we think that this paradigm could be extended to general distributed systems. The challenge in this case is to overcome the problems of network latencies, protocol overhead and diversity of platforms. We addressed the first problem by using a UDP based asynchronous reliable message transfer protocol which faces the end-to-end argument of middleware layer applications. The communication overhead could be reduced by using a relaxed consistency model, the lazy release consistency invalidate protocol, which is a software based derivation of the release consistency model. The latter one has been used in many multiprocessor architectures. The solution supports a multiple write...
Cited by 0
Gihan R. Mudalige, Michael B. Giles, Carlo Bertolli et al. · ACM SIGMETRICS Performance Evaluation Review · 2012
OP2 is an "active" library framework for the development and solution of unstructured mesh based applications. It aims to decouple the scientific specification of an application from its parallel implementation to achieve code longevity and near-optimal performance through re-targeting the backend to different multi-core/many-core hardware. This paper presents a predictive performance analysis and benchmarking study of OP2 on heterogeneous cluster systems. We first present the design of a new OP2 back-end that enables the execution of applications on distributed memory clusters, and benchmark its performance during the solution of a 1.5M and 26M edge-based CFD application written using OP2. Benchmark systems include a large-scale CrayXE6 system and an Intel Westmere/InfiniBand cluster. We then apply performance modeling to predict the application's performance on an NVIDIA Tesla C2070 based GPU cluster, enabling us to compare OP2's performance capabilities on emerging distributed memory heterogeneous systems. Results illustrate the performance benefits that can be gained through many-core solutions both on single-node and heterogeneous configurations in comparison to traditional homogeneous cluster systems for this class of applications.
Cited by 9
Siegfried Benkner, Viera Šipková · International Journal of Parallel Programming · 2003
Cited by 6
Sumit Roy, Vipin Chaudhary · 1998
This paper introduces Strings, a high performance dis-tributed shared memory system designed for clusters of symmetrical multiprocessors (SMPs). The distinguishing feature of this system is the use of a fully multi-threaded runtime system, written using POSIX threads. Strings also allows multiple application threads to be run on each node in a cluster. Since most modern UNIX systems can multiplex these threads on kernel level light weight processes, appli-cations written using Strings can use all the processors in a SMP machine. This paper describes some of the archi-tectural details of the system and analyzes the performance improvements with two example programs and a few bench-mark programs from the SPLASH-2 suite. 1.
Cited by 0