Charles J Archer · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · 2023
Methods, parallel computers, and computer program products are disclosed for remote direct memory access. Embodiments include transmitting, from an origin DMA engine on an origin compute node to a plurality target DMA engines on target compute nodes, a request to send message, the request to send message specifying a data to be transferred from the origin DMA engine to data storage on each target compute node; receiving, by each target DMA engine on each target compute node, the request to send message; preparing, by each target DMA engine, to store data according to the data storage reference and the data length, including assigning a base storage address for the data storage reference; sending, by one or more of the target DMA engines, an acknowledgment message acknowledging that all the target DMA engines are prepared to receive a data transmission from the origin DMA engine; receiving, by the origin DMA engine, the acknowledgement message from the one or more of the target DMA engines; and transferring, by the origin DMA engine, data to data storage on each of the target compute nodes according to the data storage reference using a single direct put operation.
Cited by 0Open access
Scott Pakin · 2011
Providing point-to-point messaging-passing semantics atop Put/Get hardware traditionally involves implementing a protocol comprising three network latencies. In this paper, we analyze the performance of an alternative implementation approach—receiver-initiated message passing—that eliminates one of the three network latencies. Performance measurements taken on the Cell Broadband Engine indicate that receiver-initiated message passing exhibits substantially lower latency than standard, sender-initiated message passing.
Cited by 0
Ryan E. Grant · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · 2023
A communication stack for providing remote direct memory access (RDMA) over a datagram network is disclosed. The communication stack has a user level interface configured to accept datagram related input and communicate with an RDMA enabled network interface card (NIC) via an NIC driver. The communication stack also has an RDMA protocol layer configured to supply one or more data transfer primitives for the datagram related input of the user level. The communication stack further has a direct data placement (DDP) layer configured to transfer the datagram related input from a user storage to a transport layer based on the one or more data transfer primitives by way of a lower layer protocol (LLP) over the datagram network.
Cited by 1Open access
J. Pinkerton, E. Deleganes · 2007
This document specifies an Internet standards track protocol for the Internet community, and requests discussion and suggestions for improvements. Please refer to the current edition of the "Internet Official Protocol Standards " (STD 1) for the standardization state and status of this protocol. Distribution of this memo is unlimited. This document analyzes security issues around implementation and use of the Direct Data Placement Protocol (DDP) and Remote Direct Memory Access Protocol (RDMAP). It first defines an architectural model for an RDMA Network Interface Card (RNIC), which can implement DDP or RDMAP and DDP. The document reviews various attacks against the resources defined in the architectural model and the countermeasures that can be used to protect the system. Attacks are grouped into those that can be mitigated by using secure communication channels across the network, attacks from Remote Peers, and attacks from Local
Cited by 0
Sameer Kumar · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · 2023
Methods and systems for monitoring remote transmissions of messages among a plurality of nodes are described. A processing element in a first node may allocate a sequence number to a request to read and/or update data in a second node. The processing element may be different from main processors of the first node. The processing element may send the message and the sequence number to the second node. The processing element may modify a status of the sequence number to an active state, indicating a transmission of the message is pending. The processing element may, in response to a response from the second node, modify the status of the sequence number to an inactive state, indicating a completed transmission of the message. The processing element may, in response to no response from the second node within a time period, resend the message and the sequence number to the second node.
Cited by 0Open access
Burmasov, Nazar, Adolf, Patrick · CERN Document Server (European Organization for Nuclear Research) · 2021
Over the last years both the Large Hadron Collider (LHC) experiments and the High Performance Computing (HPC) community showed an increasing interest to accommodate High Energy Physics (HEP) workloads on HPC facilities. This project is a part of a broader initiative for enabling HPC resources for HEP software. The project was carried in the context of the XRootD software framework, a storage technology widely used in the HEP community.
Cited by 0Open access
Group J. Pinkerton · 2007
This document specifies an Internet standards track protocol for the Internet community, and requests discussion and suggestions for improvements. Please refer to the current edition of the "Internet Official Protocol Standards " (STD 1) for the standardization state and status of this protocol. Distribution of this memo is unlimited. This document analyzes security issues around implementation and use of the Direct Data Placement Protocol (DDP) and Remote Direct Memory Access Protocol (RDMAP). It first defines an architectural model for an RDMA Network Interface Card (RNIC), which can implement DDP or RDMAP and DDP. The document reviews various attacks against the resources defined in the architectural model and the countermeasures that can be used to protect the system. Attacks are grouped into those that can be mitigated by using secure communication channels across the network, attacks from Remote Peers, and attacks from Local
Cited by 0
Xingda WEI, Rong CHEN, Haibo CHEN · SHILAP Revista de lepidopterología · 2018
Fast network devices with RDMA support have been price-compatible with traditional network primitives such as Ethernet,and it’s now widely deployed in modern data centers.RDMA can be used in two ways.Firstly,it can optimize the messaging primitive in distributed applications.The second way is to redesign the applications with RDMA’s onesided features.One-sided features provide high CPU utilizations and high network performance,but the system should be redesigned.The research progress of RDMA was introduced.An overview on the research efforts on using RDMA for distributed systems was presented.The works on how to use RDMA to redesign systems and the works on how to better leverage RDMA were included.The future research directions were also put forward.
Cited by 0Open access
Sebastian Surminski, Christian Niesler, Lucas Davi et al. · Lecture notes in computer science · 2023
Cited by 1
H. Huang, Kaixin Huang, Litong You et al. · 2018
For promising performance boost, recent trends of modern data centers tend to use Persistent Memory (PM) as storage and utilize the one-sided feature of Remote Direct Memory Access (RDMA) to directly access PM for I/O requests. However, accessing PM through one-sided RDMA faces two challenges: one-sided data races and remote data crash consistency. Existing systems that employ server-bypass mechanism either abandon server-bypass write to avoid the challenges at the cost of extra server loads, or support both server-bypass read/write through inefficient concurrency control without ensuring remote crash consistency. In this paper, we propose a novel server-bypass RDMA-to-PM framework, named Forca, to provide high concurrency and guarantee remote crash consistency at the same time. In Forca, we design an optimized log-structured mechanism to eliminate in-place updates, removing race conditions of one-sided RDMA and providing atomicity for each update simultaneously. We implement Forca as a generic module to support server-bypass RDMA to PM, and conduct experiments on a Forca-based key-value store called ForcaKV. The experiments show that Forca achieves higher throughput than state-of-the-art techniques by up to 1.4x under high concurrency scenarios, and its crash consistency mechanism incurs only 14.7%-19.1% time overhead.
Cited by 8
Nfsv C. Lever · 2015
draft-cel-nfsv4-rpcrdma-bidirection-01 Recent minor versions of NFSv4 work best when ONC RPC transports can send ONC RPC transactions in both directions. This document describes conventions that enable RPC-over-RDMA version 1 transport endpoints to interoperate when operation in both directions is necessary. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at
Cited by 0
Chuck Lever · 2015
Recent minor versions of NFSv4 work best when ONC RPC transports can
send ONC RPC transactions in both directions. This document describes
conventions that enable RPC-over-RDMA version 1 transport endpoints to
interoperate when operation in both directions is necessary.
Cited by 0
Lode Coene · 2007
This document describes the applicability of Remote Direct Memory Access Protocol (RDMAP) and the Direct Data Placement Protocol (DDP). It compares and contrasts the different transport options over IP that DDP can use, provides guidance to ULP developers on choosing between available transports and/or how to be indifferent to the specific transport layer used, compares use of DDP with direct use of the supporting transports, and compares DDP over IP transports with non-IP transports that support RDMA functionality.
Cited by 3Open access
R. Recio, B. Metzler, P. Culley et al. · 2007
This document specifies an Internet standards track protocol for the Internet community, and requests discussion and suggestions for improvements. Please refer to the current edition of the "Internet Official Protocol Standards " (STD 1) for the standardization state and status of this protocol. Distribution of this memo is unlimited. This document defines a Remote Direct Memory Access Protocol (RDMAP) that operates over the Direct Data Placement Protocol (DDP protocol). RDMAP provides read and write services directly to applications and enables data to be transferred directly into Upper Layer Protocol (ULP) Buffers without intermediate data copies. It also enables a
Cited by 0
Guillaume Ambal, George Hodgkins, Mark Madler et al. · Proceedings of the ACM on Programming Languages · 2026
Remote Direct Memory Access (RDMA) is a memory technology that allows remote devices to directly write to and read from each other's memory, bypassing components such as the CPU and operating system. This enables low-latency high-throughput networking, as required for many modern data centres, HPC applications and AI/ML workloads. However, baseline RDMA comprises a highly permissive weak memory model that is difficult to use in practice and has only recently been formalised. In this paper, we introduce the Library of Composable Objects (LOCO), a formally verified library for building multi-node objects on RDMA, filling the gap between shared memory and distributed system programming. LOCO objects are well-encapsulated and take advantage of the strong locality and the weak consistency characteristics of RDMA. They have performance comparable to custom RDMA systems (e.g. distributed maps), but with a far simpler programming model amenable to formal proofs of correctness. To support verification, we develop a novel modular declarative verification framework, called Mowgli, that is flexible enough to model multinode objects and is independent of a memory consistency model. We instantiate Mowgli with the RDMA memory model, and use it to verify correctness of LOCO libraries.
Cited by 0Open access
D. Gibson, Stuart Mills, Andrew MacLennan et al. · 2023
SpaceFibre [1] provides multi Gbit/s on-board networking for spaceflight applications. It has in-built quality-of-service (QoS); fault detection, isolation, and recovery (FDIR); and can run over electrical or fibre-optic cables. Using multiple lanes, it supports very high data rates. For example, with four lanes running at a lane signalling rate of 7.8125 Gbit/s, an overall link signalling rate of 31.25 Gbit/s is achieved. After adjusting for 8b/10b encoding and protocol overheads, this results in an approximate unidirectional data rate of 23.8 Gbit/s and bidirectional data rate of 23.1 Gbit/s.To support such high data rates in embedded systems with minimal processor overhead, this paper presents a SpaceFibre Endpoint that uses a Remote Direct Memory Access (RDMA) approach to provide zero-copy transferring of data between user-space applications over a SpaceFibre network.This paper first provides an overview of RDMA over SpaceFibre, then presents performance test results, and lastly describes an image transfer demonstration showing RDMA over SpaceFibre used in a real application.
Cited by 2
Randall Stewart · 2003
Cited by 0
Rémi Dulong · 2024
Les ordinateurs modernes sont construits autour de deux éléments : leur CPU et leur mémoire principale volatile, ou RAM. Depuis les années 1970, ce principe a été constamment amélioré pour offrir toujours plus de fonctionnalités et de performances. Dans cette thèse, nous étudions deux paradigmes de mémoire qui proposent de nouvelles façons d’interagir avec la mémoire dans les systèmes modernes : la mémoire non-volatile et les accès mémoire distants. Nous mettons en oeuvre des outils logiciels qui exploitent ces nouvelles approches afin de les rendre compatibles et d’exploiter leurs performances avec des applications concrètes. Nous analysons également l’impact des technologies utilisées, et les perspectives de leur évolution dans les années à venir. Pour la mémoire non-volatile, comme les performances de la mémoire sont essentielles pour atteindre le potentiel d’un CPU, cette fonctionnalité a historiquement été abandonnée. Même si les premiers ordinateurs ont été conçus avec des formes de mémoire non volatiles, les architectes informatiques ont commencé à utiliser la RAM volatile pour ses performances inégalées, et n’ont jamais remis en question cette décision pendant des années. Cependant, en 2019, Intel a commercialisé un nouveau composant appelé Optane DCPMM qui rend possible l’utilisation de NVMM. Ce produit propose une nouvelle façon de penser la persistance des données. Mais il remet également en question l’architecture de nos machines et la manière dont nous les programmons. Avec cette nouvelle forme de mémoire, nous avons implémenté NVCACHE, un cache en mémoire non-volatile qui permet d’accélérer les interactions avec des supports de stockage persistants plus lents, tels que les SSD. Nous montrons que NVCACHE est particulièrement performant pour les tâches qui nécessitent une granularité élevée des garanties de persistance, tout en étant aussi simple à utiliser que l’interface POSIX traditionnelle. Comparé aux systèmes de fichiers conçus pour NVMM, NVCACHE peut atteindre un débit similaire ou supérieur lorsque la mémoire non volatile est utilisée. De plus, NVCACHE permet aux programmes d’exploiter les performances de NVMM sans être limité par la quantité de NVMM installée sur la machine. Un autre changement majeur dans le paysage informatique a été la popularité des systèmes distribués. Alors que les machines ont individuellement tendance à atteindre des limites de performances, l’utilisation de plusieurs machines et le partage des tâches sont devenus la nouvelle façon de créer des ordinateurs puissants. Bien que ce mode de calcul permette d’augmenter le nombre de CPU utilisés simultanément, il nécessite une connexion rapide entre les noeuds de calcul. Pour cette raison, plusieurs protocoles de communication ont implémententé RDMA, un moyen de lire ou d’écrire directement dans la mémoire d’un serveur distant. RDMA offre de faibles latences et un débit élevé, contournant de nombreuses étapes de la pile réseau. Cependant, RDMA reste limité dans ses fonctionnalités natives. Par exemple, il n’existe pas d’équivalent de multicast pour les fonctions RDMA les plus efficaces. Grâce à un switch programmable (le switch Intel Tofino), nous avons implémenté un mode spécial pour RDMA qui permet de lire ou d’écrire sur plusieurs serveurs en même temps, sans pénalité de performances. Notre système appelé Byp4ss fait participer le switch aux transferts, en dupliquant les paquets RDMA. Grâce à Byp4ss, nous avons implémenté un protocole de consensus nommé DISMU. De par sa conception, DISMU est optimal en termes de latence et de débit, car il peut réduire au minimum le nombre de paquets échangés sur le réseau pour parvenir à un consensus. Enfin, en utilisant ces deux technologies, nous remarquons que les futures générations de matériel pourraient nécessiter une nouvelle interface pour les mémoires de toutes sortes, afin de faciliter l’interopérabilité dans des systèmes qui ont tendance à devenir de plus en plus hétérogènes et complexes. ABSTRACT Modern computers are built around two main parts: their Central Processing Unit (CPU), and their volatile main memory, or Random Access Memory (RAM). The basis of this architecture takes its roots in the 1970’s first computers. Since, this principle has been constantly upgraded to provide more functionnality and performance. In this thesis, we study two memory paradigms that drastically change the way we can interact with memory in modern systems: non-volatile memory and remote memory access. We implement software tools that leverage them in order to make them compatible and exploit their performance with concrete applications. We also analyze the impact of the technologies underlying these new memory medium, and the perspectives of their evolution in the coming years. For non-volatile memory, as the main memory performance is key to unlock the full potential of a CPU, this feature has historically been abandoned on the race for performance. Even if the first computers were designed with non-volatile forms of memory, computer architects started to use volatile RAM for its incomparable performance compared to durable storage, and never questioned this decision for years. However, in 2019 Intel released a new component called Optane DC Persistent Memory (DCPMM), a device that made possible the use of Non-Volatile Main Memory (NVMM). That product, by its capabilities, provides a new way of thinking about data persistence. Yet, it also challenges the hardware architecture used in our current machines and the way we program them. With this new form of memory we implemented NVCACHE, a cache designed for non-volatile memory that helps boosting the interactions with slower persistent storage medias, such as Solid State Drive (SSD). We find NVCACHE to be quite performant for workloads that require a high granularity of persistence guarantees, while being as easy to use as the traditional POSIX interface. Compared to file systems designed for NVMM, NVCACHE can reach similar or higher throughput when the non-volatile memory is used. In addition, NVCACHE allows the code to exploit NVMM performance while not being limited by the amount of NVMM installed in the machine. Another major change of in the computer landscape has been the popularity of distributed systems. As individual machines tend to reach performance limitations, using several machines and sharing workloads became the new way to build powerful computers. While this mode of computation allows the software to scale up the number of CPUs used simultaneously, it requires fast interconnection between the computing nodes. For that reason, several communication protocols implemented Remote Direct Memory Access (RDMA), a way to read or write directly into a distant machine’s memory. RDMA provides low latencies and high throughput, bypassing many steps of the traditional network stack. However, RDMA remains limited in its native features. For instance, there is no advanced multicast equivalent for the most efficient RDMA functions. Thanks to a programmable switch (the Intel Tofino), we implemented a special mode for RDMA that allows a client to read or write in multiple servers at the same time, with no performance penalty. Our system called Byp4ss makes the switch participate in transfers, duplicating RDMA packets. On top of Byp4ss, we implement a consensus protocol named DISMU, which shows the typical use of Byp4ss features and its impact on performance. By design, DISMU is optimal in terms of latency and throughput, as it can reduce to the minimum the number of packets exchanged through the network to reach a consensus. Finally, by using these two technologies, we notice that future generations of hardware may require a new interface for memories of all kinds, in order to ease the interoperability in systems that tend to get more and more heterogeneous and complex.
Cited by 0Open access
Masoud Hemmatpour, Bartolomeo Montrucchio, Maurizio Rebaudengo · Applied Sciences · 2018
Distributed systems are commonly built under the assumption that the network is the primary bottleneck, however this assumption no longer holds by emerging high-performance RDMA enabled protocols in datacenters. Designing distributed applications over such protocols requires a fundamental rethinking in communication components in comparison with traditional protocols (i.e., TCP/IP). In this paper, communication paradigms in existing systems and new possible paradigms have been investigated. Advantages and drawbacks of each paradigm have been comprehensively analyzed and experimentally evaluated. The experimental results show that writing the requests to server and reading the response presents up to 10 times better performance comparing to other communication paradigms. To further expand the investigation, the proposed communication paradigm has been substituted in a real-world distributed application, and the performance has been enhanced up to seven times.
Cited by 3Open access
Prilop, Meiko · reposiTUm (TU Wien) · 2025
Cloud computing, and serverless computing in particular, are transforming traditional software architecture by making components and even entire systems available on scalable resources as needed. Function-as-a-Service (FaaS) is a specific paradigm in which users make function code available in the cloud and operate them there in an event-driven manner. One of the fundamental performance hurdles of FaaS is the overhead introduced by state-passing. Serverless functions are typically short-lived and stateless. If one intends to pass state along function chaining, inefficient messaging protocols are necessary to achieve statefulness. In that sense, caching appears as a feasible solution to persist state between function executions. As caching introduces latency when interacting with the cache, we focus on integrating high-performance computing protocols in a common cloud stack, to efficiently gap this pitfall. Remote Direct Memory Access (RDMA) allows to bypass the CPU and to directly insert data into the memory of a peer computing instance, saving CPU cycles and thus reducing latency experienced in communication. Within this thesis, we present the design and implementation of a dedicated caching middleware layer for communication of functions in a FaaS towards a distributed caching system using RDMA. This middleware abstraction layer enables us to connect functions on a FaaS to the cache without introducing additional overhead for the functions. The middleware layer takes values to be written and stores or reads them from the cache using RDMA communication. We design the system with the aim of being functional for various hardware requirements and integrating seamlessly with established cloud technologies. In this context, we identify a software abstraction for RDMA called SoftiWARP that allows us to integrate RDMA without relying on specialized hardware. We evaluate the system we have designed with the comparable state-of-the-art caching system Redis. While we experience significant overhead for small data values of 40 bytes, we observe up to 91% improved latency values for large data values greater than 1 MB.Our results indicate that the integration of high-performance protocols, even if abstracted by software, represents an advantageous further development for the design of serverless cloud concepts.
Cited by 0Open access