Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “memory mapping”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Enabling Scalable and Extensible Memory-mapped Datastores in Userspace

Exascale workloads are expected to incorporate data-intensive processing in close coordination with traditional physics simulations. These emerging scientific, data-analytics and machine learning applications need to access a wide variety of datastores in flat files and structured databases. Programmer productivity is greatly enhanced by mapping datastores into the application process's virtual memory space to provide a unified “in-memory” interface. Currently, memory mapping is provided by system software primarily designed for generality and reliability. However, scalability at high concurrency is a formidable challenge on exascale systems. Also, there is a need for extensibility to support new datastores potentially requiring HPC data transfer services. In this article, we present UMap , a scalable and extensible userspace service for memory-mapping datastores. Furthermore, through decoupled queue management, concurrency aware adaptation, and dynamic load balancing, UMap enables application performance to scale even at high concurrency. We evaluate UMap in data-intensive applications, including sorting, graph traversal, database operations, and metagenomic analytics. Our results show that UMap as a userspace service outperforms an optimized kernel-based service across a wide range of intra-node concurrency by 1.22-1.9 × . We performed two case studies to demonstrate UMap 's extensibility. First, a new datastore residing in remote memory is incorporated into UMap as an application-specific plugin. Second, we present a persistent memory allocator Metall built atop UMap for unified storage/memory.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

UMap: An application-oriented user level memory mapping library

Exploiting the prominent role of complex memories in exascale node architecture, the UMap page fault handler offers new capabilities to access large memory-mapped data sets directly. UMap provides flexible configuration options to customize page handling to each application, including analysis of massive observational and simulation data sets. The high-performance design features I/O decoupling, dynamic load balancing, and application-level controls. Page faults triggered by application threads and processes accessing data mapped to a UMapp’ed region are handled via the Linux userfaultfd protocol, an asynchronous message-oriented kernel-user communication mechanism that avoids the context switch penalty of traditional signal fault handlers. UMap is fully open source. In this paper, we give an overview of the UMap library architecture, its extensible plugin architecture, and the use/performance of UMap in emerging heterogeneous memory hierarchies such as near-node Non-volatile Memory (NVM) and network attached memories. We highlight new capabilities in two pagefault management plugins, the NetworkStore and SparseStore. We demonstrate the integration between UMap and multiple ECP products including Caliper, Metall, ZFP, Mochi, and Ripples.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Optimizing Management of Persistent Data Structures in High-Performance Analytics

Large-scale data analytics workflows ingest massive input data into various data structures, including graphs and key-value datastores. These data structures undergo multiple transformations and computations and are typically reused in incremental and iterative analytics workflows. Persisting in-memory views of these data structures enables reusing them beyond the scope of a single program run while avoiding repetitive raw data ingestion overheads. Memory-mapped I/O enables persisting in-memory data structures without data serialization and deserialization overheads. However, memory-mapped I/O lacks the key feature of persisting consistent snapshots of these data structures for incremental ingestion and processing. The obstacles to efficient virtual memory snapshots using memory-mapped I/O include background writebacks outside the application’s control, and the significantly high storage footprint of such snapshots. To address these limitations, we present Privateer, a memory and storage management tool that enables storage-efficient virtual memory snapshotting while also optimizing snapshot I/O performance. Here, we integrated Privateer into Metall, a state-of-the-art persistent memory allocator for C++, and the Lightning Memory-Mapped Database (LMDB), a widely-used key-value datastore in data analytics and machine learning. Privateer optimized application performance by 1.22× when storing data structure snapshots to node-local storage, and up to 16.7× when storing snapshots to a parallel file system. Privateer also optimizes storage efficiency of incremental data structure snapshots by up to 11× using data deduplication and compression.

Computer science↗
DOE OSTI2024

Computing Sparse Tensor Decompositions via Chapel and C++/MPI Interoperability without Intermediate I/O

We extend an existing approach for efficient use of shared mapped memory across Chapel and C++ for graph data stored as 1-D arrays to sparse tensor data stored using a combination of 2-D and 1-D arrays. We describe the specific extensions that provide use of shared mapped memory tensor data for a particular C++ tensor decomposition tool called GentenMPI. We then demonstrate our approach on several real-world datasets, providing timing results that illustrate minimal overhead incurred using this approach. Finally, we extend our work to improve memory usage and provide convenient random access to sparse shared mapped memory tensor elements in Chapel, while still being capable of leveraging high performance implementations of tensor algorithms in C++.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Metall: A persistent memory allocator for data-centric analytics

Data analytics applications transform raw input data into analytics-specific data structures before performing analytics. Unfortunately, such data ingestion steps are often more expensive than analytics. In addition, various types of NVRAM devices are already used in many HPC systems today. Such devices will be useful for storing and reusing data structures beyond a single process life cycle. We developed Metall, a persistent memory allocator built on top of the memory-mapped file mechanism. Metall enables applications to transparently allocate custom C++ data structures into various types of persistent memories. Metall incorporates a concise and high-performance memory management algorithm inspired by Supermalloc and the rich C++ interface developed by Boost.Interprocess library. On a dynamic graph construction workload, Metall achieved up to 11.7x and 48.3x performance improvements over Boost.Interprocess and memkind (PMEM kind), respectively. We also demonstrate Metall’s high adaptability by integrating Metall into a graph processing framework, GraphBLAS Template Library. Here this study’s outcomes indicate that Metall will be a strong tool for accelerating future large-scale data analytics by allowing applications to leverage persistent memory efficiently.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Tiling Framework for Heterogeneous Computing of Matrix based Tiled Algorithms

Tiling matrix operations can improve the load balancing and performance of applications on heterogeneous computing resources. Writing a tile-based algorithm for each operation with a traditional, hand-tuned tiling approach that uses for loops in C/C++ is cumbersome and error prone. Moreover, it must enable and support the heterogeneous memory management of data objects and also explore architecture-supported, native, tiled-data transfer APIs instead of copying the tiled data to continuous memory before the data transfer. The tiling framework provides a tiled data structure for heterogeneous memory mapping and parameterization to a heterogeneous task specification API. We have integrated our tiled framework into MatRIS (Math kernels library using IRIS). IRIS is a heterogeneous run-time framework with a heterogeneous programming model, memory model, and task execution model. Experiments reveal that the tiled framework for BLAS operations has improved the programmability of tiled BLAS and improved performance by ~20% when compared against the traditional method that copies the data to continuous memory locations for heterogeneous computing.

Miniskar, Narasinga Rao↗
DOE OSTI2026

rustpix

rustpix is a high-performance, open-source Rust library with first-class Python bindings (via PyO3) for processing pixel-detector data in neutron imaging. It targets time-stamping detectors such as Timepix3 (TPX3) at ORNL's Spallation Neutron Source (VENUS beamline), where each detected neutron deposits charge across a cluster of pixels within a very high-rate event stream (96M+ hits/sec). rustpix parses TPX3 event data in parallel using memory-mapped I/O, offers four interchangeable clustering algorithms (ABS adjacency-based search, DBSCAN, graph/union-find connected components, and a parallel grid method), and extracts weighted, super-resolved centroids to produce neutron-event lists. A streaming architecture lets it process files larger than available memory. rustpix is distributed as a pip-installable Python package (with NumPy integration), Rust crates, a command-line tool, and an interactive GUI; it writes HDF5, Apache Arrow, and CSV; and it is designed to extend to TPX4 and other detector types. Released as open-source under the MIT License.

Zhang, Chen [Oak Ridge National Laboratory (ORNL),↗
DOE OSTI2021

UPC++ as_eager Working Group Draft, Revision 2020.6.2

This draft proposes an extension for a new future-based completion variant that can be more effectively streamlined for RMA and atomic access operations that happen to be satisfied at runtime using purely node-local resources. Many such operations are most efficiently performed synchronously using load/store instructions on shared-memory mappings, where the actual access may only require a few CPU instructions. In such cases we believe it’s critical to minimize the overheads imposed by the UPC++ runtime and completion queues, in order to enable efficient operation on hierarchical node hardware using shared-memory bypass. The new upcxx::{source,operation}_cx::as_eager_future() completion variant accomplishes this goal by relaxing the current restriction that future-returning access operations must return a non-ready future whose completion is deferred until a subsequent explicit invocation of user-level progress. This relaxation allows access operations that are completed synchronously to instead return a ready future, thereby avoiding most or all of the runtime costs associated with deferment of future completion and subsequent mandatory entry into the progress engine. We additionally propose to make this new as_eager_future() completion variant the new default completion for communication operations that currently default to returning a future. This should encourage use of the streamlined variant, and may provide performance improvements to some codes without source changes. A mechanism is proposed to restore the legacy behavior on-demand for codes that might happen to rely on deferred completion for correctness. Finally, we propose a new as_eager_promise() completion variant that extends analogous improvements to promise-based completion, and corresponding changes to the default behavior of as_promise().

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

JTAG-based PLC memory acquisition framework for industrial control systems

In industrial control systems (ICS), programmable logic controllers (PLC) are the embedded devices that directly control and monitor critical industrial infrastructure processes such as nuclear plants and power grid stations. Cyberattacks often target PLCs to sabotage a physical process. A memory forensic analysis of a suspect PLC can answer questions about an attack, including compromised firmware and manipulation of PLC control logic code and I/O devices. Given physical access to a PLC, collecting forensic information from the PLC memory at the hardware-level is risky and challenging. It may cause the PLC to crash or hang since PLCs have proprietary, legacy hardware with heterogeneous architecture. This paper addresses this research problem and proposes a novel JTAG (Joint Test Action Group)-based framework, Kyros, for reliable PLC memory acquisition. Kyros systematically creates a JTAG profile of a PLC through hardware assessment, JTAG pins identification, memory map creation, and optimizing acquisition parameters. It also facilitates the community of interest (such as ICS owners, operators, and vendors) to develop the JTAG profiles of PLCs. Further, we present a case study of Kyros implementation over Allen-Bradley 1756-A10/B to help understand the framework's application on a real-world PLC used in industry settings. The sample PLC memory dumps are shared with the research community to facilitate further research.

Rais, Muhammad Haris↗
DOE OSTI2026

Gaia: segmented germanium detector for high-energy X-ray fluorescence and spectroscopic imaging

We present Gaia, a monolithic array of 96 high-purity germanium pixel detectors integrated with a custom low-noise application-specific integrated circuit (ASIC) and a field-programmable gate array (FPGA)-based data acquisition system. The sensor operates at ∼100 K using a commercial closed-cycle cryocooler, with the in-vacuum electronics thermally isolated from the cold finger to ensure thermal stability. The system demonstrates an average energy resolution of 711 eV at 122 keV, measured using a 57 Co source, and 253 eV at 5.89 keV, measured with 55 Fe across all channels. The readout architecture incorporates a high-performance FPGA paired with a dual-core ARM processor, forming a complete embedded Linux-based computing platform. Communication between the processor and FPGA is handled via memory-mapped I/O, and data are streamed over high-speed gigabit Ethernet. A full-scale 384-pixel Gaia detector, based on this 96-element module, is currently under fabrication.

36 MATERIALS SCIENCE↗
DOE OSTI2020

Highly Asynchronous Visitor Queue Graph Toolkit

HavoqGT (Highly Asynchronous Visitor Queue Graph Toolkit) is a framework for expressing asynchronous vertex-centric graph algorithms, and executing them on High Performance Computing (HPC) systems. It provides a vertex 'visitor' interface, where actions are defined at an individual vertex level, and contains a suite of classic graph algorithms. HavoqGT is capable of processing large graphs stored in NVRAM (SSDs) using a memory mapped interface.

Reza, TahsinA.↗
DOE OSTI2020

XPlacer/Tracer

XPlacer/Tracer is a dynamic analysis tool that finds bad memory access patterns in heterogeneous codes. XPlacer/Tracer consists of two components. (1) A configurable ROSE plugin for source code instrumentation of heterogeneous code written in the C/C++ and CUDA programming languages. (2) A sample configuration that instruments memory accesses to dynamic memory in CPU and GPU codes, and a runtime library that tracks these memory accesses at runtime. The collected information is reported in form of a textual summary or as memory map that can be converted to images.

Pirkelbauer, PeterM.↗
DOE OSTI2025

ghostbus v0.1.0

Ghostbus is a Verilog meta-scripting tool which automates memory map creation, address assignment, and bus routing much like Register Description Language (RDL) and SystemRDL, but with a few stark differences. First, the design is entirely defined/captured within the Verilog/SystemVerilog codebase (no extra second language). Second, the Verilog code remains valid even without the auto-generated code, enhancing modularity and compatibility with other tools/workflows.

Huang, Gang [Lawrence Berkeley National Laboratory↗
DOE OSTI2021

Privateer

Privateer is a general-purpose data store that optimizes the tradeoff between storage space utilization and I/O performance. Privateer uses memory-mapped I/O with private mapping and an optimized writeback mechanism to maximize write parallelism and eliminate redundant writes; it also uses contentaddressable storage to optimize storage space via de-duplication.

Iwabuchi, Keita↗
DOE OSTI2023

Thrifty Array Format (TAF) file specifications

Thrifty Array Foram (TAF) files store numeric data in a binary format, minimizing storage requirements while preserving quick read access. Real data of any size and dimensionality can be stored in this format at varying degrees of numeric precision. Implicit array are associated with each dimension, eliminating the need to explicitly store uniformly-spaced grid vectors. Unlimited text comments may be included with the array for user documentation, and every file begins with a text synopsis of the binary structure. The format is deliberately designed for memory mapping, where portions of the array can be read without loading the entire file at once.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Control-Agnostic Beam Instrumentation with Redis at the Core

Redis isn’t a database — it’s our protocol. Fermilab’s RedisAdapter provides a high-performance, control-system-agnostic bridge between digitized beam data and downstream consumers such as ACNET and EPICS. It forms the foundation of three new software components deployed across MicroTCA-based digitizers: GMMDM, a runtime for memory-mapped data movement from Zynq-based platforms; GRAFE, a front end for Redis-to-ACNET presentation; and GREFE, an EPICS IOC front end. Together, these tools enable modular, standardized instrumentation pipelines. Precision timing is handled via White Rabbit PPS distribution, allowing nanosecond-scale synchronization across crates. This architecture, originally prototyped in Booster BPM systems, is now deployed on modern hardware and designed to meet the performance, modularity, and scalability requirements of the PIP-II era.

Steinkamp, Derek [Fermilab] (ORCID:000900027228626↗
DOE OSTI2024

Memory access statistics monitoring

Systems, apparatuses, and methods related to memory access statistics monitoring are described. A host is configured to map pages of memory for applications to a number of memory devices coupled thereto. A first memory device comprises a monitoring component configured to monitor access statistics of pages of memory mapped to the first memory device. A second memory device does not include a monitoring component capable of monitoring access statistics of pages of memory mapped thereto. The host is configured to map a portion of pages of memory for an application to the first memory device in order to obtain access statistics corresponding to the portion of pages of memory upon execution of the application despite there being space available on the second memory device and adjust mappings of the pages of memory for the application based on the obtained access statistics corresponding to the portion of pages.

Roberts, David A.↗
DOE OSTI2022

Mapping entry invalidation

A memory access system may include a first memory address translator, a second memory address translator and a mapping entry invalidator. The first memory address translator translates a first virtual address in a first protocol of a memory access request to a second virtual address in a second protocol and tracks memory access request completions. The second memory address translator is to translate the second virtual address to a physical address of a memory. The mapping entry invalidator requests invalidation of a first mapping entry of the first mapping address translator requests invalidation of a second mapping entry of the second memory address translator corresponding to the first mapping entry following invalidation of the first mapping entry and based upon the tracked memory access request completions.

Walker, Shawn K.↗