Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “computer hardware”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,081 records · Page 60

Automated Cryocooler Monitor and Control System

A system was designed to automate cryogenically cooled low-noise amplifier systems used in the NASA Deep Space Network. It automates the entire operation of the system including cool-down, warm-up, and performance monitoring. The system is based on a single-board computer with custom software and hardware to monitor and control the cryogenic operation of the system. The system provides local display and control, and can be operated remotely via a Web interface. The system controller is based on a commercial single-board computer with onboard data acquisition capability. The commercial hardware includes a microprocessor, an LCD (liquid crystal display), seven LED (light emitting diode) displays, a seven-key keypad, an Ethernet interface, 40 digital I/O (input/output) ports, 11 A/D (analog to digital) inputs, four D/A (digital to analog) outputs, and an external relay board to control the high-current devices. The temperature sensors used are commercial silicon diode devices that provide a non-linear voltage output proportional to temperature. The devices are excited with a 10-microamp bias current. The system is capable of monitoring and displaying three temperatures. The vacuum sensors are commercial thermistor devices. The output of the sensors is a non-linear voltage proportional to vacuum pressure in the 1-Torr to 1-millitorr range. Two sensors are used. One measures the vacuum pressure in the cryocooler and the other the pressure at the input to the vacuum pump. The helium pressure sensor is a commercial device that provides a linear voltage output from 1 to 5 volts, corresponding to a gas pressure from 0 to 3.5 MPa (approx. = 500 psig). Control of the vacuum process is accomplished with a commercial electrically operated solenoid valve. A commercial motor starter is used to control the input power of the compressor. The warm-up heaters are commercial power resistors sized to provide the appropriate power for the thermal mass of the particular system, and typically provide 50 watts of heat. There are four basic operating modes. "Cool " mode commands the system to cool to normal operating temperature. "Heat " mode is used to warm the device to a set temperature near room temperature. "Pump " mode is a maintenance function that allows the vacuum system to be operated alone to remove accumulated contaminants from the vacuum area. In "Off " mode, no power is applied to the system.

Britcliffe, Michael J.↗

Resilient information and inference networks under mixed-trust sensing

With ubiquitous digitization, sensing, and computational intelligence deployed in increasingly more and broader domains, including critical infrastructure, potentially misleading and destabilizing effects of multimodal anomalies and adversarial behavior are growing in importance. Here, we develop randomized and reinforcement learning-based strategies for strategically recruiting and utilizing deployed (and, thus, vulnerable and potentially faulty and/or compromised) nodes from information and inference networks, while defending against adversaries that attempt to misguide assessments of inferred variables. Recognizing that, besides communication and other costs, sampling from any observable node can either provide true data or dangerously expose our inference to misinformation (without being easily distinguishable what actually happens), the proposed strategies proceed by progressively recruiting nodes and cautiously scaling their information contribution based on assumed, or, in our reinforcement learning approach, intelligently weighed trustworthiness, with the learning approach also considering network-wide, threat-inclusive risk/value tradeoffs. While avoiding the hardware, communication, analytical and computational burden of explicit redundancy, the proposed defensive schemes enable on-the-fly assessments of underlying processes, and system-wide situational awareness with demonstrable resilience against adversarial activities.

97 - MATHEMATICS AND COMPUTING↗

Fast and Accurate Simulation of the Cray XMT Multithreaded Supercomputer

Irregular applications, such as data mining and analysis or graph-based computations, show unpredictable memory/network access patterns and control structures. Highly multithreaded architectures with large processor counts, like the Cray MTA-1, MTA-2 and XMT, appear to address their requirements better than commodity clusters. However, the research on highly multithreaded systems is currently limited by the lack of adequate architectural simulation infrastructures due to issues such as size of the machines, memory footprint, simulation speed, accuracy and customization. At the same time, Shared-memory MultiProcessors (SMPs) with multi-core processors have become an attractive platform to simulate large scale machines. In this paper, we introduce a cycle-level simulator of the highly multithreaded Cray XMT supercomputer. The simulator runs unmodified XMT applications. We discuss how we tackled the challenges posed by its development, detailing the techniques introduced to make the simulation as fast as possible while maintaining a high accuracy. By mapping XMT processors (ThreadStorm with 128 hardware threads) to host computing cores, the simulation speed remains constant as the number of simulated processors increases, up to the number of available host cores. The simulator supports zero-overhead switching among different accuracy levels at run-time and includes a network model that takes into account contention. On a modern 48-core SMP host, our infrastructure simulates a large set of irregular applications 500 to 2000 times slower than real time when compared to a 128-processor XMT, while remaining within 10\% of accuracy. Emulation is only from 25 to 200 times slower than real time.

Cray XMT↗

Development of optimized techniques and requirements for computer enhancement of structural weld radiographs. Volume 1: Technical report

A hardware and software specification covering requirements for the computer enhancement of structural weld radiographs was considered. Three scanning systems were used to digitize more than 15 weld radiographs. The performance of these systems was evaluated by determining modulation transfer functions and noise characteristics. Enhancement techniques were developed and applied to the digitized radiographs. The scanning parameters of spot size and spacing and film density were studied to optimize the information content of the digital representation of the image.

Adams, J. R.↗

Four-dimensional volume-of-interest reconstruction for cone-beam computed tomography-guided radiation therapy

Purpose: Data sufficiency are a major problem in four-dimensional cone-beam computed tomography (4D-CBCT) on linear accelerator-integrated scanners for image-guided radiotherapy. Scan times must be in the range of 4-6 min to avoid undersampling artifacts. Various image reconstruction algorithms have been proposed to accommodate undersampled data acquisitions, but these algorithms are computationally expensive, may require long reconstruction times, and may require algorithm parameters to be optimized. The authors present a novel reconstruction method, 4D volume-of-interest (4D-VOI) reconstruction which suppresses undersampling artifacts and resolves lung tumor motion for undersampled 1-min scans. The 4D-VOI reconstruction is much less computationally expensive than other 4D-CBCT algorithms. Methods: The 4D-VOI method uses respiration-correlated projection data to reconstruct a four-dimensional (4D) image inside a VOI containing the moving tumor, and uncorrelated projection data to reconstruct a three-dimensional (3D) image outside the VOI. Anatomical motion is resolved inside the VOI and blurred outside the VOI. The authors acquired a 1-min. scan of an anthropomorphic chest phantom containing a moving water-filled sphere. The authors also used previously acquired 1-min scans for two lung cancer patients who had received CBCT-guided radiation therapy. The same raw data were used to test and compare the 4D-VOI reconstruction with the standard 4D reconstruction and the McKinnon-Bates (MB) reconstruction algorithms. Results: Both the 4D-VOI and the MB reconstructions suppress nearly all the streak artifacts compared with the standard 4D reconstruction, but the 4D-VOI has 3-8 times greater contrast-to-noise ratio than the MB reconstruction. In the dynamic chest phantom study, the 4D-VOI and the standard 4D reconstructions both resolved a moving sphere with an 18 mm displacement. The 4D-VOI reconstruction shows a motion blur of only 3 mm, whereas the MB reconstruction shows a motion blur of 13 mm. With graphics processing unit hardware used to accelerate computations, the 4D-VOI reconstruction required a 40-s reconstruction time. Conclusions: 4D-VOI reconstruction effectively reduces undersampling artifacts and resolves lung tumor motion in 4D-CBCT. The 4D-VOI reconstruction is computationally inexpensive compared with more sophisticated iterative algorithms. Compared with these algorithms, our 4D-VOI reconstruction is an attractive alternative in 4D-CBCT for reconstructing target motion without generating numerous streak artifacts.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Numerical Computation of a Continuous-thrust State Transition Matrix Incorporating Accurate Hardware and Ephemeris Models

A significant body of work exists showing that providing a nonlinear programming (NLP) solver with expressions for the problem constraint gradient substantially increases the speed of program execution and can also improve the robustness of convergence, especially for local optimizers. Calculation of these derivatives is often accomplished through the computation of spacecraft's state transition matrix (STM). If the two-body gravitational model is employed as is often done in the context of preliminary design, closed form expressions for these derivatives may be provided. If a high fidelity dynamics model, that might include perturbing forces such as the gravitational effect from multiple third bodies and solar radiation pressure is used then these STM's must be computed numerically. We present a method for the power hardward model and a full ephemeris model. An adaptive-step embedded eight order Dormand-Prince numerical integrator is discussed and a method for the computation of the time of flight derivatives in this framework is presented. The use of these numerically calculated derivatieves offer a substantial improvement over finite differencing in the context of a global optimizer. Specifically the inclusion of these STM's into the low thrust missiondesign tool chain in use at NASA Goddard Spaceflight Center allows for an increased preliminary mission design cadence.

Low Thrust↗

Bridging Cloud and Edge Computing at NREL Using CONNECT: Cloud Optimized Networking for Next-Gen Edge Computing Technologies [Slides]

CONNECT is an innovative on-premise hardware and software solution that integrates edge and cloud computing infrastructure at NREL. Built on the AWS Greengrass middleware and leveraging the MQTT protocol, CONNECT enables real-time data streaming from IoT devices and gateways to both cloud and local services, empowering researchers to rapidly capture, analyze, and act upon edge-generated data while leveraging cloud capabilities. The platform addresses research infrastructure challenges by providing a pre-approved platform which is already configured with the correct networking and cybersecurity baselines thus eliminating procurement delays and enabling on-demand availability. CONNECT's hybrid architecture efficiently manages burstable workloads, allowing research teams to dynamically scale computational capacity, handle peak data loads, and reduce operational bottlenecks. Advanced capabilities include built-in GPU support for executing machine learning models which enables low-latency inference at the edge from models trained in the cloud. This architecture supports real-time analytics and filtering, providing a mechanism to allow only transmitting and processing high-value data. Cloud-based configuration management permits engineers to manage on-premise systems remotely, optimizing operational efficiency. By bridging edge and cloud computing, CONNECT provides NREL researchers with a flexible, scalable platform that accelerates scientific discovery while maintaining robust security and performance standards.

97 MATHEMATICS AND COMPUTING↗

SimpleSSD: Modeling Solid State Drives for Holistic System Simulation

Existing solid state drive (SSD) simulators unfortunately lack hardware and/or software architecture models. Consequently, they are far from capturing the critical features of contemporary SSD devices. More importantly, while the performance of modern systems that adopt SSDs can vary based on their numerous internal design parameters and storage-level configurations, a full system simulation with traditional SSD models often requires unreasonably long runtimes and excessive computational resources. In this work, we propose SimpleSSD, a high-fidelity simulator that models all detailed characteristics of hardware and software, while simplifying the nondescript features of storage internals. In contrast to existing SSD simulators, SimpleSSD can easily be integrated into publicly-available full system simulators. Here, in addition, it can accommodate a complete storage stack and evaluate the performance of SSDs along with diverse memory technologies and microarchitectures. Thus, it facilitates simulations that explore the full design space at different levels of system abstraction.

97 MATHEMATICS AND COMPUTING↗

Multigroup Thermal Radiation Transport with Tensor Trains

We investigate the application of tensor-train (TT) algorithms to multigroup thermal radiation transport (i.e., photon radiation transport). The TT framework enables simulations at discretizations that might otherwise be computationally infeasible on conventional hardware. We show that solutions to certain multigroup problems possess an intrinsic low-rank structure, which the TT representation leverages effectively. This enables us to solve problems where the discretized solution size exceeds a trillion parameters on a single node. The solver is evaluated on a range of test problems with varying levels of complexity, consistently achieving compression factors greater than 100× and speedups exceeding 2×. We also investigate alternative TT topologies by analyzing the low-rank structure of the merged spatio-spectral core to assess the potential for greater compression. This analysis suggests that compression gains could increase by factors as large as 7. Our results indicate that the low-rank structure of the merged spatio-spectral core captures the spatio-spectral complexity of the solution, largely driven by the opacity structure of the medium. Beyond identifying opportunities for improved compression, this analysis highlights the types of errors that may arise in angle-integrated quantities when exploiting this low-rank structure.

79 ASTRONOMY AND ASTROPHYSICS↗

A common path forward for the immersive visualization community

Immersive visualization makes use of the medium of virtual reality (VR) - it is a subset of virtual reality focused on the application of VR technologies to scientific and information visualization. As the name implies, there is a particular focus on the physically immersive aspect of VR that more fully engages the perceptual and kinesthetic capabilities of the scientist with the goal of producing greater insight. The immersive visualization community is uniquely positioned to address the analysis needs of the wide spectrum of domain scientists who are becoming increasingly overwhelmed by data. The outputs of computational science simulations and high-resolution sensors are creating a data deluge. Data is coming in faster than it can be analyzed, and there are countless opportunities for discovery that are missed as the data speeds by. By more fully utilizing the scientists visual and other sensory systems, and by offering a more natural user interface with which to interact with computer-generated representations, immersive visualization offers great promise in taming this data torrent. However, increasing the adoption of immersive visualization in scientific research communities can only happen by simultaneously lowering the engagement threshold while raising the measurable benefits of adoption. Scientists time spent immersed with their data will thus be rewarded with higher productivity, deeper insight, and improved creativity. Immersive visualization ties together technologies and methodologies from a variety of related but frequently disjoint areas, including hardware, software and human-computer interaction (HCI) disciplines. In many ways, hardware is a solved problem. There are well established technologies including large walk-in systems such as the CAVE{trademark} and head-based systems such as the Wide-5{trademark}. The advent of new consumer-level technologies now enable an entirely new generation of immersive displays, with smaller footprints and costs, widening the potential consumer base. While one would be hard-pressed to call software a solved problem, we now understand considerably more about best practices for designing and developing sustainable, scalable software systems, and we have useful software examples that illuminate the way to even better implementations. As with any research endeavour, HCI will always be exploring new topics in interface design, but we now have a sizable knowledge base of the strengths and weaknesses of the human perceptual systems and we know how to design effective interfaces for immersive systems. So, in a research landscape with a clear need for better visualization and analysis tools, a methodology in immersive visualization that has been shown to effectively address some of those needs, and vastly improved supporting technologies and knowledge of hardware, software, and HCI, why hasn't immersive visualization 'caught on' more with scientists? What can we do as a community of immersive visualization researchers and practitioners to facilitate greater adoption by scientific communities so as to make the transition from 'the promise of virtual reality' to 'the reality of virtual reality'.

97 MATHEMATICS AND COMPUTING↗

Report of the 2025 Workshop on Next-Generation Ecosystems for Scientific Computing: Harnessing Community, Software, and AI for Cross-Disciplinary Team Science

This report summarizes insights from the 2025 Workshop on Next-Generation Ecosystems for Scientific Computing: Harnessing Community, Software, and AI for Cross-Disciplinary Team Science, which convened more than 40 experts from national laboratories, academia, industry, and community organizations to chart a path toward more powerful, sustainable, and collaborative scientific software ecosystems. To address urgent challenges at the intersection of high-performance computing (HPC), AI, and scientific software, participants envisioned agile, robust ecosystems built through socio-technical co-design—the intentional integration of social and technical components as interdependent parts of a unified strategy. This approach combines advances in AI, HPC, and software with new models for cross-disciplinary collaboration, training, and workforce development. Key recommendations include building modular, trustworthy AI-enabled scientific software systems; enabling scientific teams to integrate AI systems into their workflows while preserving human creativity, trust, and scientific rigor; and creating innovative training pipelines that keep pace with rapid technological change. Pilot projects were identified as near-term catalysts, with initial priorities focused on hybrid AI/HPC infrastructure, cross-disciplinary collaboration and pedagogy, responsible AI guidelines, and prototyping of public-private partnerships. This report presents a vision of next-generation ecosystems for scientific computing where AI, software, hardware, and human expertise are interwoven to drive discovery, expand access, strengthen the workforce, and accelerate scientific progress.

97 MATHEMATICS AND COMPUTING↗

Software design space exploration for exascale combustion co-design

The design of hardware for next-generation exascale computing systems will require a deep understanding of how software optimizations impact hardware design trade-offs. In order to characterize how co-tuning hardware and software parameters affects the performance of combustion simulation codes, we created ExaSAT, a compiler-driven static analysis and performance modeling framework. Our framework can evaluate hundreds of hardware/software configurations in seconds, providing an essential speed advantage over simulators and dynamic analysis techniques during the co-design process. Our analytic performance model shows that advanced code transformations, such as cache blocking and loop fusion, can have a significant impact on choices for cache and memory architecture. Our modeling helped us identify tuned configurations that achieve a 90% reduction in memory traffic, which could significantly improve performance and reduce energy consumption. These techniques will also be useful for the development of advanced programming models and runtimes, which must reason about these optimizations to deliver better performance and energy efficiency.

97 MATHEMATICS AND COMPUTING↗

Exploring the relationship between deposition method, microstructure, and performance of Nb/Si-based superconducting coplanar waveguide resonators

Superconducting quantum circuits (SQC) are one of the most promising hardware platforms for quantum computing, yet their performance is currently limited by the presence of various structural defects inside the circuit's structure. Despite impressive progress in the past decade, a precise understanding of the origin of these defects from various fabrication processes and their impact on coherence is still lacking. Here, in this study, we performed a comprehensive investigation on the microstructure, superconductivity, and resonator quality factor of Nb films deposited by high-power impulse magnetron sputtering (HiPIMS) and direct current (DC) magnetron sputtering. A suite of characterization techniques, including electron microscopy with spectroscopy, secondary ion mass spectrometry, magneto-optical microscopy, and pump-probe reflectivity spectroscopy is used. We reveal that niobium (Nb) resonators fabricated using HiPIMS exhibit a smaller average grain size, thicker surface oxide with larger thickness variations (rougher surface), and a thicker amorphous Nb/Si interface layer compared to samples deposited by DC sputtering. We identified that the amorphous Nb oxides (mainly located at the Nb surface and along the grain boundaries) and Nb-Si amorphous layers (at the Nb/Si interface) are major and potential sources of two-level system (TLS), while off-stochiometric oxides and suboxides of Nb close to the surface, crystalline defects (i.e., dislocations at grain boundary, point defects introduced during deposition) are main contributors of non-TLS sources. Our findings clarify the relationship between different defects and coherence loss mechanisms, highlighting the importance of material microstructure control on performance optimization in SQC.

36 MATERIALS SCIENCE↗

Simplifying activations with linear approximations in neural networks

A key step in Neural Networks is activation. Among the different types of activation functions, sigmoid, tanh, and others involve the usage of exponents for calculation. From a hardware perspective, exponential implementation implies the usage of Taylor series or repeated methods involving many addition, multiplication, and division steps, and as a result are power-hungry and consume many clock cycles. We implement a piecewise linear approximation of the sigmoid function as a replacement for standard sigmoid activation libraries. This approach provides a practical alternative by leveraging piecewise segmentation, which simplifies hardware implementation and improves computational efficiency. In this paper, we detail piecewise functions that can be implemented using linear approximations and their implications for overall model accuracy and performance gain. Our results show that for the DenseNet, ResNet, and GoogLeNet architectures, the piecewise linear approximation of the sigmoid function provides faster execution times compared to the standard TensorFlow sigmoid implementation while maintaining comparable accuracy. Specifically, for MNIST with DenseNet, accuracy reaches 99.91% (Piecewise) vs. 99.97% (Base) with up to 1.31x speedup in execution time. For CIFAR-10 with DenseNet, accuracy improves to 98.97% (Piecewise) vs. 99.40% (Base) while achieving 1.24x faster execution. Similarly, for CIFAR-100 with DenseNet, the accuracy is 97.93% (Piecewise) vs. 98.39% (Base), with a 1.18x execution time reduction. These results confirm the proposed method’s capability to efficiently process large-scale datasets and computationally demanding tasks, offering a practical means to accelerate deep learning models, including LSTMs, without compromising accuracy.

Activation function↗

Nonunitary Variational Quantum Eigensolver with the Localized Active Space Method and Cost Mitigation

Accurately describing strongly correlated systems with affordable quantum resources remains a central challenge for quantum chemistry applications on near and intermediate term quantum computers. The localized active space self-consistent field (LASSCF) approximates the complete active space self-consistent field (CASSCF) by generating active space-based wave functions within specific fragments while treating interfragment correlation with mean-field approach, hence is computationally less expensive. Hardware-efficient ansatzes (HEA) offer affordable and shallower circuits, yet they often fail to capture the necessary correlation. Previously, Jastrow-factor-inspired nonunitary qubit operators were proposed to use with HEA for variational quantum eigensolver (VQE) calculations (so-called nuVQE), as they do not increase circuit depths and recover correlation beyond the mean-field level for Hartree–Fock initial states. Here, in this study, we explore running nuVQE with LASSCF as the initial state. The method, named LAS-nuVQE, is shown to recover interfragment correlations, reach chemical accuracy with a small number of gates (<70) in both H 4 and square cyclobutadiene (C 4 H 4 ), and produces more accurate energetics than its HEA counterparts at all circuit depths. To further address the inherent symmetry-breaking in HEA, we implemented spin-constrained LAS-nuVQE to extend the capabilities of HEA further and show spin-pure results for square cyclobutadiene. We also mitigate the increased measurement overhead of nuVQE via Pauli grouping and shot-frugal sampling, reducing measurement costs by up to 2 orders of magnitude compared to ungrouped operator, and show that one can achieve better accuracy with a small number of shots (10 3–4 ) per one expectation value calculation compared to noiseless simulations with one or two orders of magnitude more shots. Finally, wall clock time estimates show that, with our measurement mitigation protocols, nuVQE becomes a cheaper and more accurate alternative than vanilla VQE with HEA. Taken together, these developments illustrate a practical pathway toward performing multireference chemical simulations with accuracy and affordable resources on today’s quantum hardware, achieving both accuracy and affordability in challenging correlated systems.

Wang, Qiaohong [Univ. of Chicago, IL (United State↗

Nanoscale Probing of Electrical Memory Effects in van der Waals Layered PdSe 2

Tunable electronic materials that can be switched between different impedance states are fundamental to the hardware elements for neuromorphic computing architectures. This “brain-like” computing paradigm uses highly paralleled and colocated data processing, leading to greatly improved energy efficiency and performance compared to traditional architectures in which data have to be frequently transferred between processor and memory. In this work, we use scanning microwave impedance microscopy for nanoscale electrical and electronic characterization of two-dimensional layered semiconductor PdSe 2 to probe neuromorphic properties. The local resolution of tens of nanometers reveals significant differences in electronic behavior between and within PdSe 2 nanosheets (NSs). In particular, we detected both n-type and p-type behaviors, although previous reports only point to ambipolar n-type dominating characteristics. Nanoscale capacitance–voltage curves and subsequent calculation of characteristic maps revealed a hysteretic behavior originating from the creation and erasure of Se vacancies as well as the switching of defect charge states. In addition, stacks consisting of two NSs show enhanced resistive and capacitive switching, which is attributed to trapped charge carriers at the interfaces between the stacked NSs. Stacking n- and p-type NSs results in a combined behavior that allows one to tune electrical characteristics. In conclusion, as local inhomogeneities of electrical and electronic behavior can have a significant impact on the overall device performance, the demonstrated nanoscale characterization and analysis will be applicable to a wide range of semiconducting materials.

2D material↗

Earth and Space Science Informatics Perspectives on Integrated, Coordinated, Open, Networked (ICON) Science

Abstract This article is composed of three independent commentaries about the state of Integrated, Coordinated, Open, Networked (ICON) principles (Goldman, et al., 2021b, https://doi.org/10.1029/2021EO153180 ) in Earth and Space Science Informatics (ESSI) and includes discussion on the opportunities and challenges of adopting them. Each commentary focuses on a different topic: (Section 2) Global collaboration, cyberinfrastructure, and data sharing; (Section 3) Machine learning for multiscale modeling; (Section 4) Aerial and satellite remote sensing for advancing Earth system model development by integrating field and ancillary data. ESSI addresses data management practices, computation and analysis, and hardware and software infrastructure. Our role in ICON science therefore involves collaborative work to assess, design, implement, and promote practices and tools that enable effective data management, discovery, integration, and reuse for interdisciplinary work in Earth and space science disciplines. Networks of diverse people with expertise across Earth, space, and data science disciplines are essential for efficient and ethical exchanges of findable, accessible, interoperable, and reusable (FAIR) research products and practices. Our challenge is then to coordinate the development of standards, curation practices, and tools that enable integrating and reusing multiple data types, software, multi‐scale models, and machine learning approaches across disciplines in a way that is as open and/or FAIR as ethically possible. This is a major endeavor that could greatly increase the pace and potential of interdisciplinary scientific discovery.

58 GEOSCIENCES↗

Striped Data Analysis Framework

A columnar data representation is known to be an efficient way for data storage, specifically in cases when the analysis is often done based only on a small fragment of the available data structures. A data representation like Apache Parquet is a step forward from a columnar representation, which splits data horizontally to allow for easy parallelization of data analysis. Based on the general idea of columnar data storage, working on the [LDRD Project], we have developed a striped data representation, which, we believe, is better suited to the needs of High Energy Physics data analysis. A traditional columnar approach allows for efficient data analysis of complex structures. While keeping all the benefits of columnar data representations, the striped mechanism goes further by enabling easy parallelization of computations without requiring special hardware. We will present an implementation and some performance characteristics of such a data representation mechanism using a distributed no-SQL database or a local file system, unified under the same API and data representation model. The representation is efficient and at the same time simple so that it allows for a common data model and APIs for wide range of underlying storage mechanisms such as distributed no-SQL databases and local file systems. Striped storage adopts Numpy arrays as its basic data representation format, which makes it easy and efficient to use in Python applications. The Striped Data Server is a web service, which allows to hide the server implementation details from the end user, easily exposes data to WAN users, and allows to utilize well known and developed data caching solutions to further increase data access efficiency. We are considering the Striped Data Server as the core of an enterprise scale data analysis platform for High Energy Physics and similar areas of data processing. We have been testing this architecture with a 2TB dataset from a CMS dark matter search and plan to expand it to multiple 100 TB or even PB scale. We will present the striped format, Striped Data Server architecture and performance test results.

Gutsche, Oliver↗