Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data storage data management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

Opportunities and challenges to study solar neutrinos with a Q-Pix pixel readout

The study of solar neutrinos presents significant opportunities in astrophysics, nuclear physics, and particle physics. However, the low-energy nature of these neutrinos introduces considerable challenges to isolate them from background events, requiring detectors with low-energy threshold, high spatial and energy resolutions, and low data rate. We present the study of solar neutrinos with a kiloton-scale liquid argon detector located underground, instrumented with a pixel readout using the Q-Pix technology. We explore the potential of using volume fiducialization, directional topological information, light signal coincidence, and pulse-shape discrimination to enhance solar neutrino sensitivity. We find that discriminating neutrino signals below 5 MeV is very difficult. However, we show that these methods are useful for the detection of solar neutrinos when external backgrounds are sufficiently understood and when the detector is built using low-background techniques. When building a workable background model for this study, we identify 𝛾 background from the cavern walls and from capture of 𝛼 particles in radon decay chains as both critical to solar neutrino sensitivity and significantly underconstrained by existing measurements. Finally, we highlight that the main advantage of the use of Q-Pix for solar neutrino studies lies in its ability to enable the continuous readout of all low-energy events with minimal data rates and manageable storage for further off-line analyses.

multi-purpose particle detectors↗

BASTet: Shareable and Reproducible Analysis and Visualization of Mass Spectrometry Imaging Data via OpenMSI

Mass spectrometry imaging (MSI) is a transformative imaging method that supports the untargeted, quantitative measurement of the chemical composition and spatial heterogeneity of complex samples with broad applications in life sciences, bioenergy, and health. While MSI data can be routinely collected, its broad application is currently limited by the lack of easily accessible analysis methods that can process data of the size, volume, diversity, and complexity generated by MSI experiments. The development and application of cutting-edge analytical methods is a core driver in MSI research for new scientific discoveries, medical diagnostics, and commercial-innovation. However, the lack of means to share, apply, and reproduce analyses hinders the broad application, validation, and use of novel MSI analysis methods. To address this central challenge, we introduce the Berkeley Analysis and Storage Toolkit (BASTet), a novel framework for shareable and reproducible data analysis that supports standardized data and analysis interfaces, integrated data storage, data provenance, workflow management, and a broad set of integrated tools. Based on BASTet, we describe the extension of the OpenMSI mass spectrometry imaging science gateway to enable web-based sharing, reuse, analysis, and visualization of data analyses and derived data products. Here, we demonstrate the application of BASTet and OpenMSI in practice to identify and compare characteristic substructures in the mouse brain based on their chemical composition measured via MSI.

97 MATHEMATICS AND COMPUTING↗

Toward a persistent event-streaming system for high-performance computing applications

High-performance computing (HPC) applications have traditionally relied on parallel file systems and file transfer services to manage data movement and storage. Alternative approaches have been proposed that use direct communications between application components, trading persistence and fault tolerance for speed. Event-driven architectures, as popularized in enterprise contexts, present a compelling middle ground, avoiding the performance cost and API constraints of parallel file systems while retaining persistence and offering impedance matching between application components. However, adapting streaming frameworks to HPC workloads requires addressing challenges unique to HPC systems. This paper investigates the potential for a streaming framework designed for HPC infrastructures and use cases. We introduce Mofka, a persistent event-streaming framework designed specifically for HPC environments. Mofka combines the capabilities of a traditional streaming service with optimizations tailored to the HPC context, such as support for massively multicore nodes, efficient scaling for large producer-consumer workflows, RDMA-enabled high-performance network communications, specialized network fabrics with multiple links per node, and efficient handling of large scientific data payloads. Built using the Mochi suite of HPC data service components, Mofka provides a lightweight, modular, and high-performance solution for persistent streaming in HPC systems. We present the architecture of Mofka and evaluate its performance against Kafka and Redpanda using benchmarks on diverse platforms, including Argonne's Polaris and Oak Ridge's Frontier supercomputers, showing up to 8× improvement in throughput in some scenarios. We then demonstrate its utility in several real-world applications: a tomographic reconstruction pipeline, a workflow for the discovery of metal-organic frameworks for carbon capture, and the instrumentation of Dask workflows for provenance tracking and performance analysis.

HPC↗

Integration of multi-discipline data processing for earth observing systems

The first steps taken to ensure the controlled evolution of existing facilities toward greater interoperability and sharing of resources among NASA-supported earth science and applications data systems (ESADS) are described. Recommendations made by the various panels during the 1987 ESADS Workshop are presented. The panels were concerned with directories and catalogs, data archives, data manipulation software, computational facilities, data storage media, database management, and networking. Consideration was also given to the tracking and tuning of overall development and management coordination issues.

Kahn, Ralph↗

NASA Tech Briefs, April 2013

Topics covered include: Fully Integrated, Miniature, High-Frequency Flow Probe Utilizing MEMS Leadless SOI Technology; Nanoscale Surface Plasmonics Sensor With Nanofluidic Control; Advanced Dispersed Fringe Sensing Algorithm for Coarse Phasing Segmented Mirror Telescopes; Neural Network Back-Propagation Algorithm for Sensing Hypergols; Bulk Moisture and Salinity Sensor; Change-Based Satellite Monitoring Using Broad Coverage and Targetable Sensing; Circularly Polarized Microwave Antenna Element with Very Low Off-Axis Cross-Polarization; Ultra-Low Heat-Leak, High-Temperature Superconducting Current Leads for Space Applications; Flash Cracking Reactor for Waste Plastic Processing; An Automated Safe-to-Mate (ASTM) Tester; Wireless Chalcogenide Nanoionic-Based Radio-Frequency Switch; Compute Element and Interface Box for the Hazard Detection System; DOT Transmit Module; Composite Aerogel Multifoil Protective Shielding; Li-Ion Electrolytes with Improved Safety and Tolerance to High-Voltage Systems; Polymer-Reinforced, Non-Brittle, Lightweight Cryogenic Insulation; Controlled, Site-Specific Functionalization of Carbon Nanotubes with Diazonium Salts; Regenerable Sorbent for CO2 Removal; Sprayable Aerogel Bead Compositions With High Shear Flow Resistance and High Thermal Insulation Value; Lexan Linear Shaped Charge Holder with Magnets and Backing Plate; Robotic Ankle for Omnidirectional Rock Anchors; Wind, Wave, and Tidal Energy Without Power Conditioning; An Active Heater Control Concept to Meet IXO Type Mirror Module Thermal-Structural Distortion Requirement; Waterless Clothes-Cleaning Machine; Integrated Electrical Wire Insulation Repair System; LVGEMS Time-of-Flight Mass Spectrometry on Satellites; Surface Inspection Tool for Optical Detection of Surface Defects; Per-Pixel, Dual-Counter Scheme for Optical Communications; Certification-Based Process Analysis; Surface Navigation Using Optimized Waypoints and Particle Swarm Optimization; Smart-Divert Powered Descent Guidance to Avoid the Backshell Landing Dispersion Ellipse; Estimating Foreign-Object-Debris Density from Photogrammetry Data; Adaptive Sampling of Spatiotemporal Phenomena with Optimization Criteria; Building a 2.5D Digital Elevation Model From 2D Imagery; Eyes on the Earth 3D; Target Trailing With Safe Navigation for Maritime Autonomous Surface Vehicles; Adams-Based Rover Terramechanics and Mobility Simulator - ARTEMIS; ISTP CDF Skeleton Editor; Uplink Summary Generator (ULSGEN) Version 1.0; Robotics On-Board Trainer (ROBoT); Software Engineering Tools for Scientific Models; Automatic Data Filter Customization Using a Genetic Algorithm; Tracker Toolkit; Towards Efficient Scientific Data Management Using Cloud Storage; On a Formal Tool for Reasoning About Flight Software Cost Analysis; A Nanostructured Composites Thermal Switch Controls Internal and External Short Circuit in Lithium Ion Batteries; Spacecraft Crew Cabin Condensation Control; and Functional Near-Infrared Spectroscopy Signals Measure Neuronal Activity in the Cortex.

Source record↗

Incorporating Genomics and Bioinformatics across the Life Sciences Curriculum

Undergraduate life sciences education needs an overhaul, as clearly described in the National Research Council of the National Academies publication BIO 2010: Transforming Undergraduate Education for Future Research Biologists. Among BIO 2010's top recommendations is the need to involve students in working with real data and tools that reflect the nature of life sciences research in the 21st century. Education research studies support the importance of utilizing primary literature, designing and implementing experiments, and analyzing results in the context of a bona fide scientific question in cultivating the analytical skills necessary to become a scientist. Incorporating these basic scientific methodologies in undergraduate education leads to increased undergraduate and post-graduate retention in the sciences. Toward this end, many undergraduate teaching organizations offer training and suggestions for faculty to update and improve their teaching approaches to help students learn as scientists, through design and discovery (e.g., Council of Undergraduate Research [www.cur.org] and Project Kaleidoscope [www.pkal.org]). With the advent of genome sequencing and bioinformatics, many scientists now formulate biological questions and interpret research results in the context of genomic information. Just as the use of bioinformatic tools and databases changed the way scientists investigate problems, it must change how scientists teach to create new opportunities for students to gain experiences reflecting the influence of genomics, proteomics, and bioinformatics on modern life sciences research. Educators have responded by incorporating bioinformatics into diverse life science curricula. While these published exercises in, and guidelines for, bioinformatics curricula are helpful and inspirational, faculty new to the area of bioinformatics inevitably need training in the theoretical underpinnings of the algorithms. Moreover, effectively integrating bioinformatics into courses or independent research projects requires infrastructure for organizing and assessing student work. Here, we present a new platform for faculty to keep current with the rapidly changing field of bioinformatics, the Integrated Microbial Genomes Annotation Collaboration Toolkit (IMG-ACT). It was developed by instructors from both research-intensive and predominately undergraduate institutions in collaboration with the Department of Energy-Joint Genome Institute (DOE-JGI) as a means to innovate and update undergraduate education and faculty development. The IMG-ACT program provides a cadre of tools, including access to a clearinghouse of genome sequences, bioinformatics databases, data storage, instructor course management, and student notebooks for organizing the results of their bioinformatic investigations. In the process, IMG-ACT makes it feasible to provide undergraduate research opportunities to a greater number and diversity of students, in contrast to the traditional mentor-to-student apprenticeship model for undergraduate research, which can be too expensive and time-consuming to provide for every undergraduate. The IMG-ACT serves as the hub for the network of faculty and students that use the system for microbial genome analysis. Open access of the IMG-ACT infrastructure to participating schools ensures that all types of higher education institutions can utilize it. With the infrastructure in place, faculty can focus their efforts on the pedagogy of bioinformatics, involvement of students in research, and use of this tool for their own research agenda. What the original faculty members of the IMG-ACT development team present here is an overview of how the IMG-ACT program has affected our development in terms of teaching and research with the hopes that it will inspire more faculty to get involved.

59 BASIC BIOLOGICAL SCIENCES↗

Improving Data Availability for Better Access Performance: A Study on Caching Scientific Data on Distributed Workstations

Client-side data caching serves as an excellent mechanism to store and analyze the rapidly growing amount of scientific data. In our previous work, we built a distributed local cache on unreliable desktop storage contributions. This offers several desirable properties, such as performance impedance matching, improved space utilization, and high parallel I/O bandwidth. Such a low-cost, best-effort cache, however, is faced with the vagaries of storage node availability: these donated machines may be significantly less reliable than dedicated systems and cannot be controlled centrally. In this paper, we address %the tradeoffs between techniques that favor %availability or performance when it comes to cache management. the performance impact of data availability in the distributed scientific data cache setting. We then present a novel approach to storage cache management, {\em remote partial data recovery (RPDR)}. We compare our approach to two standard techniques, namely replication and erasure coding, both extended to the target caching environment. Our evaluation uses a trace-driven simulation parameterized with benchmarking results from our distributed cache prototype. The results with multiple real-world traces indicate that RPDR significantly outperforms both replication and erasure coding in many cases and overall the combination of RPDR and erasure coding yields the best performance.

99 GENERAL AND MISCELLANEOUS↗

Geodata Modeling and Query in Geographic Information Systems

Geographic information systems (GIS) deal with collecting, modeling, man- aging, analyzing, and integrating spatial (locational) and non-spatial (attribute) data required for geographic applications. Examples of spatial data are digital maps, administrative boundaries, road networks, and those of non-spatial data are census counts, land elevations and soil characteristics. GIS shares common areas with a number of other disciplines such as computer- aided design, computer cartography, database management, and remote sensing. None of these disciplines however, can by themselves fully meet the requirements of a GIS application. Examples of such requirements include: the ability to use locational data to produce high quality plots, perform complex operations such as network analysis, enable spatial searching and overlay operations, support spatial analysis and modeling, and provide data management functions such as efficient storage, retrieval, and modification of large datasets; independence, integrity, and security of data; and concurrent access to multiple users. It is on the data management issues that we devote our discussions in this monograph. Traditionally, database management technology have been developed for business applications. Such applications require, among other things, capturing the data requirements of high-level business functions and developing machine- level implementations; supporting multiple views of data and yet providing integration that would minimize redundancy and maintain data integrity and security; providing a high-level language for data definition and manipulation; allowing concurrent access to multiple users; and processing user transactions in an efficient manner. The demands on database management systems have been for speed, reliability, efficiency, cost effectiveness, and user-friendliness. Significant progress have been made in all of these areas over the last two decades to the point that many generalized database platforms are now available for developing data intensive applications that run in real-time. While continuous improvement is still being made at a very fast-paced and competitive rate, new application areas such as computer aided design, image processing, VLSI design, and GIS have been identified by many as the next generation of database applications. These new application areas pose serious challenges to the currently available database technology. At the core of these challenges is the nature of data that is manipulated. In traditional database applications, the database objects do not have any spatial dimension, and as such, can be thought of as point data in a multi-dimensional space. For example, each instance of an entity EMPLOYEE will have a unique value corresponding to every attribute such as employee id, employee name, employee address and so on. Thus, every Employee instance can be thought of as a point in a multi-dimensional space where each dimension is represented by an attribute. Furthermore, all operations on such data are one-dimensional. Thus, users may retrieve all entities satisfying one or more constraints. Examples of such constraints include employees with addresses in a certain area code, or salaries within a certain range. Even though constraints can be specified on multiple attributes (dimensions), the search for such data is essentially orthogonal across these dimensions.

Adam, Nabil↗

White Paper on Data Repository Reorganization Proposal for the xLPR Project

As the xLPR project moves along, it is important to properly manage the knowledge generated by the different groups. We focus specifically on the knowledge and communications written in files, including general documents, source code and executable files. Data generated through the project are different in nature and, for this reason, need to be treated differently. To that end, ORNL put in place a series of tools that facilitate proper storage and management of project data, document and code changes, group collaboration, knowledge transfer, transparency, accountability and auditability. This paper describes the approaches/tools that we recommend for moving the project forward on knowledge management.

Klasky, Hilda B.↗

NASA Langley Research Center's distributed mass storage system

There is a trend in institutions with high performance computing and data management requirements to explore mass storage systems with peripherals directly attached to a high speed network. The Distributed Mass Storage System (DMSS) Project at NASA LaRC is building such a system and expects to put it into production use by the end of 1993. This paper presents the design of the DMSS, some experiences in its development and use, and a performance analysis of its capabilities. The special features of this system are: (1) workstation class file servers running UniTree software; (2) third party I/O; (3) HIPPI network; (4) HIPPI/IPI3 disk array systems; (5) Storage Technology Corporation (STK) ACS 4400 automatic cartridge system; (6) CRAY Research Incorporated (CRI) CRAY Y-MP and CRAY-2 clients; (7) file server redundancy provision; and (8) a transition mechanism from the existent mass storage system to the DMSS.

Pao, Juliet Z.↗

Managing People's Data

Just imagine a mass storage system that consists of a machine with 2 CPUs, 1 Gigabyte (GB) of memory, 400 GB of disk space, 16800 cartridge tapes in the automated tape silos, 88,000 tapes located in the vault, and the software to manage the system. This system is designed to be a data repository; it will always have disk space to store all the incoming data. Currently 9.14 GB of new data per day enters the system with this rate doubling each year. To assure there is always disk space available for new data, the system. has to move data reside from the expensive disk to a much less expensive medium such as the 3480 cartridge tapes. Once the data is archived to tape, it should be able to move back to disk when someone wants to access it and the data movement should be transparent to the user. Now imagine all the tasks that a system administrator must perform to keep this system running 24 hour a day, 7 days a week. Since the filesystem maintains the illusion of unlimited disk space, data that comes to the system must get moved to tapes in an efficient manner. This paper will describe the mass storage system running at the Numerical Aerodynamic Simulation (NAS) at NASA Ames Research Center in both software and hardware aspects, then it will describe all of the tasks the system administrator has to perform on this system.

Le, Diana↗

Managing geometric information with a data base management system

The strategies for managing computer based geometry are described. The computer model of geometry is the basis for communication, manipulation, and analysis of shape information. The research on integrated programs for aerospace-vehicle design (IPAD) focuses on the use of data base management system (DBMS) technology to manage engineering/manufacturing data. The objectives of IPAD is to develop a computer based engineering complex which automates the storage, management, protection, and retrieval of engineering data. In particular, this facility must manage geometry information as well as associated data. The approach taken on the IPAD project to achieve this objective is discussed. Geometry management in current systems and the approach taken in the early IPAD prototypes are examined.

Dube, R. P.↗

External Data and Attribute Hyperlink Programs for Promis*e(Registered Trademark)

External Data and Attribute Hyperlink are computer programs that can be added to Promis*e(trademark) which is a commercial software system that automates routine tasks in the design (including drawing schematic diagrams) of electrical control systems. The programs were developed under the Stennis Space Center's (SSC) Dual Use Technology Development Program to provide capabilities for SSC's BMCS configuration management system which uses Promis*e(trademark). The External Data program enables the storage and management of information in an external database linked to a drawing. Changes can be made either in the database or on the drawing. Information that originates outside Promis*e(trademark) can be stored in custom fields that can be added to the database. Although this information is not available in Promis*e(trademark) printed drawings, it can be associated with symbols in the drawings, and can be retrieved through the drawings when the software is running. The Attribute Hyperlink program enables the addition of hyperlink information as attributes of symbols. This program enables the formation of a direct hyperlink between a schematic diagram and an Internet site or a file on a compact disk, on the user's hard drive, or on another computer on a network to which the user's computer is connected. The user can then obtain information directly related to the part (e.g., maintenance, or troubleshooting information) associated with the hyperlink.

Derengowski, Rich↗

A database management capability for Ada

The data requirements of mission critical defense systems have been increasing dramatically. Command and control, intelligence, logistics, and even weapons systems are being required to integrate, process, and share ever increasing volumes of information. To meet this need, systems are now being specified that incorporate data base management subsystems for handling storage and retrieval of information. It is expected that a large number of the next generation of mission critical systems will contain embedded data base management systems. Since the use of Ada has been mandated for most of these systems, it is important to address the issues of providing data base management capabilities that can be closely coupled with Ada. A comprehensive distributed data base management project has been investigated. The key deliverables of this project are three closely related prototype systems implemented in Ada. These three systems are discussed.

Chan, Arvola↗

Investigation into Cloud Computing for More Robust Automated Bulk Image Geoprocessing

Geospatial resource assessments frequently require timely geospatial data processing that involves large multivariate remote sensing data sets. In particular, for disasters, response requires rapid access to large data volumes, substantial storage space and high performance processing capability. The processing and distribution of this data into usable information products requires a processing pipeline that can efficiently manage the required storage, computing utilities, and data handling requirements. In recent years, with the availability of cloud computing technology, cloud processing platforms have made available a powerful new computing infrastructure resource that can meet this need. To assess the utility of this resource, this project investigates cloud computing platforms for bulk, automated geoprocessing capabilities with respect to data handling and application development requirements. This presentation is of work being conducted by Applied Sciences Program Office at NASA-Stennis Space Center. A prototypical set of image manipulation and transformation processes that incorporate sample Unmanned Airborne System data were developed to create value-added products and tested for implementation on the "cloud". This project outlines the steps involved in creating and testing of open source software developed process code on a local prototype platform, and then transitioning this code with associated environment requirements into an analogous, but memory and processor enhanced cloud platform. A data processing cloud was used to store both standard digital camera panchromatic and multi-band image data, which were subsequently subjected to standard image processing functions such as NDVI (Normalized Difference Vegetation Index), NDMI (Normalized Difference Moisture Index), band stacking, reprojection, and other similar type data processes. Cloud infrastructure service providers were evaluated by taking these locally tested processing functions, and then applying them to a given cloud-enabled infrastructure to assesses and compare environment setup options and enabled technologies. This project reviews findings that were observed when cloud platforms were evaluated for bulk geoprocessing capabilities based on data handling and application development requirements.

Brown, Richard B.↗

Parallel IO Libraries for Managing HEP Experimental Data

The computing and storage requirements of the energy and intensity frontiers will grow significantly during the Run 4 & 5 and the HL-LHC era. Similarly, in the intensity frontier, with larger trig ger readouts during supernovae explosions, the Deep Underground Neutrino Experiment (DUNE) will have unique computing challenges that could be addressed by the use of parallel and accelerated dataprocessing capabilities. Most of the requirements of the energy and intensity frontier experiments rely on increasing the role of high performance computing (HPC) in the HEP community. In this presentation, we will describe our ongoing efforts that are focused on using HPC resources for the next generation HEP experiments. The HEPCCE (High Energy Physics-Center for Computational Excellence) IOS (Input/Output and Storage) group has been developing approaches to map HEP data to the HDF5 , an IO library optimized for the HPC platforms to store the intermediate HEP data. The complex HEP data products are serialized using ROOT to allow for experiment independent general mapping approaches of the HEP data to the HDF5 format. The mapping approaches can be optimized for high performance parallel IO. Similarly, simpler data can be directly mapped into the HDF5, which can also be suitable for offloading into the GPUs directly. We will present our works on both complex and simple data model models.

Bashyal, Amit↗

Data Management in the Continuum: Cross-facility Object-based Data Transfers

Scientific workflows are evolving from relying on a monolithic storage subsystem at a single High-Performance Computing (HPC) facility to using geographically distributed file systems, repositories, and cloud storage. As a result, storing, accessing, transferring, and managing scientific data have become highly complex and prone to performance inefficiencies. This paper delves into these challenges by exploring an optimized end-to-end interface designed to seamlessly connect various local and remote storage systems, enabling efficient data movement of objects across HPC–Cloud and HPC–HPC environments. We showcase this capability through an object-focused data management runtime system, discuss the effects of relaxed consistency semantics in distributed object scenarios, and illustrate its application in an earthquake simulation workflow. Besides reducing the amount of data by selectively transferring regions of interest, our facility-local results achieved a speedup of 45 × over an optimized HDF5 usage and 15 × over the HDF5 with caching by using the new interface in PDC-XF.

Bez, Jean Luca↗

"PoliMOR: A Policy Engine \"Made-to-Order\" for Automated and Scalable Data Management in Lustre"

Modern supercomputing systems are increasingly reliant on hierarchical, multi-tiered file and storage system architectures due to cost-performance-capacity trade-offs. Within such multi-tiered systems, data management services are required to maintain healthy utilization, performance, and capacity levels. We present PoliMOR, a pragmatic and reliable policy-driven data management framework. PoliMOR is composed of modular, single-purpose agents that gather file system metadata and enforce policies on storage systems. PoliMOR facilitates automated and scalable data management with customizable agents tailored to HPC facility-specific storage systems and policies. Our evaluations demonstrate the scalability and performance of PoliMOR both by its individual agents and as a collective entity. We believe PoliMOR is widely applicable across HPC facilities with large-scale data management challenges and will garner interest from the HPC community, given its flexible and open-source nature.

George, Anjus↗