Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

Centralized Data Management Platform

The technology is an adaptive data management and integration platform designed for disparate data sources. It is built to support multitenancy, manage data governance, handle heterogeneous data formats and advance data democratization using a suite of connected, independent microservices. Each service can be used within an integrated environment, or as a standalone product, with a dedicated set of functionalities, such as metadata management, data versioning, access control, data tagging, link management, and analytics, among others.

Technology Transfer↗
NASA NTRS1971

Analysis of the Apollo spacecraft operational data management system. Executive summary

A study was made of Apollo, Skylab, and several other data management systems to determine those techniques which could be applied to the management of operational data for future manned spacecraft programs. The results of the study are presented and include: (1) an analysis of present data management systems, (2) a list of requirements for future operational data management systems, (3) an evaluation of automated data management techniques, and (4) a plan for data management applicable to future space programs.

Source record↗
NASA NTRSDate not supplied

Big-data Efficient Automated Science Transfer (BEAST): an open-source software architecture for arc jet data management, modeling, and automation

Big-data Efficient and Automated Science Transfer (BEAST) was conceived to address the existing ground testing data management of the NASA Ames arc jet facilities (e.g., manually entered Excel files and USB drive data transfers). These data management practices were seen as a choke point for future thermal protection system (TPS) development as they limit statistical tracking, resolution of diagnostics, coordination between video/time series, data throughput, and data processing speed/efficiency. Consequently, BEAST was developed to provide a new data infrastructure with streamlined data collection, processing, transfer, and analysis. This new framework also seeks to implement the FAIR principles of data stewardship: Findable, Accessible, Interoperable, and Reusable. The BEAST framework is based on a combination of the Python Django web framework and the Python data stack to provide a monolithic, open-source platform for data management, automation, and machine learning. This architecture was chosen for maintainability and scalability for a small, in-house development team. This paper will describe the application framework, deployment, and discuss the benefits and future plans for the system.

Data management↗
NASA NTRSDate not supplied

Big-data Efficient and Automated Science Transfer (BEAST): An Open-Source Software Architecture for Arc Jet Data Management, Modeling, and Automation

Big-data Efficient and Automated Science Transfer (BEAST) was conceived to address the existing ground testing data management of the NASA Ames arc jet facilities (e.g., manually entered Excel files and USB drive data transfers). These data management practices were seen as a choke point for future thermal protection system (TPS) development as they limit statistical tracking, resolution of diagnostics, coordination between video/time series, data throughput, and data processing speed/efficiency. Consequently, BEAST was developed to provide a new data infrastructure with streamlined data collection, processing, transfer, and analysis. This new framework also seeks to implement the FAIR principles of data stewardship: Findable, Accessible, Interoperable, and Reusable. The BEAST framework is based on a combination of the Python Django web framework and the Python data stack to provide a monolithic, open-source platform for data management, automation, and machine learning. This architecture was chosen for maintainability and scalability for a small, in-house development team. This paper will describe the application framework, deployment, and discuss the benefits and future plans for the system.

Data management↗
DOE OSTI2023

Conquering Data Chaos: Research Data Management with Kubernetes

Managing massive volumes of data and effectively making it accessible to researchers poses significant challenges and is a barrier to scientific discovery. In many cases, critical data is locked up in unwieldy file formats or one-off databases and is too large to effectively process on a single machine. This talk explores the role of Kubernetes, an open-source container orchestration platform, in addressing research data management challenges. I will discuss how we are using a set of publicly available open-source and home-grown tools in the National Renewable Energy Lab (NREL) Data, Analysis, and Visualization (DAV) group to help researchers overcome data-related bottlenecks. The talk will begin by providing an overview of the data challenges faced in research data management, including data storage, processing, and analysis. I will highlight Kubernetes' ability to handle large-scale data by leveraging containerization and distributed computing, including distributed storage. Kubernetes allows researchers to encapsulate data processing infrastructure and workflows into portable containers, enabling reproducibility and ease of deployment. Kubernetes can then schedule and manage the resource allocation of these containers to enable efficient utilization of limited computing resources, leading to more efficient data processing and analysis. I will discuss some limitations of traditional, siloed approaches to dealing with data and emphasize the need for solutions which foster collaboration. I will highlight how we are using Kubernetes at NREL to facilitate data sharing and cooperation among research teams. Kubernetes' flexible architecture enables the deployment of shared computing environments, such as Apache Superset, where researchers can seamlessly access and analyze shared datasets. Providing the ability to have one research team easily consume data generated by another, utilizing Kubernetes' as a central data platform, is one of the major wins we've encountered by adopting the platform. Finally, I will showcase real-world use cases from NREL where we have used Kubernetes to solve some persistent data challenges involving large volumes of sensor and monitoring data. I will discuss the challenges we encountered when creating our cluster and making it available as a production-ready resource. I will also discuss the specific suite of tools, including Postgres and Apache Druid for columnar and timeseries data, and Redpanda Kafka for streaming data we have deployed in our infrastructure, and the process that went into the selection of these tools.

collaborative environment↗
NASA NTRS1995

Data management, archiving, visualization and analysis of space physics data

A series of programs for the visualization and analysis of space physics data has been developed at UCLA. In the course of those developments, a number of lessons have been learned regarding data management and data archiving, as well as data analysis. The issues now facing those wishing to develop such software, as well as the lessons learned, are reviewed. Modern media have eased many of the earlier problems of the physical volume required to store data, the speed of access, and the permanence of the records. However, the ultimate longevity of these media is still a question of debate. Finally, while software development has become easier, cost is still a limiting factor in developing visualization and analysis software.

Russell, C. T.↗
NASA NTRS2016

Ontologies for Aviation Data Management

Managing complex aviation data can be a significant challenge for any enterprise – whether a government agency, airline, airframe manufacturer, or aviation service provider. To handle this challenge, data models are typically developed to characterize and manage the data generated, used, and stored by a given enterprise. Unfortunately, different data providers employ qualitatively different data models, and this gives rise to problems exchanging data across organizational boundaries. Over the past decade, these problems have motivated data producers and consumers to look toward standardized data exchange models to address data interoperability. In this paper we examine some of these standardized data exchange models and compare them with a new type of data model based on ontologies. Ontology models have emerged in recent years from a confluence of research in the artificial intelligence, semantic web, and information science communities. This paper introduces ontology models, provides several use cases for ontologies relevant to aviation data management, and summarizes state of the art aviation prototype applications that utilize ontologies.

artificial intelligence↗
NASA NTRS1984

CAD/CAM data management

The role of data base management in CAD/CAM, particularly for geometric data is described. First, long term and short term objectives for CAD/CAM data management are identified. Second, the benefits of the data base management approach are explained. Third, some of the additional work needed in the data base area is discussed.

Bray, O. H.↗
DOE OSTI2019

Data Jockey: Automatic Data Management for HPC Multi-Tiered Storage Systems

We present the design and implementation of Data Jockey, a data management system for HPC multi-tiered storage systems. As a centralized data management control plane, Data Jockey automates bulk data movement and placement for scientific workflows and integrates into existing HPC storage infrastructures. Data Jockey simplifies data management by eliminating human effort in programming complex data movements, laying datasets across multiple storage tiers when supporting complex workflows, which in turn increases the usability of multitiered storage systems emerging in modern HPC data centers.Specifically, Data Jockey presents a new data management scheme called “goal driven data management” that can automatically infer low-level bulk data movement plans from declarative high-level goal statements that come from the lifetime of iterative runs of scientific workflows. While doing so, Data Jockey aims to minimize data wait times by taking responsibility for datasets that are unused or to be used, and aggressively utilizing the capacity of the upper, higher performant storage tiers.We evaluated a prototype implementation of Data Jockey under a synthetic workload based on a year’s worth of Oak Ridge Leadership Computing Facility’s (OLCF) operational logs. Our evaluations suggest that Data Jockey leads to higher utilization of the upper storage tiers while minimizing the programming effort of data movement compared to human involved, per-domain adhoc data management scripts.

Shin, Woong↗
DOE OSTI2019

ASCR Workshop on In Situ Data Management: Enabling Scientific Discovery from Diverse Data Sources [Brochure]

In January 2019, the U.S. Department of Energy (DOE), Office of Science (SC) program in Advanced Scientific Computing Research (ASCR) convened a workshop to identify priority research directions (PRDs) for In Situ Data Management (ISDM). The workshop defined ISDM as the practices, capabilities, and procedures to control the organization of data and enable the coordination and communication among heterogeneous tasks, executing simultaneously in a high-performance computing (HPC) system, cooperating toward a common objective.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
NASA NTRS1977

Flexible data-management system

Combined ASRDI Data-Management and Analysis Technique (CADMAT) is system of computer programs and procedures that can be used to conduct data-management tasks. System was developed specifically for use by scientists and engineers who are confronted with management and analysis of large quantities of data organized into records of events and parametric fields. CADMAT is particularly useful when data are continually accumulated, such as when the need of retrieval and analysis is ongoing.

Pelouch, J. J., Jr.↗
NASA NTRS2017

Development of Increasingly Autonomous Traffic Data Manager Using Pilot Relevancy and Ranking Data

NASA's Safe Autonomous Systems Operations (SASO) project goal is to define and safely enable all future airspace operations by justifiable and optimal autonomy for advanced air, ground, and connected capabilities. This work showcases how Increasingly Autonomous Systems (IAS) could create operational transformations beneficial to the enhancement of civil aviation safety and efficiency. One such IAS under development is the Traffic Data Manager (TDM). This concept is a prototype 'intelligent party-line' system that would declutter and parse out non-relevant air traffic, displaying only relevant air traffic to the aircrew in a digital data communications (Data Comm) environment. As an initial step, over 22,000 data points were gathered from 31 Airline Transport Pilots to train the machine learning algorithms designed to mimic human experts and expertise. The test collection used an analog of the Navigation Display. Pilots were asked to rate the relevancy of the displayed traffic using an interactive tablet application. Pilots were also asked to rank the order of importance of the information given, to better weight the variables within the algorithm. They were also asked if the information given was enough data, and more importantly the "right" data to best inform the algorithm. The paper will describe the findings and their impact to the further development of the algorithm for TDM and, in general, address the issue of how can we train supervised machine learning algorithms, critical to increasingly autonomous systems, with the knowledge and expertise of expert human pilots.

Le Vie, Lisa R.↗
DOE OSTI2023

Unified Software Architecture for Advanced Materials and Manufacturing Technologies Data Management and Processing: FY 2023 Multidimensional Data Correlation Platform

This report details the various digital manufacturing activities ongoing at ORNL as part of the Advanced Materials and Manufacturing Technologies (AMMT) program. The AMMT program is exploring a data-driven approach to demonstrate the use of AM for the fabrication of components for nuclear applications, with the goal of providing a greater understanding of manufacturing quality outcomes that would pave the way toward the development of standards for certification and qualification. The objective of this work package is to establish a digital manufacturing discipline common to all participants of the AMMT program to improve the performance, reliability, and lifetime of nuclear components. As part of this effort, we will develop a unified software architecture for AMMT data management and processing, deploy the digital platform across AMMT participants’ facilities, and generate pedigreed datasets in a common format across multiple labs and facilities. To this end, the MDDC work package has focused on three activities during FY23. First, the MDF Digital Tool was overhauled to better serve the needs of the AMMT program. Next, multiple laser powder bed fusion (L-PBF) systems at the MDF were upgraded to a common sensor package for collecting comparable in situ data across machines. Finally, various improvements relevant to the AMMT program were implemented in the ORNL-developed software tool, Peregrine. This report marks the completion of FY23 milestone M3CR-22OR0403051: Report Describing the Architecture of the Digital Platform to Support AMMT Activities.

36 MATERIALS SCIENCE↗
NASA NTRS2004

Data Management for Mars Exploration Rovers

Data Management for the Mars Exploration Rovers (MER) project is a comprehensive system addressing the needs of development, test, and operations phases of the mission. During development of flight software, including the science software, the data management system can be simulated using any POSIX file system. During testing, the on-board file system can be bit compared with files on the ground to verify proper behavior and end-to-end data flows. During mission operations, end-to-end accountability of data products is supported, from science observation concept to data products within the permanent ground repository. Automated and human-in-the-loop ground tools allow decisions regarding retransmitting, re-prioritizing, and deleting data products to be made using higher level information than is available to a protocol-stack approach such as the CCSDS File Delivery Protocol (CFDP).

Mars missions↗