Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “research data management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Geospatial Data Platform for All

Spatiotemporal data has evolved in scale due to augmented use in cross-domain applications. Simultaneously, there is substantial growth in the availability of Geographic Information Systems (GIS) data provided by the United States Geological Survey (USGS) along with other federal, state, county, or local agencies through open-data portals and public access APIs. However, data availability does not equate with accessibility. Large-scale analyses and applications require robust, performant data management with co-location of data storage and computing. The insufficiency of data management infrastructure compels researchers to adopt ad hoc project- specific GIS data storage solutions (e.g., copying data to High-Performance computer file systems). As an ad hoc storage strategy does not scale, it hampers cross-domain analyses causing difficulty in data reuse and utilizing existing code bases. Furthermore, GIS data is complex and requires expertise to analyze and manipulate due to its intricate data structures and data-specific projection transformations. Despite the challenges, we recognize that derived GIS data products, e.g., satellite or LIDAR-based images, can be used in downstream applications such as AI by domain, but non-GIS experts. To address the data needs and overcome the challenges, we are working towards a GIS Data Platform focused on efficient data storage, data discovery and access, and an API to enable common workflows. We propose a knowledge-graph (KG) approach for data discovery, whereby datasets are semantically linked to higher- level constructs such as projects and research areas. The semantic data links enable researchers to explore datasets in a top-down approach by specifying relevant and meaningful terms (assists in finding hidden data). An advantage is that the nodes and edges in a knowledge graph create built-in semantic documentation. Deeper spatiotemporal connections between data sources can be encoded via Graph Neural Networks (GNN) (Zhang et al., 2021). The KG approach can be extended to integrate the data itself in a Virtual KG (VKG). Our work will derive inspiration from large-scale VKG efforts that have been undertaken or are currently underway as part of the OpenStreetMap project (Ding et al., 2021). For DOE Data Days, we share the proposed geospatial data platform hybrid (cloud/on-prem) architecture, our work-to-date on storing, retrieving, and transforming LiDAR and raster data relevant to two important NREL use-cases, including the Renewable Energy Potential (reV) Model, and present our proposal for a KG based data discovery engine.

data platform↗
DOE OSTI2021

A Lakehouse Architecture for the Management and Analysis of Heterogeneous Data for Biomedical Research and Mega-biobanks

Data Lakehouse is a new paradigm in data architectures that embodies and integrates already established concepts for the systematic management of disparate, large-scale data – a data lake for heterogeneous data management, use of open standards for high-performance querying, and systematic maintenance of the data "freshness". In addition to being a new concept, the data lakehouse is also still a conceptual construct. Many projects that use the lakehouse require maturing, empirical studies, and specific implementations. In this paper, we present our implementation of the data lakehouse concept in a biomedical research and health data analytics domain, and we discuss the implementation of some unique and novel features such as support for specialized access controls in support of HIPAA regulation and IRB protocols, and support for the FAIR standard.

Begoli, Edmon↗
DOE OSTI2024

Enhancing Discoverability and Management of Atmospheric Data at Scale: Solutions from the ARM Data Center

The Atmospheric Radiation Measurement (ARM) is a multi-laboratory and multi-institutional U.S. Department of Energy (DOE) Office of Science National User Facility. The ARM Data Center (ADC), located at Oak Ridge National Laboratory, collects, archives, and shares vast atmospheric data crucial for climate research. The ADC manages over 7 PB of data from 460 instruments worldwide, processing it into more than 11,000 diverse data products using the Network Common Data Form (NetCDF) for machine-independent accessibility. The primary challenge addressed in this paper is the efficient management and distribution of vast and diverse datasets essential for the climate research community, enhancing accessibility through advanced tools like Data Discovery. The ADC has developed advanced infrastructure and software architecture to handle the continuous influx of heterogeneous data to enhance data discoverability, resulting in increased scientific collaboration. In 2023, users from over 34 countries downloaded and utilized ARM data, resulting in 1,455 publications. The ADC’s efforts have significantly improved the discoverability and usability of atmospheric data, fostering extensive scientific research and collaboration. This paper details the solutions implemented by the ADC team for efficient data discovery and distribution, and it demonstrates ARM’s capability of staging processed data for scientific analysis.

Shah, Chirag [ORNL] (ORCID:0000000203145737)↗
DOE OSTI2022

Position Papers for the ASCR Workshop on the Management and Storage of Scientific Data

The purpose of this workshop is to identify priority research directions in the area of data management for high-performance and scientific computing above and beyond HPC’s traditional "the parallel file system is the data-management system" model. Supporting the breadth of the DOE mission, including the explosion of AI uses and the growing needs of experimental and observational science, motivates revisiting our assumptions about data management. There are many facets of this topic to explore including: (1) Interfaces for accessing data that resides on traditional persistent storage as well as memory devices; (2) Storage-system architecture design that supports scientific workflows on varied hierarchical storage and networking devices; (3) Devising metadata management infrastructure to support FAIR principles (Findability, Accessibility, Interoperability, and Reusability); (4) Capturing provenance information about scientific data; (5) Utilizing AI to learn I/O patterns of emerging workloads for efficient data management; (6) Providing data management support for AI and complex workflows; and (7) Understanding the overlap between traditional storage systems and I/O (SSIO) efforts and data management. While the program committee has identified these topics as important areas for discussion, we welcome position papers from the community that propose additional topics of interest for discussion at the workshop. The workshop agenda will include breakout sessions for discussing these and selected topic areas to inform priority research directions for data management for high-performance and scientific computing.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Report for the ASCR Workshop on the Management and Storage of Scientific Data

The purpose of this workshop is to identify priority research directions in the area of data management for high-performance and scientific computing above and beyond HPC’s traditional "the parallel file system is the data-management system" model. Supporting the breadth of the DOE mission, including the explosion of AI uses and the growing needs of experimental and observational science, motivates revisiting our assumptions about data management. There are many facets of this topic to explore including: (1) Interfaces for accessing data that resides on traditional persistent storage as well as memory devices; (2) Storage-system architecture design that supports scientific workflows on varied hierarchical storage and networking devices; (3) Devising metadata management infrastructure to support FAIR principles (Findability, Accessibility, Interoperability, and Reusability); (4) Capturing provenance information about scientific data; (5) Utilizing AI to learn I/O patterns of emerging workloads for efficient data management; (6) Providing data management support for AI and complex workflows; and (7) Understanding the overlap between traditional storage systems and I/O (SSIO) efforts and data management. While the program committee has identified these topics as important areas for discussion, we welcome position papers from the community that propose additional topics of interest for discussion at the workshop. The workshop agenda will include breakout sessions for discussing these and selected topic areas to inform priority research directions for data management for high-performance and scientific computing.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1977

Instrumentation and data acquisition systems

A comprehensive and integrated measurement system was identified and a design and development effort initiated to meet the criteria imposed by the National Transonic Facility operating environment. Specific measurement areas receiving concentrated attention include: data acquisition, force measurement, pressure instrumentation, flow visualization techniques, model attitude and model deformation measurement, and temperature measurement. The NTF instrument complex will be centered around four 32-bit, 1-microsecond-cycle-time central processing units connected in a multipoint-distributed network configuration. The principal activities to be supported by these computers are: (1) data base management and processing; (2) research measurement data acquisition and display; (3) tunnel and model control; and (4) process monitoring and communication control. The distributed network approach was chosen to modularize the functional software into definable and implementable parts by the various groups involved in the design and to permit use of similar hardware configurations to improve reliability and maintainability.

Guarino, J. F.↗
NASA NTRSDate not supplied

Exploring the Landscape of Earth and Space Science Informatics using Latent Topic Modeling

AGU Earth and Space Science Informatics (ESSI) is at the forefront of data management, analysis, large scale experimentation, and infrastructure development pertaining to Earth and Space Science interests. The key topics of interest within ESSI are also evolving and diversifying over time. We aim to observe and quantify the various topics covered in ESSI, analyze their trends over time, and identify the contributors’ affiliations to gain an understanding of the landscape of ESSI and the direction of the research and management. The data for this work are abstracts submitted to AGU’s ESSI Fall meeting; They serve as a proxy for key research and development areas within ESSI. We use an unsupervised topic modeling technique called Latent Dirichlet Allocation to observe the underlying topics covered in ESSI and their trends over time. With this presentation, we showcase our results from the analysis and insights gained.

Muthukumaran Ramasubramanian↗
NASA NTRS1984

The pilot climate data system

The Pilot Climate Data System (PCDS) is an interactive scientific information management system for locating, obtaining, manipulating, and displaying climate-research data. The PCDS was developed to manage a large collection of data of interest to the National Aeronautics and Space Administration's (NASA) research community and currently provides such support for approximately twenty data sets. In order to provide the PCDS capabilities, NASA's Goddard Space Flight Center (NASA/GSFC) has integrated the capabilities of several general-purpose software packages with specialized software for reading and reformatting the supported data sets. These capabilities were integrated in a manner which allows the PCDS to be easily expanded, either to provide support for additional data sets or to provide additional functional capabilities. This also allows the PCDS to take advantage of new technology as it becomes available, since parts of the system can be replaced with more powerful components without significantly affecting the user interface.

Reph, M. G.↗
NASA NTRS2007

Full Life-Cycle Defect Management Assessment: Initial Inspection Data Collection Results and Research Questions for Further Study

It is often the case in software projects that when schedule and budget resources are limited, the Verification and Validation (V&V) activities suffer. Fewer V&V activities can be afforded and moreover, short-term challenges can result in V&V activities being scaled back or dropped altogether. As a result, too often the default solution is to save activities for improving software quality until too late in the life-cycle, relying on late-term code inspections followed by thorough testing activities to reduce defect counts to acceptable levels. As many project managers realize, however, this is a resource-intensive way of achieving the required quality for software. The Full Life-cycle Defect Management Assessment Initiative, funded by NASA s Office of Safety and Mission Assurance under the Software Assurance Research Program, aims to address these problems by: Improving the effectiveness of early life-cycle V&V activities to make their benefits more attractive to team leads. Specifically, we focus on software inspection, a proven method that can be applied to any software work product, long before executable code has been developed; Better communicating this effectiveness to software development teams, along with suggestions for parameters to improve in the future to increase effectiveness; Analyzing the impact of early life-cycle V&V on the effectiveness and cost required for late life-cycle V&V activities, such as testing, in order to make the tradeoffs more apparent. This white paper reports on an initial milestone in this work, the development of a preliminary model of inspection effectiveness across multiple NASA Centers. This model contributes toward reaching our project goals by: Allowing an examination of inspection parameters, across different types of projects and different work products, for an analysis of factors that impact defect detection effectiveness. Allowing a comparison of this NASA-specific model to existing recommendations in the literature regarding how to plan effective inspections. Forming a baseline model which can be extended to incorporate factors describing: the numbers and types of defects that are missed by inspections; how such defects flow downstream through software development phases; how effectively they can be caught by testing activities in the late stages of development. The model has been implemented in a prototype web-enabled decision-support tool which allows developers to enter their inspection data and receive feedback based on a comparison against the model. The tool also allows users to access reusable materials (such as checklists) from projects included in the baseline. Both the tool itself and the model underlying it will continue to be extended throughout the remainder of this initiative. As results of analyzing inspection effectiveness for defect containment are determined, they can be shared via the tool and also via updates to existing training courses on metrics and software inspections. Moreover, the tool will help satisfy key CMMI requirements for the NASA Centers, as it will enable NASA to take a global view across peer review results for various types of projects to identify systemic problems. This analysis can result in continuous improvements to the approach to verification.

Shull, Forrest↗
NASA NTRS1998

Open Architecture Data System for NASA Langley Combined Loads Test System

The Combined Loads Test System (COLTS) is a new structures test complex that is being developed at NASA Langley Research Center (LaRC) to test large curved panels and cylindrical shell structures. These structural components are representative of aircraft fuselage sections of subsonic and supersonic transport aircraft and cryogenic tank structures of reusable launch vehicles. Test structures are subjected to combined loading conditions that simulate realistic flight load conditions. The facility consists of two pressure-box test machines and one combined loads test machine. Each test machine possesses a unique set of requirements or research data acquisition and real-time data display. Given the complex nature of the mechanical and thermal loads to be applied to the various research test articles, each data system has been designed with connectivity attributes that support both data acquisition and data management functions. This paper addresses the research driven data acquisition requirements for each test machine and demonstrates how an open architecture data system design not only meets those needs but provides robust data sharing between data systems including the various control systems which apply spectra of mechanical and thermal loading profiles.

Lightfoot, Michael C.↗
NASA NTRS1989

Proceedings of the Scientific Data Compression Workshop

Continuing advances in space and Earth science requires increasing amounts of data to be gathered from spaceborne sensors. NASA expects to launch sensors during the next two decades which will be capable of producing an aggregate of 1500 Megabits per second if operated simultaneously. Such high data rates cause stresses in all aspects of end-to-end data systems. Technologies and techniques are needed to relieve such stresses. Potential solutions to the massive data rate problems are: data editing, greater transmission bandwidths, higher density and faster media, and data compression. Through four subpanels on Science Payload Operations, Multispectral Imaging, Microwave Remote Sensing and Science Data Management, recommendations were made for research in data compression and scientific data applications to space platforms.

Ramapriyan, H. K.↗
NASA NTRS1982

A crops and soils data base for scene radiation research

Management and planning activities with respect to food production require accurate and timely information on crops and soils on a global basis. The needed information can be obtained with the aid of satellite-borne sensors, if the relations between the spectral properties and the important biological-physical parameters of crops and soils are known. In order to obtain this knowledge, the development of a crops and soils scene radiation research data base was initiated. Work related to the development of this data base is discussed, taking into account details regarding the conducted experiments, the performed measurements, the calibration of spectral data, questions of data base access, and the expansion of the crops and soils scene radiation data base for 1982.

Biehl, L. L.↗
NASA NTRS2010

Terrestrial Hydrological Data from NASA's Hydrology Data and Information Services Center (HDISC): Products, Services, and Applications

Terrestrial hydrological variables are important in global hydrology, climate, and carbon cycle studies. The North American and Global Land Data Assimilation Systems (NLDAS and GLDAS, respectively) have been generating a series of land surface states (soil moisture, snow, and temperature) and fluxes (evapotranspiration, radiation, and heat flux) variables. These data, hosted at and available from NASA s Hydrology Data and Information Services Center (HDISC), include the NLDAS hourly 1/8 degree products and the GLDAS 3-hourly 0.25 and 1.0 degree products. HDISC provides easy access and visualization and analysis capabilities for these products, thus reducing the time and resources spent by scientists on data management and facilitating hydrological research. Users can perform spatial and parameter subsetting, data format transformation, and data analysis operations without needing to first download the data. HDISC is continually being developed as a data and services portal that supports weather and climate forecasts, and water and energy cycle research.

Fang, Hongliang↗
NASA NTRS2009

Interactive, Automated Management of Icing Data

IceVal DatAssistant is software (see figure) that provides an automated, interactive solution for the management of data from research on aircraft icing. This software consists primarily of (1) a relational database component used to store ice shape and airfoil coordinates and associated data on operational and environmental test conditions and (2) a graphically oriented database access utility, used to upload, download, process, and/or display data selected by the user. The relational database component consists of a Microsoft Access 2003 database file with nine tables containing data of different types. Included in the database are the data for all publicly releasable ice tracings with complete and verifiable test conditions from experiments conducted to date in the Glenn Research Center Icing Research Tunnel. Ice shapes from computational simulations with the correspond ing conditions performed utilizing the latest version of the LEWICE ice shape prediction code are likewise included, and are linked to the equivalent experimental runs. The database access component includes ten Microsoft Visual Basic 6.0 (VB) form modules and three VB support modules. Together, these modules enable uploading, downloading, processing, and display of all data contained in the database. This component also affords the capability to perform various database maintenance functions for example, compacting the database or creating a new, fully initialized but empty database file.

Levinson, Laurie H.↗
NASA NTRS2012

NASA/FAA North Texas Research Station Overview

NTX Research Staion: NASA research assets embedded in an interesting operational air transport environment. Seven personnel (2 civil servants, 5 contractors). ARTCC, TRACON, Towers, 3 air carrier AOCs(American, Eagle and Southwest), and 2 major airports all within 12 miles. Supports NASA Airspace Systems Program with research products at all levels (fundamental to system level). NTX Laboratory: 5000 sq ft purpose-built, dedicated, air traffic management research facility. Established data links to ARTCC, TRACON, Towers, air carriers, airport and NASA facilities. Re-configurable computer labs, dedicated radio tower, state-of-the-art equipment.

Borchers, Paul F.↗