Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Scientific Data Management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 343 records · Page 19

HDF5 in the exascale era: Delivering efficient and scalable parallel I/O for exascale applications

Accurately modeling real-world systems requires scientific applications at exascale to generate massive amounts of data and manage data storage efficiently. However, parallel input and output (I/O) faces challenges due to new application workflows and the state-of-the-art memory, interconnect, and storage architectures considered in exascale designs. The storage hierarchy has expanded with node-local persistent memory, solid-state storage, and traditional disk and tape-based storage, thus requiring efficiency at each layer and much more efficient data movement among these layers. This paper discusses how the ExaHDF5 project improved the I/O performance and data management for exascale architectures by enhancing HDF5, a widely used parallel I/O library. The team developed an Asynchronous I/O Virtual Object Layer (VOL) connector that allowed overlapping I/O with computation. They also created a Cache VOL to complement asynchronous I/O by incorporating fast storage layers, such as burst buffer and node-local storage, into the parallel I/O workflow through caching and staging data. Additionally, the team enabled data aggregation and I/O at the node level by using a Subfiling Virtual File Driver (VFD). To demonstrate superior I/O performance with HDF5 at exascale, the ExaHDF5 team collaborated with several exascale applications. In this paper, we show I/O performance improvements for three applications: Cabana (a particle-based simulation library), EQSIM (a regional earthquake simulation software), and E3SM (a climate system modeling library).

Asynchronous I/Ol↗

The JGOFS North Atlantic Bloom Experiment: An overview

The North Atlantic Bloom Experiment (NABE) of JGOFS presents a unique opportunity and challenge to the data management community because of the diversity and large size of biogeochemical data sets collected. NABE was a pilot study for JGOFS and has also served as a pilot study within the U.S. NODC for management and archiving of the data sets. Here I present an overview to some of the scientific results of NABE, which will be published as an Introduction to a special volume of NABE results in Deep-Sea Research later this year. An overview of NABE data management is given elsewhere in the present report. This is the first collection of papers from the Joint Global Ocean Flux Study (JGOFS). Formed as an international program in 1987, JGOFS has four principal elements: modelling and data management, multidisciplinary regional process studies, a global survey of biogeochemical properties and long-term time series observatories. In 1989-1990 JGOFS conducted a pilot process study of the spring phytoplankton bloom, the North Atlantic Bloom Experiment (NABE). JGOFS decided to conduct a large scale, internationally-coordinated pilot study in the North Atlantic because of its proximity to the founding nations of the project, the size and predictability of the bloom and its fundamental impact on ocean bio-geochemistry (Billett et al., 1983; Watson and Whitfield, 1985; Pfannkuche, 1992). In 1989, six research vessels from Canada, Germany, The Netherlands, the United Kingdom and the USA and over 200 scientists and students from more than a dozen nations participated in NABE. Some of their initial results are reported in this volume.

Ducklow, Hugh W.↗

AI applications to conceptual aircraft design

This paper presents in viewgraph form several applications of artificial intelligence (AI) to the conceptual design of aircraft, including: an access manager for automated data management, AI techniques applied to optimization, and virtual reality for scientific visualization of the design prototype.

Kathryn M. Chalfan↗

Automating CapCom Using Mobile Agents and Robotic Assistants

We have developed and tested an advanced EVA communications and computing system to increase astronaut self-reliance and safety, reducing dependence on continuous monitoring and advising from mission control on Earth. This system, called Mobile Agents (MA), is voice controlled and provides information verbally to the astronauts through programs called personal agents. The system partly automates the role of CapCom in Apollo-including monitoring and managing EVA navigation, scheduling, equipment deployment, telemetry, health tracking, and scientific data collection. EVA data are stored automatically in a shared database in the habitat/vehicle and mirrored to a site accessible by a remote science team. The program has been developed iteratively in the context of use, including six years of ethnographic observation of field geology. Our approach is to develop automation that supports the human work practices, allowing people to do what they do well, and to work in ways they are most familiar. Field experiments in Utah have enabled empirically discovering requirements and testing alternative technologies and protocols. This paper reports on the 2004 system configuration, experiments, and results, in which an EVA robotic assistant (ERA) followed geologists approximately 150 m through a winding, narrow canyon. On voice command, the ERA took photographs and panoramas and was directed to move and wait in various locations to serve as a relay on the wireless network. The MA system is applicable to many space work situations that involve creating and navigating from maps (including configuring equipment for local topology), interacting with piloted and unpiloted rovers, adapting to environmental conditions, and remote team collaboration involving people and robots.

Clancey, William J.↗

Scientific Process Automation Improves Data Interaction

This is an article written for the September 09 Scientific Computing magazine about the work of the Scientific Process Automation team of The U.S. Department of Energy (DOE) Scientific Discovery through Advanced Computing (SciDAC) program. The SPA team is focused on developing and deploying automated workflows for a variety of computational science domains. Scientific workflows are the formalization of a scientific process that is frequently and repetitively performed.

97 MATHEMATICS AND COMPUTING↗

The NASA Ames Research Center Institutional Scientific Collection: History, Best Practices and Scientific Opportunities

The NASA Ames Life Sciences Institutional Scientific Collection (ISC), which is composed of the Ames Life Sciences Data Archive (ALSDA) and the Biospecimen Storage Facility (BSF), is managed by the Space Biosciences Division and has been operational since 1993. The ALSDA is responsible for archiving information and animal biospecimens collected from life science spaceflight experiments and matching ground control experiments. Both fixed and frozen spaceflight and ground tissues are stored in the BSF within the ISC. The ALSDA also manages a Biospecimen Sharing Program, performs curation and long-term storage operations, and makes biospecimens available to the scientific community for research purposes via the Life Science Data Archive public website (https:lsda.jsc.nasa.gov). As part of our best practices, a viability testing plan has been developed for the ISC, which will assess the quality of archived samples. We expect that results from the viability testing will catalyze sample use, enable broader science community interest, and improve operational efficiency of the ISC. The current viability test plan focuses on generating disposition recommendations and is based on using ribonucleic acid (RNA) integrity number (RIN) scores as a criteria for measurement of biospecimen viablity for downstream functional analysis. The plan includes (1) sorting and identification of candidate samples, (2) conducting a statiscally-based power analysis to generate representaive cohorts from the population of stored biospecimens, (3) completion of RIN analysis on select samples, and (4) development of disposition recommendations based on the RIN scores. Results of this work will also support NASA open science initiatives and guides development of the NASA Scientific Collections Directive (a policy on best practices for curation of biological collections). Our RIN-based methodology for characterizing the quality of tissues stored in the ISC since the 1980s also creates unique scientific opportunities for temporal assessment across historical missions. Support from the NASA Space Biology Program and the NASA Human Research Program is gratefully acknowledged.

ALSDA↗

Stakeholder analysis for designing an urban air quality data governance ecosystem in smart cities

Cities, the world over, are fuelling economic growth. At the same time, rapid urbanization is a root cause of serious environmental damage. Recent WHO global air pollution guidelines highlight air pollution as a critical environmental threat along with climate change. To address these threats, smart cities and clean air programs are on a rise. In smart cities, data and Information and Communication Technologies (ICT) are major drivers of city transformations. The 4th Industrial Revolution (4IR) technologies such as the Internet of Things (IoT), big data, artificial intelligence (AI), and cloud computing have the potential to accelerate these transformations toward urban resilience. However, the success of smart cities and clean air programs depends on cohesive multi-sector stakeholder contributions. This study conducted interdisciplinary participative stakeholder analysis to understand the data, and sectorial challenges, to outline the technological opportunities to facilitate clean air programs in Indian smart cities. The research highlights gaps due to siloed stakeholder operations, lack of data calibration, non-alignment of smart city and air quality management services, non-availability of health exposure data, and difficulty in translating scientific data into implementable actions. Stakeholders expressed potential ‘fit for the purpose’ use of IoT devices, satellites, smartphones, and mobility data augmented by AI methods in bridging these gaps. In conclusion, the analysis points toward a need to develop an easily accessible and ubiquitous urban data governance ecosystem enabling seamless cross-sector data exchanges to build trusting relationships among the stakeholders across the air quality management value chain.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

EspressoDB: A scientific database for managing high-performance computing workflows

EspressoDB is a programmatic object-relational mapping (ORM) data management framework implemented in Python and based on the Django web framework. EspressoDB was developed to streamline data management, centralize and promote data integrity, while providing domain flexibility and ease of use. It is designed to directly integrate in utilized software to allow dynamical access to vast amount of relational data at runtime.

Chang, Chia↗

The development of an intelligent user interface for NASA's scientific databases

The National Space Science Data Center (NSSDC) has initiated an Intelligent Data Management (IDM) research effort which has as one of its components, the development of an Intelligent User Interface (IUI). The intent of the IUI effort is to develop a friendly and intelligent user interface service that is based on expert systems and natural language processing technologies. This paper presents the design concepts, development approach and evaluation of performance of a prototype Intelligent User Interface Subsystem (IUIS) supporting an operational database.

Campbell, William J.↗

The development of an intelligent user interface for NASA's scientific databases

The National Space Science Data Center (NSSDC) has initiated an Intelligent Data Management (IDM) research effort which has as one of its components, the development of an Intelligent User Interface (IUI). The intent of the IUI effort is to develop a friendly and intelligent user interface service that is based on expert systems and natural language processing technologies. This paper presents the design concepts, development approach and evaluation of performance of a prototype Intelligent User Interface Subsystem (IUIS) supporting an operational database.

Campbell, William J.↗

The Dark Energy Survey Data Management System

The Dark Energy Survey (DES) is a project with the goal of building, installing and exploiting a new 74 CCD-camera at the Blanco telescope, in order to study the nature of cosmic acceleration. It will cover 5000 square degrees of the southern hemisphere sky and will record the positions and shapes of 300 million galaxies up to redshift 1.4. The survey will be completed using 525 nights during a 5-year period starting in 2012. About O(1 TB) of raw data will be produced every night, including science and calibration images. The DES data management system has been designed for the processing, calibration and archiving of these data. It is being developed by collaborating DES institutions, led by NCSA. In this contribution, we describe the basic functions of the system, what kind of scientific codes are involved and how the Data Challenge process works, to improve simultaneously the Data Management system algorithms and the Science Working Group analysis codes.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Seismology in the Cloud: A New Streaming Workflow

Data-intensive research in seismology is experiencing a recent boom, driven in part by large volumes of available data and advances in the growing field of data science. However, there are significant barriers to processing large data volumes, such as long retrieval times from data repositories, complex data management, and limited computational resources. New tools and platforms have reduced the barriers to entry for scientific cluster computing, including the maturation of the commercial cloud as an accessible instrument for research. Here, we build a customized research cluster in the cloud to test a new workflow for large-scale seismic analysis, in which data are processed as a stream (retrieved on-the-fly and acted upon without storing), with data from the Incorporated Research Institutions for Seismology Data Management Center. We use this workflow to deploy a spectral peak detection algorithm over 5.6 TB of compressed continuous seismic data from 2074 stations of the USArray Transportable Array EarthScope network. Using a 50-node cluster in the cloud, we completed the noise survey in 80 hr, with an average data throughput of 1.7 GB per minute. By varying cluster sizes, we find the scaling of our analysis to be sublinear, due to a combination of algorithmic limitations and data center response times. The cloud-based streaming workflow represents an order-of-magnitude increase in acquisition and processing speed compared to a traditional download-store-process workflow, and offers the additional benefits of employing a flexible, accessible, and widely used computing architecture. It is limited, however, due to its reliance on Internet transfer speeds and data center service capacity, and may not work well for repeated analyses or those for which even higher data throughputs are needed. These research applications will require a new class of cloud-native approaches in which both data and analysis are in the cloud.

58 GEOSCIENCES↗

Systems aspects of COBE science data compression

A general approach to compression of diverse data from large scientific projects has been developed and this paper addresses the appropriate system and scientific constraints together with the algorithm development and test strategy. This framework has been implemented for the COsmic Background Explorer spacecraft (COBE) by retrofitting the existing VAS-based data management system with high-performance compression software permitting random access to the data. Algorithms which incorporate scientific knowledge and consume relatively few system resources are preferred over ad hoc methods. COBE exceeded its planned storage by a large and growing factor and the retrieval of data significantly affects the processing, delaying the availability of data for scientific usage and software test. Embedded compression software is planned to make the project tractable by reducing the data storage volume to an acceptable level during normal processing.

Freedman, I.↗

The Virtual Climate Data Server (vCDS): An iRODS-Based Data Management Software Appliance Supporting Climate Data Services and Virtualization-as-a-Service in the NASA Center for Climate Simulation

Scientific data services are becoming an important part of the NASA Center for Climate Simulation's mission. Our technological response to this expanding role is built around the concept of a Virtual Climate Data Server (vCDS), repetitive provisioning, image-based deployment and distribution, and virtualization-as-a-service. The vCDS is an iRODS-based data server specialized to the needs of a particular data-centric application. We use RPM scripts to build vCDS images in our local computing environment, our local Virtual Machine Environment, NASA s Nebula Cloud Services, and Amazon's Elastic Compute Cloud. Once provisioned into one or more of these virtualized resource classes, vCDSs can use iRODS s federation capabilities to create an integrated ecosystem of managed collections that is scalable and adaptable to changing resource requirements. This approach enables platform- or software-asa- service deployment of vCDS and allows the NCCS to offer virtualization-as-a-service: a capacity to respond in an agile way to new customer requests for data services.

Schnase, John L.↗

Panta Rhei benchmark dataset: socio-hydrological data of paired events of floods and droughts

As the adverse impacts of hydrological extremes increase in many regions of the world, a better understanding of the drivers of changes in risk and impacts is essential for effective flood and drought risk management and climate adaptation. However, there is currently a lack of comprehensive, empirical data about the processes, interactions, and feedbacks in complex human–water systems leading to flood and drought impacts. Here we present a benchmark dataset containing socio-hydrological data of paired events, i.e. two floods or two droughts that occurred in the same area. The 45 paired events occurred in 42 different study areas and cover a wide range of socio-economic and hydro-climatic conditions. The dataset is unique in covering both floods and droughts, in the number of cases assessed and in the quantity of socio-hydrological data. The benchmark dataset comprises (1) detailed review-style reports about the events and key processes between the two events of a pair; (2) the key data table containing variables that assess the indicators which characterize management shortcomings, hazard, exposure, vulnerability, and impacts of all events; and (3) a table of the indicators of change that indicate the differences between the first and second event of a pair. The advantages of the dataset are that it enables comparative analyses across all the paired events based on the indicators of change and allows for detailed context- and location-specific assessments based on the extensive data and reports of the individual study areas. The dataset can be used by the scientific community for exploratory data analyses, e.g. focused on causal links between risk management; changes in hazard, exposure and vulnerability; and flood or drought impacts. The data can also be used for the development, calibration, and validation of socio-hydrological models. The dataset is available to the public through the GFZ Data Services (Kreibich et al., 2023, https://doi.org/10.5880/GFZ.4.4.2023.001).

54 ENVIRONMENTAL SCIENCES↗

Energy Material Network Data Hubs

In early 2015 the United States Department of Energy conceived of a consortium of collaborative bodies based on shared expertise, data, and resources that could be targeted towards the more difficult problems in energy materials research. The concept of virtual laboratories had been envisioned and discussed earlier in the decade in response to the advent of the Materials Genome Initiative and similar scientific thrusts. To be effective, any virtual laboratory needed a robust method for data management, communication, security, data sharing, dissemination, and demonstration to work efficiently and effectively for groups of remote researchers. With the accessibility of new, easily deployed cloud technology and software frameworks, such individual elements could be integrated, and the required collaboration architecture is now possible. The developers have leveraged open-source software frameworks, customized them, and merged them into a platform to enable collaborative energy materials science, regardless of the geographic dispersal of the people and resources. After five years in operations, the systems are demonstratively an effective platform for enabling research within the Energy Material Networks (EMN). This paper will show the design and development of a secured scientific data sharing platform, the ability to customize the system to support diverse workflows, and examples of the enabled research and results connected with some of the Energy Material Networks.

97 MATHEMATICS AND COMPUTING↗