Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Scientific Data Management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 361 records · Page 20

Seismology in the Cloud: A New Streaming Workflow

Data-intensive research in seismology is experiencing a recent boom, driven in part by large volumes of available data and advances in the growing field of data science. However, there are significant barriers to processing large data volumes, such as long retrieval times from data repositories, complex data management, and limited computational resources. New tools and platforms have reduced the barriers to entry for scientific cluster computing, including the maturation of the commercial cloud as an accessible instrument for research. Here, we build a customized research cluster in the cloud to test a new workflow for large-scale seismic analysis, in which data are processed as a stream (retrieved on-the-fly and acted upon without storing), with data from the Incorporated Research Institutions for Seismology Data Management Center. We use this workflow to deploy a spectral peak detection algorithm over 5.6 TB of compressed continuous seismic data from 2074 stations of the USArray Transportable Array EarthScope network. Using a 50-node cluster in the cloud, we completed the noise survey in 80 hr, with an average data throughput of 1.7 GB per minute. By varying cluster sizes, we find the scaling of our analysis to be sublinear, due to a combination of algorithmic limitations and data center response times. The cloud-based streaming workflow represents an order-of-magnitude increase in acquisition and processing speed compared to a traditional download-store-process workflow, and offers the additional benefits of employing a flexible, accessible, and widely used computing architecture. It is limited, however, due to its reliance on Internet transfer speeds and data center service capacity, and may not work well for repeated analyses or those for which even higher data throughputs are needed. These research applications will require a new class of cloud-native approaches in which both data and analysis are in the cloud.

58 GEOSCIENCES↗

Systems aspects of COBE science data compression

A general approach to compression of diverse data from large scientific projects has been developed and this paper addresses the appropriate system and scientific constraints together with the algorithm development and test strategy. This framework has been implemented for the COsmic Background Explorer spacecraft (COBE) by retrofitting the existing VAS-based data management system with high-performance compression software permitting random access to the data. Algorithms which incorporate scientific knowledge and consume relatively few system resources are preferred over ad hoc methods. COBE exceeded its planned storage by a large and growing factor and the retrieval of data significantly affects the processing, delaying the availability of data for scientific usage and software test. Embedded compression software is planned to make the project tractable by reducing the data storage volume to an acceptable level during normal processing.

Freedman, I.↗

The Virtual Climate Data Server (vCDS): An iRODS-Based Data Management Software Appliance Supporting Climate Data Services and Virtualization-as-a-Service in the NASA Center for Climate Simulation

Scientific data services are becoming an important part of the NASA Center for Climate Simulation's mission. Our technological response to this expanding role is built around the concept of a Virtual Climate Data Server (vCDS), repetitive provisioning, image-based deployment and distribution, and virtualization-as-a-service. The vCDS is an iRODS-based data server specialized to the needs of a particular data-centric application. We use RPM scripts to build vCDS images in our local computing environment, our local Virtual Machine Environment, NASA s Nebula Cloud Services, and Amazon's Elastic Compute Cloud. Once provisioned into one or more of these virtualized resource classes, vCDSs can use iRODS s federation capabilities to create an integrated ecosystem of managed collections that is scalable and adaptable to changing resource requirements. This approach enables platform- or software-asa- service deployment of vCDS and allows the NCCS to offer virtualization-as-a-service: a capacity to respond in an agile way to new customer requests for data services.

Schnase, John L.↗

Panta Rhei benchmark dataset: socio-hydrological data of paired events of floods and droughts

As the adverse impacts of hydrological extremes increase in many regions of the world, a better understanding of the drivers of changes in risk and impacts is essential for effective flood and drought risk management and climate adaptation. However, there is currently a lack of comprehensive, empirical data about the processes, interactions, and feedbacks in complex human–water systems leading to flood and drought impacts. Here we present a benchmark dataset containing socio-hydrological data of paired events, i.e. two floods or two droughts that occurred in the same area. The 45 paired events occurred in 42 different study areas and cover a wide range of socio-economic and hydro-climatic conditions. The dataset is unique in covering both floods and droughts, in the number of cases assessed and in the quantity of socio-hydrological data. The benchmark dataset comprises (1) detailed review-style reports about the events and key processes between the two events of a pair; (2) the key data table containing variables that assess the indicators which characterize management shortcomings, hazard, exposure, vulnerability, and impacts of all events; and (3) a table of the indicators of change that indicate the differences between the first and second event of a pair. The advantages of the dataset are that it enables comparative analyses across all the paired events based on the indicators of change and allows for detailed context- and location-specific assessments based on the extensive data and reports of the individual study areas. The dataset can be used by the scientific community for exploratory data analyses, e.g. focused on causal links between risk management; changes in hazard, exposure and vulnerability; and flood or drought impacts. The data can also be used for the development, calibration, and validation of socio-hydrological models. The dataset is available to the public through the GFZ Data Services (Kreibich et al., 2023, https://doi.org/10.5880/GFZ.4.4.2023.001).

54 ENVIRONMENTAL SCIENCES↗

Energy Material Network Data Hubs

In early 2015 the United States Department of Energy conceived of a consortium of collaborative bodies based on shared expertise, data, and resources that could be targeted towards the more difficult problems in energy materials research. The concept of virtual laboratories had been envisioned and discussed earlier in the decade in response to the advent of the Materials Genome Initiative and similar scientific thrusts. To be effective, any virtual laboratory needed a robust method for data management, communication, security, data sharing, dissemination, and demonstration to work efficiently and effectively for groups of remote researchers. With the accessibility of new, easily deployed cloud technology and software frameworks, such individual elements could be integrated, and the required collaboration architecture is now possible. The developers have leveraged open-source software frameworks, customized them, and merged them into a platform to enable collaborative energy materials science, regardless of the geographic dispersal of the people and resources. After five years in operations, the systems are demonstratively an effective platform for enabling research within the Energy Material Networks (EMN). This paper will show the design and development of a secured scientific data sharing platform, the ability to customize the system to support diverse workflows, and examples of the enabled research and results connected with some of the Energy Material Networks.

97 MATHEMATICS AND COMPUTING↗

Enabling Command-and-Control in Advanced In Situ Workflows

Scientific discovery is progressing towards autonomous science with the combination of scientific instruments, high-performance computing, and artificial intelligence in complex workflows. This evolution introduces new requirements for managing scientific workflows, including feedback loops, near real-time constraints, and the ability to dynamically control workflow execution. In situ workflows that analyze and visualize data as it is generated are well-suited to satisfy stringent time constraints and their iterative nature offers greater opportunities for command-and-control. However, only a few of the many workflow management systems available have been specifically designed to manage in situ workflows and often lack support for automated feedback loops that allow analysis and visualization components to interact with the main scientific data producer. To address this need, we present in this paper how to add command-and-control capabilities to a workflow management system. We identify the functional design requirements of such a command-and-control system, detail its architecture, interface, and core mechanisms, and illustrate how advanced in situ workflows can leverage command-and-control in three use cases: graceful termination with checkpoint, dynamic and adaptive data reduction, and event-triggered analysis.

Mehta, Kshitij [ORNL] (ORCID:0000000297149981)↗

A View from ORNL: Scientific Data Research Opportunities in the Big Data Age

One of the core issues across computer and computational science today is adapting to, managing, and learning from the influx of "Big Data". In the commercial space, this problem has led to a huge investment in new technologies and capabilities that are well adapted to dealing with the sorts of human-generated logs, videos, texts, and other large-data artifacts that are processed and resulted in an explosion of useful platforms and languages (Hadoop, Spark, Pandas, etc.). However, translating this work from the enterprise space to the computational science and HPC community has proven somewhat difficult, in part because of some of the fundamental differences in type and scale of data and timescales surrounding its generation and use. We describe a forward-looking research and development plan which centers around the concept of making Input/Output (I/O) intelligent for users in the scientific community, whether they are accessing scalable storage or performing in situ workflow tasks. Much of our work is based on our experience with the Adaptable I/O System (ADIOS 1.X), and our next generation version of the software ADIOS 2.X [1].

Klasky, Scott↗

Database Maintenance, Data Sharing Policy, Collaboration

Scientific questions of today are now more global than ever before. The answers to these questions are buried within multiple disciplines and across a diverse range of scientists and institutions. The expanse and complexity of data required by researchers often exceed the means of a single scientist. Data sharing in the form of its distributed collection and analysis is increasingly common. Collective research now takes place in what may be called 'collaboratories' or in 'centers without walls' (Clery 2006). Creating effective artifacts, which enable scientists to collaborate on data analyses, continues to be a significant challenge for today's science activities. It is rare that providing a file system abstraction on distributed data enables acceleration of scientific discoveries. By explicitly identifying and addressing the different requirements for data contributors, data curators, and data consumers, we can create a data management architecture which enables the creation of datasets that evolve over time with growing and changing data, data annotations, participants, and use rules. This involves also a crucial contribution by the teams and people collecting the data, that in addition to carefully acquire and process the measurements and to be ready to share their measurements within the scientific community, need to follow general rules that help to make their data well documented and safely stored and to maximize visibility to their works and sites. In this chapter, we provide examples of the types of functions and capabilities typically provided within the data management systems, focusing in particular on databases structures and characteristics, data practices, and data user services. Finally, the importance and advantages of collective efforts like data sharing for synthesis activities and the relative data policy options are discussed and analyzed.

99 GENERAL AND MISCELLANEOUS↗

Integrated Hydro-Terrestrial Modeling: Development of a National Capability

Water is one of our most important natural resources and is essential to our national economy and security. Multiple federal government agencies have mission elements that address national needs related to water. Each water-related agency champions a unique science and/or operational mission focused on advancing a portion of the nation’s ability to meet our water-related challenges. These diverse mission needs have engendered a rich and extensive base of water-related data and modeling capabilities. While useful for their intended purposes, these capabilities are not well integrated to address complex regional problems and overarching national problems. These major investments by a number of federal agencies, however, lay the foundation for an integrated hydro-terrestrial modeling and data infrastructure that will enhance knowledge, understanding, prediction, and management of the nation’s diverse water challenges. Creating a more seamless national hydro-terrestrial modeling and data capacity presents an enormous opportunity to advance operational as well as research capabilities leading to more effective water management. Advances are necessary not only in operational tools for forecasting but also in research to identify and resolve knowledge and data gaps that lead to unacceptable uncertainties in forecast outcomes. As such, close coordination across scientific, operational, and resource management communities is required. To this end, an interagency workshop on “Integrated Hydro-Terrestrial Modeling: Development of a National Capacity” was held at the National Science Foundation (NSF) headquarters in Alexandria, Virginia in September 2019, led jointly by the NSF, the U.S. Department of Energy (DOE) and the U.S. Geological Survey (USGS) and with broader interagency support provided through an interagency steering committee. This workshop provided a venue to bring together representatives of water-related agencies and their scientific partners (including university researchers) to initiate and refine a vision for a national Integrated Hydro-Terrestrial Modeling (IHTM) and data infrastructure and advance ideas towards its development. The workshop was designed to address three critical foci to advance the development of a national IHTM capacity: “Priority Water Challenges” around which to motivate and initiate development; Technical and methodological obstacles related to data and modeling; Organizational, structural, and cultural barriers that heretofore have impeded integration of capabilities across the federal and research landscapes. The following “Priority Water Challenge” domain areas represent targets for initiating development of the IHTM and were identified and selected in alignment with priorities of the administration’s Water Sub-Cabinet: (1) Nutrient loading, hypoxia, and harmful algal blooms; (2) Water availability in the western United States; and (3) Extreme weather-related water hazards. These water challenges span agency mission boundaries and encompass a broad range of geographies, complex system dynamics and feedbacks, and critical processes spanning hydrological, climatic, and biophysical systems as well as land-use/land-cover, agricultural, built infrastructure, societal, economic, and decisional environments. These three Priority Water Challenges cannot be fully addressed without leveraging complementary and synergistic capabilities across multiple agencies.

99 GENERAL AND MISCELLANEOUS↗

NASA space and Earth science data on CD-ROM

The National Space Science Data Center (NSSDC) is very interested in facilitating the widest possible use of the scientific data acquired through NASA spaceflight missions. Therefore, NSSDC has participated with projects and data management elements throughout the NASA science environment in the creation, archiving, and dissemination of data using Compact Disk-Read Only Memory (CD-ROM). This CD-ROM technology has the potential to enable the dissemination of very large data volumes at very low prices to a great many researchers, students and their teachers, and others. This catalog identifies and describes the scientific CD-ROM's now available from NSSDC including the following data sets: Einstein Observatory CD-ROM, Galileo Cruise Imaging on CD-ROM, International Halley Watch, IRAS Sky Survey Atlas, Infrared Thermal Mapper (IRTM), Magellan (MIDR), Magellan (ARCDR's), Magellan (GxDR's), Mars Digital Image Map (MDIM), Outer Planets Fields & Particles Data, Pre-Magellan, Selected Astronomical Catalogs, TOMS Gridded Ozone Data, TOMS Ozone Image Data, TOMS Update, Viking Orbiter Images of Mars, and Voyager Image.

Towheed, Syed S.↗

HPC Campaign Management: Remote data access with user-defined error bound using ADIOS and ZFP

Remote access to large-scale scientific datasets, like those generated by combustion simulations or other high-performance computing (HPC) applications, presents a significant challenge. Downloading entire datasets is often impractical due to their size and the bandwidth limitations of typical networks. To address this challenge, we propose a novel approach that enables efficient remote access to large datasets distributed across multiple facilities. Our method enables technologies to download only the data values of a select variable, in a select region of interest, to a user-defined accuracy. For this purpose, we extended the ADIOS IO library to provide read functions with user-defined accuracy, a remote data server that understands multidimensional selections of specific variables, steps and accuracy from an ADIOS dataset, and which uses lossy compression on the remote site to reduce the data to be transferred back to the client. In addition, our extension of the ADIOS library collects metadata from multiple datasets in small files called Campaign Archives, which can be shared among project participants on any HPC, cloud or laptop, and which can easily facilitate the discovery of content and pointers to the data location as well as remote access to the data by local tools as if data was local. This feature called Campaign Management, enables a group of scientists to manage related datasets stored in multiple files, across multiple facilities as if it was in a single file/database. We demonstrate the effectiveness of our approach using a 1.5 TB dataset from the S3D combustion simulation on Frontier at the Oak Ridge Leadership Facility. Even a single variable from this dataset, at 64 GB, is too large to be processed on a standard laptop. We show two different reading patterns for 2D plots and 3D visualization, with careful settings that a scientist studying combustion data would do and show that running the same Python scripts on Frontier directly takes comparable time than running them on the local laptop with remote access to the data on Frontier.

Podhorszki, Norbert [ORNL] (ORCID:000000019647542X↗

A Step Beyond Simple Keyword Searches: Services Enabled by a Full Content Digital Journal Archive

The problems of managing and searching large archives of scientific journal articles can potentially be addressed through data mining and statistical techniques matured primarily for quantitative scientific data analysis. A journal paper could be represented by a multivariate descriptor, e.g., the occurrence counts of a number key technical terms or phrases (keywords), perhaps derived from a controlled vocabulary ( e . g . , the American Meteorological Society's Glossary of Meteorology) or bootstrapped from the journal archive itself. With this technique, conventional statistical classification tools can be leveraged to address challenges faced by both scientists and professional societies in knowledge management. For example, cluster analyses can be used to find bundles of "most-related" papers, and address the issue of journal bifurcation (when is a new journal necessary, and what topics should it encompass). Similarly, neural networks can be trained to predict the optimal journal (within a society's collection) in which a newly submitted paper should be published. Comparable techniques could enable very powerful end-user tools for journal searches, all premised on the view of a paper as a data point in a multidimensional descriptor space, e.g.: "find papers most similar to the one I am reading", "build a personalized subscription service, based on the content of the papers I am interested in, rather than preselected keywords", "find suitable reviewers, based on the content of their own published works", etc. Such services may represent the next "quantum leap" beyond the rudimentary search interfaces currently provided to end-users, as well as a compelling value-added component needed to bridge the print-to-digital-medium gap, and help stabilize professional societies' revenue stream during the print-to-digital transition.

Boccippio, Dennis J.↗

Optimizing Sample Collection and Accessibility through the Biospecimen and Tissue Sharing Collection (BTSC) Program

The Space Radiation Element (SRE) of the Human Research Program (HRP) is dedicated to establishing a robust biospecimen and tissue sharing collection (BTSC) program that enhances sample collection, tracking, access, distribution, and usability, with the goal of maximizing scientific return. By leveraging biospecimens and tissues from previous experiments, HRP effectively achieves its scientific objectives in characterizing and mitigating the human health impacts of spaceflight while optimizing resource utilization. To further improve the usability and accessibility of the current biospecimen archive, the project aims to expand upon NASA's existing resources and institutional knowledge, ensuring ongoing modernization. To facilitate seamless navigation of the program's workflow, an educational series on the BTSC program is provided to Principal Investigators (PIs). This comprehensive series equips PIs with crucial information on submitting their inventory via the BTSC Metadata Intake Form, ultimately leading to the public availability of their data on NASA's Life Science Portal (NLSP). Covering various aspects such as metadata submission instructions and backend processes for transferring metadata to the Laboratory Information Management System (LIMS), the series incorporates guidance from NASA's Biological Institutional Scientific Collection (NBISC) and Ames Life Sciences Data Archive (ALSDA). The BTSC program represents a significant stride towards enhancing the usability and accessibility of biospecimens for space research. By enabling NASA to deepen its understanding of the health implications of long-term spaceflight, this initiative plays a pivotal role in ensuring the safety and well-being of astronauts.

Shelita Renee Augustus↗

Wrap-up session

Listening to the case studies that were presented it became quite evident that the best data management systems were the ones where data managers and research scientists worked as a team developed in the early stages of project planning. Examples that were given included WOCE Data Assembly Centres e.g., Drifters, the Global Temperature Salinity Pilot Project (GTSPP) collaboration with Joint Analyses Centres in the U.S. and Australia, and JGOFS/BOFS development of Topical Centres. While each of these has some elements unique to the project, each had brought together 'teams' of Principal Investigators (PI's) and data management experts at an early stage of project development. Conversely, projects which had considered data management as a totally separate activity with lower priority often failed to provide the service required to meet scientific objectives. Therefore, the following actions should be brought to the attention of relevant groups within the IOC and other international organizations: (1) Publicize, at the national and international level, underway data/scientist collaborations that may be used as models in planning for the future. (2) Reduce adversarial situations where data managers and scientists appear to be in competition. (3) Colocation and other forms of collaboration often results in very high quality data sets and more timely data submission. Improved timeliness of data submissions was a common theme throughout the workshop and must be considered an important element in all future plans. Improvements in timely submission of data were noted. In order to continue this trend the advantages of timely submission of data must be stressed to those planning new ocean science projects.

Holland, Geoffrey↗

Soil organic carbon is not just for soil scientists: measurement recommendations for diverse practitioners

Soil organic carbon (SOC) regulates terrestrial ecosystem functioning, provides diverse energy sources for soil microorganisms, governs soil structure, and regulates the availability of organically bound nutrients. Investigators in increasingly diverse disciplines recognize how quantifying SOC attributes can provide insight about ecological states and processes. Today, multiple research networks collect and provide SOC data, and robust, new technologies are available for managing, sharing, and analyzing large data sets. Here, we advocate that the scientific community capitalize on these developments to augment SOC data sets via standardized protocols. We describe why such efforts are important and the breadth of disciplines for which it will be helpful, and outline a tiered approach for standardized sampling of SOC and ancillary variables that ranges from simple to more complex. We target scientists ranging from those with little to no background in soil science to those with more soil-related expertise, and offer examples of the ways in which the resulting data can be organized, shared, and discoverable.

54 ENVIRONMENTAL SCIENCES↗

Exploring MDSplus data-acquisition software and custom devices

MDSplus is a software tool designed for data acquisition, storage, and analysis of complex scientific experiments. Over the years, MDSplus has primarily been used for data management for fusion experiments. This paper demonstrates that MDSplus can be used for a much wider variety of systems and experiments. We present a step-by-step tutorial describing how to create a simple experiment, manage the data, and analyze it using MDSplus and Python. To this end, a custom example device was developed to be used as the data source. This device was built on an open-source electronic hardware platform, and it consists of a microcontroller and two sensors. Finally, we read data from these sensors, store it in MDSplus, and use JupyterLab to visualize and process it.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Exploring MDSplus data-acquisition software and custom devices

MDSplus is a software tool designed for data acquisition, storage, and analysis of complex scientific experiments. Over the years, MDSplus has primarily been used for data management for fusion experiments. This paper demonstrates that MDSplus can be used for a much wider variety of systems and experiments. We present a step-by-step tutorial describing how to create a simple experiment, manage the data, and analyze it using MDSplus and Python. To this end, a custom example device was developed to be used as the data source. This device was built on an opensource electronic hardware platform, and it consists of a microcontroller and two sensors. We read data from these sensors, store it in MDSplus, and use JupyterLab to visualize and process it. This project and code demo are available on the GitHub site at this URL: https://github.com/santorofer/MDSplusAndCustomeDevices

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

1999 NCCS Highlights

The NASA Center for Computational Sciences (NCCS) is a high-performance scientific computing facility operated, maintained and managed by the Earth and Space Data Computing Division (ESDCD) of NASA Goddard Space Flight Center's (GSFC) Earth Sciences Directorate. The mission of the NCCS is to advance leading-edge science by providing the best people, computers, and data storage systems to NASA's Earth and space sciences programs and those of other U.S. Government agencies, universities, and private institutions. Among the many computationally demanding Earth science research efforts supported by the NCCS in Fiscal Year 1999 (FY99) are the NASA Seasonal-to-Interannual Prediction Project, the NASA Search and Rescue Mission, Earth gravitational model development efforts, the National Weather Service's North American Observing System program, Data Assimilation Office studies, a NASA-sponsored project at the Center for Ocean-Land-Atmosphere Studies, a NASA-sponsored microgravity project conducted by researchers at the City University of New York and the University of Pennsylvania, the completion of a satellite-derived global climate data set, simulations of a new geodynamo model, and studies of Earth's torque. This document presents highlights of these research efforts and an overview of the NCCS, its facilities, and its people.

Bennett, Jerome↗