Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Scientific Data Management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 217 records · Page 12

Integrated Science Investigation of the Sun (ISIS): Design of the Energetic Particle Investigation

The Integrated Science Investigation of the Sun (ISIS) is a complete science investigation on the Solar Probe Plus (SPP) mission, which flies to within nine solar radii of the Sun's surface. ISIS comprises a two-instrument suite to measure energetic particles over a very broad energy range, as well as coordinated management, science operations, data processing, and scientific analysis. Together, ISIS observations allow us to explore the mechanisms of energetic particles dynamics, including their: (1) Origins-defining the seed populations and physical conditions necessary for energetic particle acceleration; (2) Acceleration-determining the roles of shocks, reconnection, waves, and turbulence in accelerating energetic particles; and (3) Transport-revealing how energetic particles propagate from the corona out into the heliosphere. The two ISIS Energetic Particle Instruments measure lower (EPI-Lo) and higher (EPI-Hi) energy particles. EPI-Lo measures ions and ion composition from approx. 20 keV/nucleon-15 MeV total energy and electrons from approx.25-1000 keV. EPI-Hi measures ions from approx. 1-200 MeV/nucleon and electrons from approx. 0.5-6 MeV. EPI-Lo comprises 80 tiny apertures with fields-of-view (FOVs) that sample over nearly a complete hemisphere, while EPI-Hi combines three telescopes that together provide five large-FOV apertures. ISIS observes continuously inside of 0.25 AU with a high data collection rate and burst data (EPI-Lo) coordinated with the rest of the SPP payload; outside of 0.25 AU, ISIS runs in low-rate science mode whenever feasible to capture as complete a record as possible of the solar energetic particle environment and provide calibration and continuity for measurements closer in to the Sun. The ISIS Science Operations Center plans and executes commanding, receives and analyzes all ISIS data, and coordinates science observations and analyses with the rest of the SPP science investigations. Together, ISIS' unique observations on SPP will enable the discovery, untangling, and understanding of the important physical processes that govern energetic particles in the innermost regions of our heliosphere, for the first time. This paper summarizes the ISIS investigation at the time of the SPP mission Preliminary Design Review in January 2014.

Coronal Mass Ejections↗

Accelerating Science: A Computing Research Agenda

The emergence of "big data" offers unprecedented opportunities for not only accelerating scientific advances but also enabling new modes of discovery. Scientific progress in many disciplines is increasingly enabled by our ability to examine natural phenomena through the computational lens, i.e., using algorithmic or information processing abstractions of the underlying processes; and our ability to acquire, share, integrate and analyze disparate types of data. However, there is a huge gap between our ability to acquire, store, and process data and our ability to make effective use of the data to advance discovery. Despite successful automation of routine aspects of data management and analytics, most elements of the scientific process currently require considerable human expertise and effort. Accelerating science to keep pace with the rate of data acquisition and data processing calls for the development of algorithmic or information processing abstractions, coupled with formal methods and tools for modeling and simulation of natural processes as well as major innovations in cognitive tools for scientists, i.e., computational tools that leverage and extend the reach of human intellect, and partner with humans on a broad range of tasks in scientific discovery (e.g., identifying, prioritizing formulating questions, designing, prioritizing and executing experiments designed to answer a chosen question, drawing inferences and evaluating the results, and formulating new questions, in a closed-loop fashion). This calls for concerted research agenda aimed at: Development, analysis, integration, sharing, and simulation of algorithmic or information processing abstractions of natural processes, coupled with formal methods and tools for their analyses and simulation; Innovations in cognitive tools that augment and extend human intellect and partner with humans in all aspects of science.

97 MATHEMATICS AND COMPUTING↗

Data Management and Its Role in Delivering Science at DOE BES User Facilities – Past, Present, and Future

Abstract. The primary mission of user facilities operated by Basic Energy Sciences under the Department of Energy is to produce data for users in support of open science and basic research [1]. We trace back almost 30 years of history across selected user facilities illustrating the evolution of facility data management practices and how these practices have related to performing scientific research. The facilities cover multiple techniques such as X-ray and neutron scattering, imaging and tomography sciences. Over time, detector and data acquisition technologies have dramatically increased the ability to produce prolific volumes of data challenging the traditional paradigm of users taking data home upon completion of their experiments to process and publish their results. During this time, computing capacity has also increased dramatically, though the size of the data has grown significantly faster than the capacity of one’s laptop to manage and process this new facility produced data. Trends indicate that this will continue to be the case for yet some time. Thus users face a quandary for how to manage today’s data complexity and size as these may exceed the computing resources users have available to themselves. This same quandary can also stifle collaboration and sharing. Realizing this, some facilities are already providing web portal access to data and computing thereby providing users access to resources they need [2]. Portal based computing is now driving researchers to think about how to use the data collected at multiple facilities in an integrated way to perform their research, and also how to collaborate and share data. In the future, inter-facility data management systems will enable next tier cross-instrument-cross facility scientific research fuelled by smart applications residing upon user computer resources. We can learn from the medical imaging community that has been working since the early 1990’s to integrate data from across multiple modalities to achieve better diagnoses [3] – similarly, data fusion across BES facilities will lead to new scientific discoveries.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Information Management for a Large Multidisciplinary Project

In 1989, NASA's Langley Research Center (LaRC) initiated the High-Speed Airframe Integration Research (HiSAIR) Program to develop and demonstrate an integrated environment for high-speed aircraft design using advanced multidisciplinary analysis and optimization procedures. The major goals of this program were to evolve the interactions among disciplines and promote sharing of information, to provide a timely exchange of information among aeronautical disciplines, and to increase the awareness of the effects each discipline has upon other disciplines. LaRC historically has emphasized the advancement of analysis techniques. HiSAIR was founded to synthesize these advanced methods into a multidisciplinary design process emphasizing information feedback among disciplines and optimization. Crucial to the development of such an environment are the definition of the required data exchanges and the methodology for both recording the information and providing the exchanges in a timely manner. These requirements demand extensive use of data management techniques, graphic visualization, and interactive computing. HiSAIR represents the first attempt at LaRC to promote interdisciplinary information exchange on a large scale using advanced data management methodologies combined with state-of-the-art, scientific visualization techniques on graphics workstations in a distributed computing environment. The subject of this paper is the development of the data management system for HiSAIR.

Kennie H Jones↗

Technologies and Methods Used at the Laboratory for Atmospheric and Space Physics (LASP) to Serve Solar Irradiance Data

The Laboratory for Atmospheric and Space Physics (LASP) at the University of Colorado in Boulder, USA operates the Solar Radiation and Climate Experiment (SORCE) NASA mission, as well as several other NASA spacecraft and instruments. Dozens of Solar Irradiance data sets are produced, managed, and disseminated to the science community. Data are made freely available to the scientific immediately after they are produced using a variety of data access interfaces, including the LASP Interactive Solar Irradiance Datacenter (LISIRD), which provides centralized access to a variety of solar irradiance data sets using both interactive and scriptable/programmatic methods. This poster highlights the key technological elements used for the NASA SORCE mission ground system to produce, manage, and disseminate data to the scientific community and facilitate long-term data stewardship. The poster presentation will convey designs, technological elements, practices and procedures, and software management processes used for SORCE and their relationship to data quality and data management standards, interoperability, NASA data policy, and community expectations.

Pankratz, Chris↗

Data Crosscutting Requirements Review

In April 2013, a diverse group of researchers from the U.S. Department of Energy (DOE) scientific community assembled to assess data requirements associated with DOE-sponsored scientific facilities and large-scale experiments. Participants in the review included facilities staff, program managers, and scientific experts from the offices of Basic Energy Sciences, Biological and Environmental Research, High Energy Physics, and Advanced Scientific Computing Research. As part of the meeting, review participants discussed key issues associated with three distinct aspects of the data challenge: 1) processing, 2) management, and 3) analysis. These discussions identified commonalities and differences among the needs of varied scientific communities. They also helped to articulate gaps between current approaches and future needs, as well as the research advances that will be required to close these gaps. Moreover, the review provided a rare opportunity for experts from across the Office of Science to learn about their collective expertise, challenges, and opportunities. The "Data Crosscutting Requirements Review" generated specific findings and recommendations for addressing large-scale data crosscutting requirements.

ASCR↗

Performance characterization of scientific workflows for the optimal use of Burst Buffers

Scientific discoveries are significantly rely on the analysis of large volumes of data from observations and simulations of complex phenomena. Scientists compose the complex analyses as workflows and execute them on large-scale HPC systems. The workflow structures are in contrast with monolithic single simulations that have often been the primary use case on HPC systems. At the same time, new storage paradigms such as Burst Buffers are becoming available on HPC platforms. In this paper, we analyze the performance characteristics of a Burst Buffer and two representative scientific workflows with the aim of optimizing the usage of a Burst Buffer, extending our previous analyses (Daley et al., 2016). Our key contributions are (a) developing a performance analysis methodology pertinent to Burst Buffers, (b) improving the use of a Burst Buffer in workflows with bandwidth-sensitive and metadata-sensitive I/O workloads, (c) highlighting the key data management challenges when incorporating a Burst Buffer in the studied scientific workflows.

97 MATHEMATICS AND COMPUTING↗

Data Management at the Advanced Photon Source

Data are crucial to the scientific discoveries enabled by experiments performed at the APS. Presently the APS generates approximately 10 PB of raw experimental data per year from its sixty-six operating beamlines that house over 100 unique instruments. This data is generated as a part of over 6,000 annual experiments performed by over 5,500 facility users each year. Similar to other synchrotrons, the amount of data generated at the APS continues to quickly increase due to beamline advances, such as new measurement techniques, technological advances in detectors, high-throughput instrumentation, multi-modal instruments that can acquire several measurements in a single experiment, and advanced data processing algorithms. This trend is expected to continue in the future. The revolutionized high-energy synchrotron facility proposed in the APS-U project will increase brightness and coherence, leading to further increases in data rates. Successful management of data is of vital importance to the current and future scientific productivity of the APS.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Data Management and Science at DOE BES User Facilities – Past, Present, and Future

The primary mission of user facilities operated by Basic Energy Sciences under the Department of Energy is to produce data for users in support of open science and basic research. We trace back almost 30 years of history across selected user facilities illustrating the evolution of facility data management practices and how these practices have related to performing scientific research. The facilities cover multiple techniques such as X-ray and neutron scattering, imaging and tomography sciences. Over time, detector and data acquisition technologies have dramatically increased the ability to produce prolific volumes of data challenging the traditional paradigm of users taking data home upon completion of their experiments to process and publish their results. During this time, computing capacity has also increased dramatically, though the size of the data has grown significantly faster than the capacity of one's laptop to manage and process this new facility produced data. Trends indicate that this will continue to be the case for yet some time. Thus users face a quandary for how to manage today's data complexity and size as these may exceed the computing resources users have available to themselves. This same quandary can also stifle collaboration and sharing. Realizing this, some facilities are already providing web portal access to data and computing thereby providing users access to resources they need. Portal based computing is now driving researchers to think about how to use the data collected at multiple facilities in an integrated way to perform their research, and also how to collaborate and share data. In the future, inter-facility data management systems will enable next tier cross-instrument-cross facility scientific research fuelled by smart applications residing upon user computer resources. We can learn from the medical imaging community that has been working since the early 1990's to integrate data from across multiple modalities to achieve better diagnoses - similarly, data fusion across BES facilities will lead to new scientific discoveries.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

HEP Software Foundation Community White Paper Working Group -- Data Organization, Management and Access (DOMA)

Without significant changes to data organization, management, and access (DOMA), HEP experiments will find scientific output limited by how fast data can be accessed and digested by computational resources. In this white paper we discuss challenges in DOMA that HEP experiments, such as the HL-LHC, will face as well as potential ways to address them. A research and development timeline to assess these changes is also proposed.

Berzano, Dario↗

Data Archive and Portal (DAP) Platform for Solid Phase Processing Technologies

The scale and speed of data generated by modern scientific experiments have constantly challenged the research community to store, curate, manage and optimally use it to drive scientific discoveries. In this work, we have developed a data archive and portal (DAP) platform including analytics capabilities to collect, curate, and manage data and metadata stream for solid phase processing (SPP) techniques. We successfully hosted around ~347K files of data related to processing parameters, microscopic images, and spectroscopic data related to solid phase processing. The DAP platform for SPP will establish an enduring capability to support machine learning and grow collaboration at the intersection of materials science and data science.

36 MATERIALS SCIENCE↗

Resource Management for Processing Wide Area Data Streams on Supercomputers

Modern scientific instruments generate enormous amount of data. Typically, the data collected from the instruments are stored in one or more files that are then moved to a distant supercomputer for processing. The final results are sent back to the user. In order to make effective use of the time on expensive instruments, experimenters want to process the data as they are generated. They want to stream the data from instruments' memory directly to a supercomputer's memory for analysis. Since the compute nodes in a supercomputer are not connected directly to the wide area network, the data streams need to be passed through intermediate gateway nodes. As opposed to the best effort file transfers, data streaming applications require resources at a specific time for a specific period. In this paper, we present a system model for enabling data streaming through gateway nodes and an algorithm to efficiently allocate gateway node resources along with compute nodes. We evaluate the algorithm using real-world traces on the Chameleon Cloud. The results show that our system can schedule compute and gateway resources efficiently for streaming analysis.

data streaming↗

Performance Understanding and Analysis for Exascale Data Management Workflows

The goal of the Exascale Data Management Workflows project is to create tools to support techniques for in-situ data analysis and data management. Online data management co-running with simulations accelerates the scientific processes being carried out, provides rapid and timely scientific insights, and can help avoid unnecessary and scientifically invalid simulation computations, particularly when combined with or able to utilize data from experimental instrument and/or from past simulation runs. The University of Oregon team is contributing to the performance instrumentation and analysis aspect of this project. Our focus has been on efficient performance monitoring of data management at scale. The general approach of the MONA project is depicted in Figure 1 below. The figure shows performance monitoring applied to an I/O workflow associated with a scientific simulation, with online measurements captured from this workflow informing methods for workflow reconfiguration and dynamic adjustment. The goal is to maintain suitable levels of Quality of Service for workflow execution, by understanding the under lying causes of workflow performance and behavior. One outcome is workflow performance models able to characterize realistic workflows. Another outcome is performance ’mini-apps’ implementing such workflow behavior. Out of scope for this project are advanced methods for online workflow control.

97 MATHEMATICS AND COMPUTING↗

NASA Scientific Data Purchase Project: From Collection to User

NASA's Scientific Data Purchase (SDP) project is currently a $70 million operation managed by the Earth Science Applications Directorate at Stennis Space Center. The SDP project was developed in 1997 to purchase scientific data from commercial sources for distribution to NASA Earth science researchers. Our current data holdings include 8TB of remote sensing imagery consisting of 18 products from 4 companies. Our anticipated data volume is 60 TB by 2004, and we will be receiving new data products from several additional companies. Our current system capacity is 24 TB, expandable to 89 TB. Operations include tasking of new data collections, archive ordering, shipment verification, data validation, distribution, metrics, finances, customer feedback, and technical support. The program has been included in the Stennis Space Center Commercial Remote Sensing ISO 9001 registration since its inception. Our operational system includes automatic quality control checks on data received (with MatLab analysis); internally developed, custom Web-based interfaces that tie into commercial-off-the-shelf software; and an integrated relational database that links and tracks all data through operations. We've distributed nearly 1500 datasets, and almost 18,000 data files have been downloaded from our public web site; on a 10-point scale, our customer satisfaction index is 8.32 at a 23% response level. More information about the SDP is available on our Web site.

Nicholson, Lamar↗

A Scientific Data Processing Framework for Time Series NetCDF Data

ARM Data Integrator (ADI) is a framework to streamline the development of scientific algorithms that analyze time-series NetCDF data, and to improve the content and consistency of the output data products produced by these algorithms. ADI achieves these goals by automating the process of retrieving and preparing data for analysis, supporting the definition of output data products through a graphical interface, and providing a modular, flexible software development architecture. The input data, preprocessing, and output data specifications are defined through a graphical interface and stored in a database. ADI also includes a workflow for data integration, a library of software modules to support the workflow, and a source code generator that produces C, IDL and Python templates. Data preparation support includes automated retrieval of data from input files, merging the retrieved data into appropriately sized chunks, and transformation of the data onto a common coordinate system grid. Through the graphical interface, users can view the details of both their data products and those in the ARM catalog. The variable and attribute definitions of the existing data products can be used to build new output data products. In addition, the rules that make up the ARM archive’s data standards are laid on top of the view of the new data product providing the user with a visual cue indicating where their output violates an archive standard. The necessary configurations are stored in a database that is accessed by the ADI libraries. This paper discusses the ADI framework, its supporting components, and how ADI can significantly decrease the time and cost of implementing scientific algorithms while improving the ability of scientists to disseminate their results.

Atmospheric Science↗

CAD/CAM data management needs, requirements and options

The requirements for a data management system in support of technical or scientific applications and possible courses of action were reviewed. Specific requirements were evolved while working towards higher level integration impacting all phases of the current design process and through examination of commercially marketed systems and related data base research. Arguments are proposed for varied approaches in implementing data base systems ranging from no action necessary to immediate procurement of an existing data base management system.

Lopatka, R. S.↗

Experiences with a Flexible User Research Process to Build Data Change Tools

Scientific software development processes are understood to be distinct from commercial software development practices due to uncertain and evolving states of scientific knowledge. Sustaining these software products is a recognized challenge, but under-examined is the usability and usefulness of such tools to their scientific end users. User research is a well-established set of techniques (e.g., interviews, mockups, usability tests) applied in commercial software projects to develop foundational, generative, and evaluative insights about products and the people who use them. Currently these approaches are not commonly applied and discussed in scientific software development work. The use of user research techniques in scientific environments can be challenging due to the nascent, fluid problem spaces of scientific work, varying scope of projects and their user communities, and funding/economic constraints on projects.In this paper, we reflect on our experiences undertaking a multi-method user research process in the Deduce project. The Deduce project is investigating data change to develop metrics, methods, and tools that will help scientists make decisions around data change. There is a lack of common terminology since the concept of systematically measuring and managing data change is under explored in scientific environments. To bridge this gap we conducted user research that focuses on user practices, needs, and motivations to help us design and develop metrics and tools for data change. This paper contributes reflections and the lessons we have learned from our experiences. We offer key takeaways for scientific software project teams to effectively and flexibly incorporate similar processes into their projects.

97 MATHEMATICS AND COMPUTING↗

Tracking-Data-Conversion Tool

Object Oriented Data Technology (OODT) is a software framework for creating a Web-based system for exchange of scientific data that are stored in diverse formats on computers at different sites under the management of scientific peers. OODT software consists of a set of cooperating, distributed peer components that provide distributed peer-topeer (P2P) services that enable one peer to search and retrieve data managed by another peer. In effect, computers running OODT software at different locations become parts of an integrated data-management system.

Flora-Adams, Dana↗