Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Scientific Data Management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 379 records · Page 21

Soil organic carbon is not just for soil scientists: measurement recommendations for diverse practitioners

Soil organic carbon (SOC) regulates terrestrial ecosystem functioning, provides diverse energy sources for soil microorganisms, governs soil structure, and regulates the availability of organically bound nutrients. Investigators in increasingly diverse disciplines recognize how quantifying SOC attributes can provide insight about ecological states and processes. Today, multiple research networks collect and provide SOC data, and robust, new technologies are available for managing, sharing, and analyzing large data sets. Here, we advocate that the scientific community capitalize on these developments to augment SOC data sets via standardized protocols. We describe why such efforts are important and the breadth of disciplines for which it will be helpful, and outline a tiered approach for standardized sampling of SOC and ancillary variables that ranges from simple to more complex. We target scientists ranging from those with little to no background in soil science to those with more soil-related expertise, and offer examples of the ways in which the resulting data can be organized, shared, and discoverable.

54 ENVIRONMENTAL SCIENCES↗

Exploring MDSplus data-acquisition software and custom devices

MDSplus is a software tool designed for data acquisition, storage, and analysis of complex scientific experiments. Over the years, MDSplus has primarily been used for data management for fusion experiments. This paper demonstrates that MDSplus can be used for a much wider variety of systems and experiments. We present a step-by-step tutorial describing how to create a simple experiment, manage the data, and analyze it using MDSplus and Python. To this end, a custom example device was developed to be used as the data source. This device was built on an open-source electronic hardware platform, and it consists of a microcontroller and two sensors. Finally, we read data from these sensors, store it in MDSplus, and use JupyterLab to visualize and process it.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Exploring MDSplus data-acquisition software and custom devices

MDSplus is a software tool designed for data acquisition, storage, and analysis of complex scientific experiments. Over the years, MDSplus has primarily been used for data management for fusion experiments. This paper demonstrates that MDSplus can be used for a much wider variety of systems and experiments. We present a step-by-step tutorial describing how to create a simple experiment, manage the data, and analyze it using MDSplus and Python. To this end, a custom example device was developed to be used as the data source. This device was built on an opensource electronic hardware platform, and it consists of a microcontroller and two sensors. We read data from these sensors, store it in MDSplus, and use JupyterLab to visualize and process it. This project and code demo are available on the GitHub site at this URL: https://github.com/santorofer/MDSplusAndCustomeDevices

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

1999 NCCS Highlights

The NASA Center for Computational Sciences (NCCS) is a high-performance scientific computing facility operated, maintained and managed by the Earth and Space Data Computing Division (ESDCD) of NASA Goddard Space Flight Center's (GSFC) Earth Sciences Directorate. The mission of the NCCS is to advance leading-edge science by providing the best people, computers, and data storage systems to NASA's Earth and space sciences programs and those of other U.S. Government agencies, universities, and private institutions. Among the many computationally demanding Earth science research efforts supported by the NCCS in Fiscal Year 1999 (FY99) are the NASA Seasonal-to-Interannual Prediction Project, the NASA Search and Rescue Mission, Earth gravitational model development efforts, the National Weather Service's North American Observing System program, Data Assimilation Office studies, a NASA-sponsored project at the Center for Ocean-Land-Atmosphere Studies, a NASA-sponsored microgravity project conducted by researchers at the City University of New York and the University of Pennsylvania, the completion of a satellite-derived global climate data set, simulations of a new geodynamo model, and studies of Earth's torque. This document presents highlights of these research efforts and an overview of the NCCS, its facilities, and its people.

Bennett, Jerome↗

Human Support Technology Research, Development and Demonstration

The Human Support Technology research, development, and demonstration program address es the following areas at TRL: Advanced Power and Propulsion. Cryogenic fluid management. Closed-loop life support and Habitability. Extravehicular activity systems. Scientific data collection and analysis. and Planetary in-situ resource utilization.

Joshi, Jitendra↗

Onboard Radar Processing Development for Rapid Response Applications

We are developing onboard processor (OBP) technology to streamline data acquisition on-demand and explore the potential of the L-band SAR instrument onboard the proposed DESDynI mission and UAVSAR for rapid response applications. The technology would enable the observation and use of surface change data over rapidly evolving natural hazards, both as an aid to scientific understanding and to provide timely data to agencies responsible for the management and mitigation of natural disasters. We are adapting complex science algorithms for surface water extent to detect flooding, snow/water/ice classification to assist in transportation/ shipping forecasts, and repeat-pass change detection to detect disturbances. We are near completion of the development of a custom FPGA board to meet the specific memory and processing needs of L-band SAR processor algorithms and high speed interfaces to reformat and route raw radar data to/from the FPGA processor board. We have also developed a high fidelity Matlab model of the SAR processor that is modularized and parameterized for ease to prototype various SAR processor algorithms targeted for the FPGA. We will be testing the OBP and rapid response algorithms with UAVSAR data to determine the fidelity of the products.

dual polarized data↗

Union: An Automatic Workload Manager for Accelerating Network Simulation

With the rapid growth of the machine learning applications, the workloads of future HPC systems are anticipated to be a mix of scientific simulation, big data analytics, and machine learning applications. Simulation is a great research vehicle to understand the performance implications of co-running scientific applications with big data and machine learning workloads on large-scale systems. In this paper, we present Union, a workload manager that provides an automatic framework to facilitate hybrid workload simulation in CODES. Furthermore, we use Union, along with CODES, to investigate various hybrid workloads composed of traditional simulation applications and emerging learning applications on two dragonfly systems. The experiment results show that both message latency and communication time are important performance metrics to evaluate network interference. Network interference on HPC applications is more reflected by the message latency variation, whereas ML application performance depends more on the communication time.

heterogeneous workloads↗

Balloon stratospheric research flights, April 1976 to December 1976

These flights were designed to measure the vertical concentration profile of trace stratospheric species which form major links in the chlorine photochemical system of the upper atmosphere, to measure the vertical concentration profiles of atomic oxygen, the hydroxyl radical and ozone in the stratosphere. An overview of the scientific goals of the program, a statement of program management and support functions, a brief description of the instrumentation flown, pertinent engineering and payload operations data, and a summary of the scientific data obtained for four flights are presented.

Allen, N. C.↗

Dynamic object management for distributed data structures

In distributed-memory multiprocessors, remote memory accesses incur larger delays than local accesses. Hence, insightful allocation and access of distributed data can yield substantial performance gains. The authors argue for the use of dynamic data management policies encapsulated within individual distributed data structures. Distributed data structures offer performance, flexibility, abstraction, and system independence. This approach is supported by data from a trace-driven simulation study of parallel scientific benchmarks. Experimental data on memory locality, message count, message volume, and communication delay suggest that data-structure-specific data management is superior to a single, system-imposed policy.

Totty, Brian K.↗

The NASA Scientific and Technical Information (STI) Program's Implementation of Open Archives Initiation (OAI) for Data Interoperability and Data Exchange

Interoperability and data-exchange are critical for the survival of government information management programs. E-government initiatives are transforming the way the government interacts with the public. More information is to be made available through web-enabled technologies. Programs such as the NASA's Scientific and Technical Information (STI) Program Office are tasked to find more effective ways to disseminate information to the public. The NASA STI Program is an agency-wide program charged with gathering, organizing, storing, and disseminating NASA-produced information for research and public use. The program is investigating the use of a new protocol called the Open Archives Initiative (OAI) as a means to improve data interoperability and data collection. OAI promotes the use of the OAI harvesting protocol as a simple way for data sharing among repositories. In two separate initiatives, the STI Program is implementing OAI In collaboration with the Air Force, Department of Energy, and Old Dominion University, the NASA STI Program has funded research on implementing the OAI to exchange data between the three organizations. The second initiative is the deployment of OAI for the NASA technical report server (TRS) environment. The NASA TRS environment is comprised of distributed technical report servers with a centralized search interface. This paper focuses on the implementation of OAI to promote interoperability among diverse data repositories.

Rocker, JoAnne↗

Data management for JGOFS: Theory and design

The Joint Global Ocean Flux Study (JGOFS), currently being organized under the auspices of the Scientific Committee for Ocean Research (SCOR), is intended to be a decade long internationally coordinated program. The main goal of JGOFS is to determine and understand on a global scale the processes controlling the time-varying fluxes of carbon and associated biogenic elements in the ocean and to evaluate the related exchanges with the atmosphere, sea floor and continental boundaries. 'A long-term goal of JGOFS will be to establish strategies for observing, on long time scales, changes in ocean biogeochemical cycles in relation to climate change'. Participation from a large number of U.S. and foreign institutions is expected. JGOFS investigators have begun a set of time-series measurements and global surveys of a wide variety of biological, chemical and physical quantities, detailed process-oriented studies, satellite observations of ocean color and wind stress and modeling of the bio-geochemical processes. These experiments will generate data in amounts unprecedented in the biological and chemical communities; rapid and effortless exchange of these data will be important to the success of JGOFS.

Flierl, Glenn R.↗

Program status

The Pilot Climate Data System (PCDS) was designed to support a variety of users that have been arbitrarily categorized into four groups: researchers, data producers, occasional users, and management. The expanding capabilities of the system are attracting the attention of both academic and other scientific institutions worldwide. Highlighted by progress in networking capabilities, hardware acquisitions, software developments, data set additions, and tutorial developments, exciting advances have taken place since the First PCDS Workshop. In the plans for the 1986 fiscal year, recommendations from an ad hoc users' group meeting in May 1985 and from the First PCDS workshop are apparent. This year's plans are listed, along with comments made at the users' group meeting. Although the PCDS is presently considered to be in a developmental phase, plans for making the transition to an operational phase are being implemented.

Smith, P.↗

NASA Goddard Giovanni Support for YOTC

The fundamental challenges to overcoming our shortcomings in understanding and modeling/predicting tropical convection have been twofold: I) the need to represent the broad range of scales applicable to the tropical organization problem (i.e. cumulus to planetary), and II) the lack of observations that adequately and simultaneously characterize this broad range of scales and that also provide three-dimensional information on thermodynamic, radiative and dynamical interactions, including cloud microphysical processes. In regards to the second challenge, it should be stressed that this problem will not be solved through the production and examination of one or a few high quality long-term records of fundamental quantities (e.g., SST, water vapor, cloud fraction). Rather, an alternative and more comprehensive paradigm is needed, one that integrates the multitude of applicable resources and measures of tropical convection in a manner that that can be better utilized by the diagnostic, modeling and forecasting communities to more completely and coherently focus on the problem.Because the goal of YOTC involves examining a scientifically complex, multi-scale process , rather than documenting the characteristics of a single parameter (e.g., SST, cloud cover), YOTC has an IOP perspective that targets a period, May 2008 April 2010, long enough to encompass many cases of tropical convection activity in many of its most challenging yet influential forms. This includes mesoscale and synoptic variability, easterly waves and hurricanes, convectively coupled waves, the MJO and the culmination of these in terms of the monsoon, their interactions with the extra-tropics, and mean characteristics such as tropical-to-subtropical transitions. The YOTC time period and length are driven in part by the following: 1) keeping the multi-sensor/multi-platform and model-analyses data sets and associated infrastructure manageable, 2) facilitating a focused effort by the research and operational communities on a specific scientific problem, and 3) capitalizing on the recent key additions to the armada of satellites (e.g., CloudSat and CALIPSO). The proposed dissemination framework for the YOTC satellite data archive is based on the Giovanni system. Giovanni is a web-based application developed by the NASA Goddard Earth Science (GES) Data and Information Service Center (DISC) that provides a simple and intuitive way to visualize, analyze, and access/download vast amounts of Earth science remote sensing data. A prototype YOTC Giovanni System (hereafter YOTC-GS) is in the process of being developed. YOTC-GS will provide access to level 2 (i.e. swath level data) and/or level 3 (i.e. gridded/mapped data) forms of satellite data, the choice or both depending on what is appropriate and relevant. The former is needed and better suited for detailed process examination, exploiting the highest temporal-spatial resolutions available and comparison to regional cloud-system resolving model / cloud resolving model (CSRM/CRM) model output. The latter is needed and more well suited for examination of phenomena, conditions and processes on large to global scales, and for comparisons to global model analyses, prediction and simulation output.

Ostrenga, Dana↗

Big Data Analaysis and Visualization: What Do LINACS and Tropical Storms Have In Common?

While there is wisdom in the old adage "the two constants in life are death and taxes,'' there are unavoidable truths facing modern experimental and computational science. First is the growing "impedance mismatch'' between our ability to collect and generate data, and our ability to store, manage, and gain understanding from it. The second is the fact that we cannot continue to rely on the same software technologies that have worked well for the past couple of decades for data management, analysis, and visualization. A third is that these complementary activities must be considered in a holistic, rather than Balkanized way. The inseparable interplay between data management, analysis, visualization, and high performance computational infrastructure, are best viewed through the lens of case studies from multiple scientific domains, where teams of computer and scientists combine forces to tackle challenging data understanding problems.

97 MATHEMATICS AND COMPUTING↗

Velo and REXAN - Integrated Data Management and High Speed Analysis for Experimental Facilities

The Chemical Imaging Initiative at the Pacific Northwest National Laboratory (PNNL) is creating a ‘Rapid Experimental Analysis’ (REXAN) Framework, based on the concept of reusable component libraries. REXAN allows developers to quickly compose and customize high throughput analysis pipelines for a range of experiments, as well as supporting the creation of multi-modal analysis pipelines. In addition, PNNL has coupled REXAN with its collaborative data management and analysis environment Velo to create an easy to use data management and analysis environments for experimental facilities. This paper will discuss the benefits of Velo and REXAN in the context of three examples: PNNL High Resolution Mass Spectrometry - reducing analysis times from hours to seconds, and enabling the analysis of much larger data samples (100KB to 40GB) at the same time · ALS X-Ray tomography - reducing analysis times of combined STXM and EM data collected at the ALS from weeks to minutes, decreasing manual work and increasing data volumes that can be analysed in a single step ·Multi-modal nano-scale analysis of STXM and TEM data - providing a semi automated process for particle detection The creation of REXAN has significantly shortened the development time for these analysis pipelines. The integration of Velo and REXAN has significantly increased the scientific productivity of the instruments and their users by creating easy to use data management and analysis environments with greatly reduced analysis times and improved analysis capabilities.

Kleese van Dam, Kerstin↗

Report from the 2016 CrossConnects workshop: improving data mobility & management for bioinformatics

Due to significant declines in the price of genome sequencing technology, the bioinformatics sciences are experiencing a massive upswing in data generation resulting in an increasing need for data distribution and access. The sheer number of biological areas of study, many of which benefit from the scientific breakthroughs of one another, are adding to the increase of shared data usage. The need for effective data management, analysis, and access are becoming more critical. While there are commonalities facing both precision medicine and metagenomics, each area has its own unique challenges and needs. A workshop was held in April 2016 at Lawrence Berkeley National Laboratory that brought together scientists from both fields, along with experts in computing and networking. Presenters and attendees discussed current research and pressing data issues facing the bioinformatics field today and in the near future.

97 MATHEMATICS AND COMPUTING↗

Integration Of PanDA Workload Management System With Supercomputers for ATLAS and Data Intensive Science

The Large Hadron Collider (LHC), operating at the international CERN Laboratory in Geneva, Switzerland, is leading Big Data driven scientific explorations. Experiments at the LHC explore the fundamental nature of matter and the basic forces that shape our universe, and were recently credited for the discovery of a Higgs boson. ATLAS, one of the largest collaborations ever assembled in the sciences, is at the forefront of research at the LHC. To address an unprecedented multi-petabyte data processing challenge, the ATLAS experiment is relying on a heterogeneous distributed computational infrastructure. The ATLAS experiment uses PanDA (Production and Data Analysis) Workload Management System for managing the workflow for all data processing on over 150 data centers. Through PanDA, ATLAS physicists see a single computing facility that enables rapid scientific breakthroughs for the experiment, even though the data centers are physically scattered all over the world. While PanDA currently uses more than 250,000 cores with a peak performance of 0.3 petaFLOPS, LHC data taking runs require more resources than Grid computing can possibly provide. To alleviate these challenges, LHC experiments are engaged in an ambitious program to expand the current computing model to include additional resources such as the opportunistic use of supercomputers. We will describe a project aimed at integration of PanDA WMS with supercomputers in United States, Europe and Russia (in particular with Titan supercomputer at Oak Ridge Leadership Computing Facility (OLCF), MIRA supercomputer at Argonne Leadership Computing Facilities (ALCF), Supercomputer at the National Research Center Kurchatov Institute , IT4 in Ostrava and others). Current approach utilizes modified PanDA pilot framework for job submission to the supercomputers batch queues and local data management, with light-weight MPI wrappers to run single threaded workloads in parallel on LCFs multi-core worker nodes. This implementation was tested with a variety of Monte-Carlo workloads on several supercomputing platforms for ALICE and ATLAS experiments and it is in full production for the ATLAS experiment since September 2015. We will present our current accomplishments with running PanDA WMS at supercomputers and demonstrate our ability to use PanDA as a portal independent of the computing facilities infrastructure for High Energy and Nuclear Physics as well as other data-intensive science applications, such as bioinformatics and astro-particle physics.

Data-intensive science↗