Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Scientific Data Management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,099 records · Page 61

Earth Resources: a continuing bibliography with indexes

This bibliography lists 337 reports, articles, and other documents introduced into the NASA scientific and technical information system between July 31, 1980 and September 30, 1980. Emphasis is placed on the use of remote sensing and geophysical instrumentation in spacecraft and aircraft to survey and inventory natural resources and urban areas. Subject matter is grouped according to agriculture and forestry, environmental changes and cultural resources, geodesy and cartography, geology and mineral resources, hydrology and water management, data processing and distribution systems, instrumentation and sensors, and economic analysis.

Source record↗

Earth Resources: A continuing bibliography, issue 28

This bibliography lists 436 reports, articles, and other documents introduced into the NASA scientific and technical information system between October 1, 1980 and December 31, 1980. Emphasis is placed on the use of remote sensing and geophysical instrumentation in spacecraft and aircraft to survey and inventory natural resources and urban areas. Subject matter is grouped according to agriculture and forestry, environmental changes and cultural resources, geodesy and cartography, geology and mineral resources, hydrology and water management, data processing and distribution systems instrumentation and sensors, and economic analysis.

Source record↗

Earth Resources: A continuing bibliography with indexes, issue 29, April 1981

This bibliography lists 308 reports, articles, and other documents introduced into the NASA scientific and technical information system between January 1, 1981 and March 31, 1981. Emphasis is placed on the use of remote sensing and geophysical instrumentation in spacecraft and aircraft to survey and inventory natural resources and urban areas. Subject matter is grouped according to agriculture and forestry, environmental changes and cultural resources, geodesy and cartography, geology and mineral resources, hydrology and water management, data processing and distribution systems, instrumentation and sensors, and economic analysis.

Source record↗

Curatorial functions and the U.S. Antarctic meteorite program

The discovery of concentrations of meteorites in Antarctica by Japanese field parties in 1969, and subsequently by joint U.S.-Japanese and U.S. field parties since 1976 has provided a significant new resource for understanding the origin and evolution of the solar system. The number of meteorites as well as the variety of meteorites has increased dramatically, and substantial amounts of data derived from their study has begun to appear in the scientific literature. The U.S. program of investigation has drawn on curatorial experience derived from the lunar program to: (1) develop specific collection and preliminary examination protocols; (2) provide documented samples for scientific investigations in response to specific requests; and (3) coordinate research by scientific consortia. The productivity of scientific research is significantly enhanced by these management approaches. Some of the results of the curatorial program for Antarctic meteorites carried out over the past three years are described.

Duke, M. B.↗

Visualization: A pathway to enhanced scientific productivity in the expanding missions of Space and Earth Sciences

The movement toward the solution of problems involving large-scale system science, the ever-increasing capabilities of three-dimensional, time-dependent numerical models, and the enhanced capabilities of 'in situ' and remote sensing instruments bring a new era of scientific endeavor that requires an important change in our approach to mission planning and the task of data reduction and analysis. Visualization is at the heart of the requirements for a much-needed enhancement in scientific productivity as we face these new challenges. This article draws a perspective on the problem as it crosses discipline boundaries from solar physics to atmospheric and ocean sciences. It also attempts to introduce visualization as a new approach to scientific discovery and a tool which expedites and improves our insight into physically complex problems. A set of simple illustrations demonstrates a number of visualization techniques and the discussion emphasizes the trial-and-error and search-and-discover modes that are necessary for the techniques to reach their full potential. Further discussions also point to the importance of integrating data access, management, mathematical operations, and visualization into a single system. Some of the more recent developments in this area are reviewed.

Szuszczewicz, E. P.↗

Kepler Science Operations Center Architecture

We give an overview of the operational concepts and architecture of the Kepler Science Data Pipeline. Designed, developed, operated, and maintained by the Science Operations Center (SOC) at NASA Ames Research Center, the Kepler Science Data Pipeline is central element of the Kepler Ground Data System. The SOC charter is to analyze stellar photometric data from the Kepler spacecraft and report results to the Kepler Science Office for further analysis. We describe how this is accomplished via the Kepler Science Data Pipeline, including the hardware infrastructure, scientific algorithms, and operational procedures. The SOC consists of an office at Ames Research Center, software development and operations departments, and a data center that hosts the computers required to perform data analysis. We discuss the high-performance, parallel computing software modules of the Kepler Science Data Pipeline that perform transit photometry, pixel-level calibration, systematic error-correction, attitude determination, stellar target management, and instrument characterization. We explain how data processing environments are divided to support operational processing and test needs. We explain the operational timelines for data processing and the data constructs that flow into the Kepler Science Data Pipeline.

Middour, Christopher↗

NAS Exploratory Concepts & Technologies (NExCT) Upper Class E Traffic Management (ETM) Collaborative Evaluation #1 (CE-1)

NASA, in partnership with AeroVironment and Aerostar, recently demonstrated a first-of-its-kind air traffic management concept that could pave the way for aircraft to safely operate at higher altitudes. This work seeks to open the door for increased internet coverage, improved disaster response, expanded scientific missions, and even supersonic flight. The concept is referred to as an Upper-Class E traffic management, or ETM. NASA and its partners have developed an ETM traffic management system that allows aircraft to autonomously share location and flight plans, enabling aircraft to stay safely separated. This concept was demonstrated during the recent traffic management simulation in the Airspace Operations Laboratory at Ames, data from multiple air vehicles was displayed across dozens of traffic control monitors and shared with partner computers off site. The study details and the initial results are presented at a regular, informal ETM industry meetings held virtually.

Upper Class E Traffic Management (ETM)↗

Performance Prediction of Big Data Transfer Through Experimental Analysis and Machine Learning

Big data transfer in next-generation scientific applications is now commonly carried out over connections with guaranteed bandwidth provisioned in High-performance Networks (HPNs) through advance bandwidth reservation. To use HPN resources efficiently, provisioning agents need to carefully schedule data transfer requests and allocate appropriate bandwidths. Such reserved bandwidths, if not fully utilized by the requesting user, could be simply wasted or cause extra overhead and complexity in management due to exclusive access. This calls for the capability of performance prediction to reserve bandwidth resources that match actual needs. Towards this goal, we employ machine learning algorithms to predict big data transfer performance based on extensive performance measurements, which are collected over a span of several years from a large number of data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated HPN testbeds. We first identify a comprehensive list of attributes involved in a typical big data transfer process, including end host system configurations, network connection properties, and control parameters of data transfer methods. We then conduct an in-depth exploratory analysis of their impacts on application-level throughput, which provides insights into big data transfer performance and motivates the use of machine learning. We also investigate the applicability of machine learning algorithms and derive their general performance bounds for performance prediction of big data transfer in HPNs. Experimental results show that, with appropriate data preprocessing, the proposed machine learning-based approach achieves 95% or higher prediction accuracy in up to 90% of the cases with very noisy real-life performance measurements.

Yun, Daqing↗

Interactive access and management for four-dimensional environmental data sets using McIDAS

This grant has fundamentally changed the way that meteorologists look at the output of their atmospheric models, through the development and wide distribution of the Vis5D system. The Vis5D system is also gaining acceptance among oceanographers and atmospheric chemists. Vis5D gives these scientists an interactive three-dimensional movie of their very large data sets that they can use to understand physical mechanisms and to trace problems to their sources. This grant has also helped to define the future direction of scientific visualization through the development of the VisAD system and its lattice data model. The VisAD system can be used to interactively steer and visualize scientific computations. A key element of this capability is the flexibility of the system's data model to adapt to a wide variety of scientific data, including the integration of several forms of scientific metadata.

Hibbard, William L.↗

BigData Express: Toward Predictable, Schedulable, and High-Performance Data Transfer [Slides]

BigData Express: a schedulable, predictable, and high-performance data transfer service. A peer-to-peer, scalable, and extensible data transfer model; A visually appealing, easy-to-use web portal; A high-performance data transfer engine; On-demand provisioning of end-to-end network paths with guaranteed QoS; Robust and flexible error handling; CILogon-based security (certificate version); A rich set of REST APIs to support scientific workflows

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Exploring the All-to-All Collective Optimization Space with ConnectX CORE-Direct

The all-to-all collective communication operation is used by many scientific applications, and is one of the most time consuming and challenging collective operation to optimize. The algorithms for all-to-all operations typically fall into two classes, logarithmic and linear scaling algorithms, with Bruck's algorithm, a logarithmic scaling algorithm, used in many small-data all-to-all implementations. The recent addition of InfiniBand CORE-Direct support for network management of collective communications offers new opportunities for optimizing all-to-all operation as well as supporting truly asynchronous implementations of these operations. This paper presents several new enhancements to the Bruck small-data algorithm that leverage CORE-Direct and other InfiniBand network capabilities to produce efficient implementations of this collective operation. These include RDMA, SR-RNR, and SR-RTR algorithms. In addition, nonblocking implementations of these collective operations are also presented. Benchmark results show that the RDMA algorithm, which uses CORE-Direct capabilities to offload collective communication management to the Host Channel Adapter (HCA), hardware gather support for sending non-continuous data, and low-latency RDMA semantics, performs the best. For a 64 processes and 128 byte-per-process all-to-all, the RDMA algorithm performs 27% better than Bruck's algorithm implementation in Open MPI and 136% better than the SR-RTR algorithm. In addition, the nonblocking versions of these algorithms have the same performance characteristics as the blocking algorithms. Finally, measurements of computation/communication overlap capacity show that all offloaded algorithms achieve about 98% overlap for large data all-to-all, whereas implementations using host-based progress achieve only about 9.5% overlap.

Venkata, Manjunath Gorentla↗

BASTet: Shareable and Reproducible Analysis and Visualization of Mass Spectrometry Imaging Data via OpenMSI

Mass spectrometry imaging (MSI) is a transformative imaging method that supports the untargeted, quantitative measurement of the chemical composition and spatial heterogeneity of complex samples with broad applications in life sciences, bioenergy, and health. While MSI data can be routinely collected, its broad application is currently limited by the lack of easily accessible analysis methods that can process data of the size, volume, diversity, and complexity generated by MSI experiments. The development and application of cutting-edge analytical methods is a core driver in MSI research for new scientific discoveries, medical diagnostics, and commercial-innovation. However, the lack of means to share, apply, and reproduce analyses hinders the broad application, validation, and use of novel MSI analysis methods. To address this central challenge, we introduce the Berkeley Analysis and Storage Toolkit (BASTet), a novel framework for shareable and reproducible data analysis that supports standardized data and analysis interfaces, integrated data storage, data provenance, workflow management, and a broad set of integrated tools. Based on BASTet, we describe the extension of the OpenMSI mass spectrometry imaging science gateway to enable web-based sharing, reuse, analysis, and visualization of data analyses and derived data products. Here, we demonstrate the application of BASTet and OpenMSI in practice to identify and compare characteristic substructures in the mouse brain based on their chemical composition measured via MSI.

97 MATHEMATICS AND COMPUTING↗

Repometer v.1.0

SAND2022-4256 O Online version control platforms offer insight into traffic and engagement with source code repositories, but only in limited ways and over short windows of time. Scientific software teams at Sandia National Laboratories and elsewhere want to collect and store engagement data to help tell their story and the impact their work has on the community. Repometer aims to supplement existing capabilities by collecting timely and insightful data from GitHub and GitLab repositories and passing it to a database for longer-term storage. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Mundt, Miranda↗

Community Modeling and Long-Term Predictions of the Integrated Water Cycle. Report from the September 2012 Workshop

Water is a key component of the earth and human systems due to its strong interactions with the energy cycle and its vital roles in the energy-water-land system. Uncertainties in predicting the integrated water cycle can limit our abilities to address the energy and environmental challenges today and in the future. Modeling the integrated water cycle contributes to the Department of Energy’s (DOE) core competencies in integrative modeling, drawing from unique and highly relevant research on cloud, aerosol, terrestrial ecosystem, carbon cycle, and subsurface processes, as well as climate and earth system modeling and integrated assessment modeling. Synthesizing new process knowledge and innovative computational methods in integrated models of the human-earth system can advance predictive capabilities relevant to DOE missions. This report describes the DOE workshop on Community Modeling and Long-Term Predictions of the Integrated Water Cycle, held September 2012 in Washington DC. The workshop serves as a launching point and major organizing event to identify challenges and plan the development of next-generation human-earth system models for improving long-term predictions of the regional-scale integrated water cycle. More specifically, the workshop aims to: identify core modeling capabilities while identifying key research gaps, with an emphasis on improving model fidelity; reveal relevant and critical capabilities and needs for observations, analytical frameworks, and data management that underpin the development, testing, and validation of our main models and model components; engage the research community in strategies for improving synthesis and integration; elucidate opportunities for collaborations within DOE and with other agencies and institutions that have complementary and essential expertise in specific aspects of the water cycle; and improve understanding of the nature and characteristics of long-term scientific information requirements for DOE’s energy and environmental missions and, as appropriate, for mission needs of other partner agencies. The workshop benefited from substantial inputs provided by the broad scientific community and interagency participation. The topical and crosscutting research challenges identified at the workshop have been synthesized into three overarching Science Grand Challenges and three Integrative Modeling Experiments summarized in this report. These challenges represent remarkable opportunities for interagency collaborations to improve predictions of the integrated water cycle for significant scientific and user impacts.

54 ENVIRONMENTAL SCIENCES↗

Object-oriented structures supporting remote sensing databases

Object-oriented databases show promise for modeling the complex interrelationships pervasive in scientific domains. To examine the utility of this approach, we have developed an Intelligent Information Fusion System based on this technology, and applied it to the problem of managing an active repository of remotely-sensed satellite scenes. The design and implementation of the system is compared and contrasted with conventional relational database techniques, followed by a presentation of the underlying object-oriented data structures used to enable fast indexing into the data holdings.

Wichmann, Keith↗

GPS Sounding Rocket Developments

Sounding rockets are suborbital launch vehicles capable of carrying scientific payloads several hundred miles in altitude. These missions return a variety of scientific data including; chemical makeup and physical processes taking place In the atmosphere, natural radiation surrounding the Earth, data on the Sun, stars, galaxies and many other phenomena. In addition, sounding rockets provide a reasonably economical means of conducting engineering tests for instruments and devices used on satellites and other spacecraft prior to their use in more expensive activities. The NASA Sounding Rocket Program is managed by personnel from Goddard Space Flight Center Wallops Flight Facility (GSFC/WFF) in Virginia. Typically around thirty of these rockets are launched each year, either from established ranges at Wallops Island, Virginia, Poker Flat Research Range, Alaska; White Sands Missile Range, New Mexico or from Canada, Norway and Sweden. Many times launches are conducted from temporary launch ranges in remote parts of the world requi6ng considerable expense to transport and operate tracking radars. An inverse differential GPS system has been developed for Sounding Rocket. This paper addresses the NASA Wallops Island history of GPS Sounding Rocket experience since 1994 and the development of a high accurate and useful system.

Bull, Barton↗

The FIFE Project at Fermilab

The FabrIc for Frontier Experiments (FIFE) project is an initiative within the Fermilab Scientific Computing Division designed to steer the computing model for non-LHC Fermilab experiments across multiple physics areas. FIFE is a collaborative effort between experimenters and computing professionals to design and develop integrated computing models for experiments of varying size, needs, and infrastructure. The major focus of the FIFE project is the development, deployment, and integration of solutions for high throughput computing, data management, database access and collaboration management within an experiment. To accomplish this goal, FIFE has developed workflows that utilize Open Science Grid compute sites along with dedicated and commercial cloud resources. The FIFE project has made significant progress integrating into experiment computing operations several services including a common job submission service, software and reference data distribution through CVMFS repositories, flexible and robust data transfer clients, and access to opportunistic resources on the Open Science Grid. The progress with current experiments and plans for expansion with additional projects will be discussed. FIFE has taken the leading role in defining the computing model for Fermilab experiments, aided in the design of experiments beyond those hosted at Fermilab, and will continue to define the future direction of high throughput computing for future physics experiments worldwide.

Box, D.↗

Experiences with Testing the Largest Ground System NASA Has Ever Built

In the 1980s, the National Aeronautics and Space Administration (NASA) embarked upon a major Earth-focused program called Mission to Planet Earth. The Goddard Space Flight Center (GSFC) was selected to manage and develop a key component - the Earth Observing System (EOS). The EOS consisted of four major missions designed to monitor the Earth. The missions included 4 spacecraft. Terra (launched December 1999), Aqua (launched May 2002), ICESat (Ice, Cloud, and Land Elevation Satellite, launched January 2003), and Aura (scheduled for launch January 2004). The purpose of these missions was to provide support for NASA s long-term research effort for determining how human-induced and natural changes affect our global environment. The EOS Data and Information System (EOSDIS), a globally distributed, large-scale scientific system, was built to support EOS. Its primary function is to capture, collect, process, and distribute the most voluminous set of remotely sensed scientific data to date estimated to be 350 Gbytes per day. The EOSDIS is composed of a diverse set of elements with functional capabilities that require the implementation of a complex set of computers, high-speed networks, mission-unique equipment, and associated Information Technology (IT) software along with mission-specific software. All missions are constrained by schedule, budget, and staffing resources, and rigorous testing has been shown to be critical to the success of each mission. This paper addresses the challenges associated with the planning, test definition. resource scheduling, execution, and discrepancy reporting involved in the mission readiness testing of a ground system on the scale of EOSDIS. The size and complexity of the mission systems supporting the Aqua flight operations, for example, combined with the limited resources available, prompted the project to challenge the prevailing testing culture. The resulting success of the Aqua Mission Readiness Testing (MRT) program was due in no small measure to re-structuring the traditional programmatic and technical approach to a more efficient and robust program. Programmatically, it meant gaining the endorsement, commitment, and cooperation of the numerous subsystem element managers and other stakeholder organizations. Technically, it required an MRT program that was agile, could rapidly adapt to requirements changes, and was flexible in its overall approach. Furthermore, this paper addresses the following questions: 1. What are the key ingredients (e.g., test tools, organization) needed to conduct a successful MRT program? 2. What distinguishes EOS MRT from the traditional system testing approach? 3. Where should the focus of testing be since it is infeasible to test every element or subsystem? 4. How can MRT be applied effectively to other systems or missions? To provide answers to these questions, this paper relies heavily on real-life, hands-on experiences ("lessons learned") gained during mission readiness testing of the Terra ground system and, most recently, the Aqua and ICESat missions. Moreover, this paper explores how lessons learned were turned into lessons applied for the upcoming Aura mission. Although derived from the EOS missions, MRT techniques and strategies can be applied to enhance the testing of other missions.

Lehtonen, Ken↗