Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “massive data sets”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS2012

The Role in the Virtual Astronomical Observatory in the Era of Massive Data Sets

The Virtual Observatory (VO) is realizing global electronic integration of astronomy data. One of the long-term goals of the U.S. VO project, the Virtual Astronomical Observatory (VAO), is development of services and protocols that respond to the growing size and complexity of astronomy data sets. This paper describes how VAO staff are active in such development efforts, especially in innovative strategies and techniques that recognize the limited operating budgets likely available to astronomers even as demand increases. The project has a program of professional outreach whereby new services and protocols are evaluated.

data-driven science↗
NASA NTRS2001

A strategy for compression and analysis of massive geophysical data sets

This paper describes a method for summaraizing data in a way that approximately preserves high-resolution data structure while reducing data volume and maintaining global integrity of very large, remote sensing data sets. The method is under development for one of Terra's instruments, the Multi-angle Imaging SpectroRadiometer (MISR).

geophysical data sets compression↗
NASA NTRS2018

Remote Sensing, Uncertainty Quantification, and a Theory of Data Systems; Workshop Report

The purpose of the workshop was to invite statisticians, applied mathematicians, computer scientists, data system architects, experts in remote sensing technology, and Climate and Earth System scientists to review, discuss, and plan research on issues related to large-scale, efficient analysis of distributed data using spatial statistical methods. Our motivation in organizing this event was to catalyze interchange among experts on the fast-emerging problem of analysis of distributed data. As part of SAMSI's 2017-2018 Program on Mathematical and Statistical Methods for Climate and the Earth System, a Working Group on Remote Sensing was established to address statistical and mathematical research problems in the analysis of remote sensing data. The Working Group has five subgroups: 1) Spatial Retrieval Methodology (the so-called \Spatial-X" subgroup); 2) Spatial Analysis for Hyperspectral Data (the so-called \Spatial-Y" subgroup); 3) Emulators for Complex Forward Models; 4) Optimization for Remote Sensing Retrievals; and 5) Theory of Data Systems (ToDS). The ToDS subgroup spent the first half of this academic year formulating a framework in which to consider the joint problem of a) optimizing statistical methods for environments where data are distributed and too large to move to a central location, and b) the design of data system infrastructures within which to implement those statistical methods. To x ideas, the Workshop focused on spatial statistical methods. To date there are many new spatial statistical methods designed with massive data sets in mind, in the literature. However, very few have been implemented for remote sensing data, and none have been implemented in operational settings like those used by NASA and NOAA. A major impediment to their use in these cases is that the data are not only massive, but are stored in different physical locations. These data must be brought together in some way in order to estimate spatial covariance functions, but moving data to a central location for analysis is tedious at best and impossible at worst. Some remote data reduction is almost certainly necessary, but how much? What are the consequences for inference? The fundamental issue underlying these questions is how to navigate the trade-space between costs and uncertainty in the estimates or inferences that are ultimately produced.

Braverman, Amy↗
NASA NTRS2013

Mapping Impervious Surfaces Globally at 30m Resolution Using Global Land Survey Data

Impervious surfaces, mainly artificial structures and roads, cover less than 1% of the world's land surface (1.3% over USA). Regardless of the relatively small coverage, impervious surfaces have a significant impact on the environment. They are the main source of the urban heat island effect, and affect not only the energy balance, but also hydrology and carbon cycling, and both land and aquatic ecosystem services. In the last several decades, the pace of converting natural land surface to impervious surfaces has increased. Quantitatively monitoring the growth of impervious surface expansion and associated urbanization has become a priority topic across both the physical and social sciences. The recent availability of consistent, global scale data sets at 30m resolution such as the Global Land Survey from the Landsat satellites provides an unprecedented opportunity to map global impervious cover and urbanization at this resolution for the first time, with unprecedented detail and accuracy. Moreover, the spatial resolution of Landsat is absolutely essential to accurately resolve urban targets such a buildings, roads and parking lots. With long term GLS data now available for the 1975, 1990, 2000, 2005 and 2010 time periods, the land cover/use changes due to urbanization can now be quantified at this spatial scale as well. In the Global Land Survey - Imperviousness Mapping Project (GLS-IMP), we are producing the first global 30 m spatial resolution impervious cover data set. We have processed the GLS 2010 data set to surface reflectance (8500+ TM and ETM+ scenes) and are using a supervised classification method using a regression tree to produce continental scale impervious cover data sets. A very large set of accurate training samples is the key to the supervised classifications and is being derived through the interpretation of high spatial resolution (approx. 2 m or less) commercial satellite data (Quickbird and Worldview2) available to us through the unclassified archive of the National Geospatial Intelligence Agency (NGA). For each continental area several million training pixels are derived by analysts using image segmentation algorithms and tools and then aggregated to the 30m resolution of Landsat. Here we will discuss the production/testing of this massive data set for Europe, North and South America and Africa, including assessments of the 2010 surface reflectance data. This type of analysis is only possible because of the availability of long term 30m data sets from GLS and shows much promise for integration of Landsat 8 data in the future.

global land survey↗
NASA NTRS1998

Method and system for data clustering for very large databases

Multi-dimensional data contained in very large databases is efficiently and accurately clustered to determine patterns therein and extract useful information from such patterns. Conventional computer processors may be used which have limited memory capacity and conventional operating speed, allowing massive data sets to be processed in a reasonable time and with reasonable computer resources. The clustering process is organized using a clustering feature tree structure wherein each clustering feature comprises the number of data points in the cluster, the linear sum of the data points in the cluster, and the square sum of the data points in the cluster. A dense region of data points is treated collectively as a single cluster, and points in sparsely occupied regions can be treated as outliers and removed from the clustering feature tree. The clustering can be carried out continuously with new data points being received and processed, and with the clustering feature tree being restructured as necessary to accommodate the information from the newly received data points.

Zhang, Tian↗
NASA NTRS2005

Bayesian Analysis of the Power Spectrum of the Cosmic Microwave Background

There is a wealth of cosmological information encoded in the spatial power spectrum of temperature anisotropies of the cosmic microwave background. The sky, when viewed in the microwave, is very uniform, with a nearly perfect blackbody spectrum at 2.7 degrees. Very small amplitude brightness fluctuations (to one part in a million!!) trace small density perturbations in the early universe (roughly 300,000 years after the Big Bang), which later grow through gravitational instability to the large-scale structure seen in redshift surveys... In this talk, I will discuss a Bayesian formulation of this problem; discuss a Gibbs sampling approach to numerically sampling from the Bayesian posterior, and the application of this approach to the first-year data from the Wilkinson Microwave Anisotropy Probe. I will also comment on recent algorithmic developments for this approach to be tractable for the even more massive data set to be returned from the Planck satellite.

Bayesian inference↗
NASA NTRS2019

Exploring Beyond Earth's Atmosphere with Human-Machine Teams

NASA's highly successful Kepler Mission has revolutionized our understanding of the Galaxy. We now know that planets, even Earth-size planets in the habitable zone, are common. With the end of the Kepler Mission we now look to the future with the Transiting Exoplanet Survey Satellite (TESS) which will discover thousands of exoplanets in orbit around the brightest stars in the sky. In a two-year survey, TESS will perform an all-sky search of more than 200,000 stars for temporary drops in brightness caused by planetary transits. With Kepler and TESS, humanity is finally at the verge of studying the masses, sizes, densities, orbits, and atmospheres of a large cohort of small planets, including a sample of rocky worlds in the habitable zones of their host stars which may prove to host life. The massive data sets generated by Kepler and TESS must be meticulously combed for the weakest planetary signals every month. While a daunting and error-prone task for humans, this is an exciting opportunity for the breakthroughs recently seen in machine learning. Specifically, traditional methods for identifying planet transits require extensive data processing pipelines followed by extensive human vetting. This manual process risks loss of information due to the data processing and to inconsistency and biases due to individual human vetters. The latest advancements in machine learning will allow an objective classifier to minimize the losses of information and greatly lessen the burden on the human vetters, in addition to providing assessment of quality and score to each planet candidate, freeing the humans to concentrate on border cases and other more interesting investigations.

Smith, Jeffrey C.↗
NASA NTRS2004

Visual data mining for quantized spatial data

In previous papers we've shown how a well known data compression algorithm called Entropy-constrained Vector Quantization ( can be modified to reduce the size and complexity of very large, satellite data sets. In this paper, we descuss how to visualize and understand the content of such reduced data sets.

cluster analysis↗
NASA NTRS2016

Decision Manifold Approximation for Physics-Based Simulations

With the recent surge of success in big-data driven deep learning problems, many of these frameworks focus on the notion of architecture design and utilizing massive databases. However, in some scenarios massive sets of data may be difficult, and in some cases infeasible, to acquire. In this paper we discuss a trajectory-based framework that quickly learns the underlying decision manifold of binary simulation classifications while judiciously selecting exploratory target states to minimize the number of required simulations. Furthermore, we draw particular attention to the simulation prediction application idealized to the case where failures in simulations can be predicted and avoided, providing machine intelligence to novice analysts. We demonstrate this framework in various forms of simulations and discuss its efficacy.

Wong, Jay Ming↗
NASA NTRS2007

Understanding Local Structure Globally in Earth Science Remote Sensing Data Sets

Empirical probability distributions derived from the data are the signatures of physical processes generating the data. Distributions defined on different space-time windows can be compared and differences or changes can be attributed to physical processes. This presentation discusses on ways to reduce remote sensing data in a way that preserves information, focusing on the rate-distortion theory and using the entropy-constrained vector quantization algorithm.

massive data sets↗
NASA NTRS1991

GLOBEC (Global Ocean Ecosystems Dynamics: Northwest Atlantic program

The specific objective of the meeting was to plan an experiment in the Northwestern Atlantic to study the marine ecosystem and its role, together with that of climate and physical dynamics, in determining fisheries recruitment. The underlying focus of the GLOBEC initiative is to understand the marine ecosystem as it related to marine living resources and to understand how fluctuation in these resources are driven by climate change and exploitation. In this sense the goal is a solid scientific program to provide basic information concerning major fisheries stocks and the environment that sustains them. The plan is to attempt to reach this understanding through a multidisciplinary program that brings to bear new techniques as disparate as numerical fluid dynamic models of ocean circulation, molecular biology and modern acoustic imaging. The effort will also make use of the massive historical data sets on fisheries and the state of the climate in a coordinated manner.

Source record↗
NASA NTRS2006

Using Image Tour to Explore Multiangle, Multispectral Satellite Image

This viewgraph presentation reviews the use of Image Tour to explore the multiangle, multispectral satellite imagery. Remote sensing data are spatial arrays of p-dimensional vectors where each component corresponds to one of p variables. Applying the same R(exp p) to R(exp d) projection to all pixels creates new images, which may be easier to analyze than the original because d < p. Image grand tour (IGT) steps through the space of projections, and d=3 outputs a sequence of RGB images, one for each step. In this talk, we apply IGT to multiangle, multispectral data from NASA's MISR instrument. MISR views each pixel in four spectral bands at nine view angles. Multiple views detect photon scattering in different directions and are indicative of physical properties of the scene. IGT allows us to explore MISR's data structure while maintaining spatial context; a key requirement for physical interpretation. We report results highlighting the uniqueness of multiangle data and how IGT can exploit it.

visualization↗