Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Big data applications”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

A Spatiotemporal Indexing Approach for Efficient Processing of Big Array-Based Climate Data with MapReduce

Climate observations and model simulations are producing vast amounts of array-based spatiotemporal data. Efficient processing of these data is essential for assessing global challenges such as climate change, natural disasters, and diseases. This is challenging not only because of the large data volume, but also because of the intrinsic high-dimensional nature of geoscience data. To tackle this challenge, we propose a spatiotemporal indexing approach to efficiently manage and process big climate data with MapReduce in a highly scalable environment. Using this approach, big climate data are directly stored in a Hadoop Distributed File System in its original, native file format. A spatiotemporal index is built to bridge the logical array-based data model and the physical data layout, which enables fast data retrieval when performing spatiotemporal queries. Based on the index, a data-partitioning algorithm is applied to enable MapReduce to achieve high data locality, as well as balancing the workload. The proposed indexing approach is evaluated using the National Aeronautics and Space Administration (NASA) Modern-Era Retrospective Analysis for Research and Applications (MERRA) climate reanalysis dataset. The experimental results show that the index can significantly accelerate querying and processing (10 speedup compared to the baseline test using the same computing cluster), while keeping the index-to-data ratio small (0.0328). The applicability of the indexing approach is demonstrated by a climate anomaly detection deployed on a NASA Hadoop cluster. This approach is also able to support efficient processing of general array-based spatiotemporal data in various geoscience domains without special configuration on a Hadoop cluster.

big data↗

Applications of Anomaly Detection and Precursor Identification in Airspace Operations

As we continue to advance the U.S. National Airspace into the next generation of air traffic, we face challenges in both increase in complexity, as well as, a significant growth in traffic volume. Addressing these challenges, while maintaining the same level of safety is an important application of data mining. Because of these significant shifts in airspace design and usage there is a need to identify current and emergent safety risks along with their potential precursors. In recent years NASA has made advancements in developing scalable methods to address this effort in the Big Data paradigm. Multiple kernel anomaly detection approaches have been employed on both surveillance radar data and flight operational quality assurance data to identify operationally significant safety risks. Additionally, events have been explored with a recently developed precursor identification tool to discover states that reveal an increased probability of a safety event. These tools can be used to discover emerging safety risks that may not be currently monitored, which allows for mitigation tactics to be employed and ultimately make the overall airspace safer. This talk will discuss an overview of these methods and a discussion of the findings.

anomaly detection↗

Western energy related overhead monitoring project. Phase 2: Summary

Assistance by NASA to EPA in the establishment and maintenance of a fully operational energy-related monitoring system included: (1) regional analysis applications based on LANDSAT and auxiliary data; (2) development of techniques for using aircraft MSS data to rapidly monitor site specific surface coal mine activities; and (3) registration of aircraft MSS data to a map base. The coal strip mines used in the site specific task were in Campbell County, Wyoming; Big Horn County, Montana; and the Navajo mine in San Juan County, New Mexico. The procedures and software used to accomplish these tasks are described.

Anderson, J. E.↗

Bayesian Analysis of the Power Spectrum of the Cosmic Microwave Background

There is a wealth of cosmological information encoded in the spatial power spectrum of temperature anisotropies of the cosmic microwave background. The sky, when viewed in the microwave, is very uniform, with a nearly perfect blackbody spectrum at 2.7 degrees. Very small amplitude brightness fluctuations (to one part in a million!!) trace small density perturbations in the early universe (roughly 300,000 years after the Big Bang), which later grow through gravitational instability to the large-scale structure seen in redshift surveys... In this talk, I will discuss a Bayesian formulation of this problem; discuss a Gibbs sampling approach to numerically sampling from the Bayesian posterior, and the application of this approach to the first-year data from the Wilkinson Microwave Anisotropy Probe. I will also comment on recent algorithmic developments for this approach to be tractable for the even more massive data set to be returned from the Planck satellite.

Bayesian inference↗

Using the SPoRT POES/GOES Hybrid Product in OCONUS Forecasting

The SPoRT (Short-term Prediction and Research Transition) Program at the NASA/Marshall Space Flight Center has been providing unique NASA and NOAA data and techniques to partner Weather Forecast Offices (WFOs) for ten years. Data are provided in the Decision Support System used by WFO forecasters: AWIPS. For the last couple of years, SPoRT has been producing the POES/GOES Hybrid. This suite of products combines the strength ofl5- minute animations of GOES imagery - providing temporal continuity, with the higher resolution, relatively random availability, of polar orbiting (POES) imagery data. The product was first introduced with only MODIS data from NASA's Terra and Aqua satellites, but recently the VIIRS instrument onboard the Suomi-NPP satellite was added, providing better high-resolution coverage. These products represent SPoRT's efforts to prepare for higher resolution, higher frequency GOES-R imagery - as well as helping to move VIIRS (JPSS) data into the mainstream of weather forecasting. SPoRT generates 5 products for this dataset: Visible, Longwave Infrared (11 micrometers), Shortwave IR (3.7 micrometers), Water Vapor (6.7 micrometers), and Fog (Difference of 11 micrometer and 3.7 micrometer channels). The Water Vapor hybrid product has a Red-Blue-Green image from MODIS inlaid, since it provides even more qualitative information than water vapor alone. Animated examples of the products will be shown in this presentation. While the resolution at nadir of GOES imagery is nominally Han (4km for IR channels), the inlaid polar orbiter imagery has a resolution of 250m (lkm for IR channels). This has tremendous application in the continental US. However, in high latitudes, since the usefulness of GOES degrades poleward rapidly, the contrast of GOES and POES data is stark. The consistent temporal nature of GOES, even though at a reduced resolution at high latitudes, provides basic situational awareness, but the introduction of polar data is very helpful in seeing the big picture with clarity - even if only briefly. This presentation will offer real situations where these products helped forecasters make better informed decisions quickly. Plans to augment the product further with the addition of data from several A VHRR instruments will be described.

Smith, Matt↗

Machine Intelligence for Radiation Science: Summary of the Radiation Research Society 67th Annual Meeting Symposium

The era of high-throughput techniques created big data in the medical field and research disciplines. Machine intelligence (MI) approaches can overcome critical limitations on how those large-scale data sets are processed, analyzed, and interpreted. The 67 th Annual Meeting of the Radiation Research Society featured a symposium on MI approaches to highlight recent advancements in the radiation sciences and their clinical applications. This article summarizes three of those presentations regarding recent developments for metadata processing and ontological formalization, data mining for radiation outcomes in pediatric oncology, and imaging in lung cancer.

radiation↗

Near 40 Years MERRA-2 Data at NASA GES DISC -Opportunity and Challenge to Support Extremes Study

To the end of 2019, 40 years NASA climate reanalysis data sets from the Modern Era Retrospective-analysis for Research and Applications, Version 2 (MERRA-2) will be available at NASA Goddard Earth Sciences Data and Information Services Center (GES DISC). MERRA-2 consists atmosphere, land, and ocean data, which may be used for the studies ranging from the short scale weather events to the large scale decadal vulnerabilities. The hourly products, such as precipitation, soil moisture, temperature, and aerosols etc., have been used widely to study extreme events.In supporting users from broad communities, GES DISC have developed various data access services, including subsetter for downloading only data of interest with preferred format; OPeNDAP - for machine-to-machine data access; and Giovanni- for online visualization and analysis, etc. A big challenge for extreme study is to downloading and processing long-term hourly or daily data. The data downloading performance is not very satisfied by many users with current services and the native archived data structure. Late June 2019, many people in Europe had experienced extreme heat waves. The temperatures in several countries exceeded 40°C (104°F). For example, MERRA-2 shows that the near surface daily maximum temperature of June 28 2019 over Marseille, a city in southern France, reached 41.1 °C (106°F), which is the record breaking temperature in the last 40 years. GES DISC is working together with domain science experts to improve the performance of long time series access, making analysis ready data sets in supporting application researches, such as extreme study. In this presentation, using Europe heat wave as an example, we will show prototype of the in developing service for finding extremes from near 40 years MERRA-2 data at a given location. MERRA-2 data can be accessed from NASA GES DISC(https://disc.gsfc.nasa.gov/ ) by search keyword "MERRA-2".

Shen, Suhung↗

Open Source Application of Fusing Aerosol Products from GEO and LEO Satellites

Retrieving aerosol optical depths (AODs) from sun-synchronous polar orbiting (aka low earth orbit, LEO) satellites, such as MODISs, and VIIRSs, OMI, TROPOMI, etc, has become well-established as a tool for extracting information on particulate matter (PM) and related processes in the atmosphere. However, with recently launched geostationary satellites (GEO), such as GOES-16/17/18, and Himawari-8/9, and Meteosat Third Generation (MTG) they provide a much higher temporal resolution (order of 10 minutes), typically an image once or more per hour during daylight compared to LEO once per day. By combining these observations, we may be able to characterize the diurnal cycle of global AOD at the local, regional and global scale. While the science community is still exploring the new data from GEO observations, we have been thinking about how to properly combine/merge/fuse those data considering differences in their spatial and temporal resolutions. However, this poses a “Big Data” challenge. The big data challenge is not just about data storage, but also about data discoverability, and accessibility, and even more, about data migration/mirroring in the cloud-computing environment. This paper is merely showing some of the efforts and approaches we have attempted in fusing six satellites’ Level 2 aerosol data (three are from GEO (GOES-16/17 and Himawari-8), and the other three are from LEO (TERRA/MODIS, AQUA/MODIS, SNPP-VIIRS) from Dark Target (DT) aerosol retrieval algorithm. Having the on-demand capability of fusing remote sensing products onto the desired temporal and spatial domain enables researchers and application practitioners to better manipulate and work with satellite and sensor data. It is our hopeWe hope that by making such an open-source package, and the accompanying functionality, the scientific community will be granted easier access to aerosol data processing resources. The MEaSUREs Program (Making Earth System Data Records for Use in Research Environments) expands our understanding of the Earth's current system through atmospheric and surface measurements. In an effort to aid the scientific research component and improve open source methods, this project developed Python code for fusing six satellite Level 2 aerosol data (three are from geostationary satellites (GEO), and the other three are from low earth orbital satellites (LEO)) from Dark Target Aerosol Retrieval Algorithm.

Jennifer Wei↗

Air Traffic Management TestBed Simulation Architect: User's Guide

The Air Traffic Management (ATM) TestBed is a Platform as a Service that is being developed by the National Aeronautics and Space Administration (NASA) to help design, configure, integrate, run, and monitor air traffic simulations. The platform provides cloud services including back-end big-data analytics tools, on-demand computing resource management, data storage, and communication middleware. The ATM TestBed reduces the time to test concepts and technologies, supports interactions among various concepts such as human-in-the-loop and automation-in-the-loop simulations, and enables collaborative simulations by sharing technologies and tools in the ATM community. The Simulation Architect application provides a graphical user interface tool for designing traffic scenarios and simulations using blocks representing components and links representing message channels linking them. This guide describes a high-level user interface design of Simulation Architect and provides information for a new user to compose traffic scenarios and simulations.

Software User Guide↗

Air Traffic Management TestBed Traffic Viewer: Developer's Guide

The Air Traffic Management (ATM) TestBed is a Platform as a Service that is being developed by the National Aeronautics and Space Administration (NASA) to help design, configure, integrate, run, and monitor air traffic simulations. The platform is designed to provide cloud services including back-end, big-data analytics tools, on-demand computing resource management, data storage, and communication middleware. The ATM TestBed reduces the time to test concepts and technologies, supports interactions among various methods such as human-in-the-loop and automation-in-the-loop simulations, and enables collaborative simulations by sharing technologies and tools in the ATM community. The Traffic Viewer application provides a graphical user interface tool for visualizing real and simulated air traffic as well as airspace definition in two-dimensional space. This guide describes a high-level design and implementation of Traffic Viewer and provides information for a new developer or a user to add new capabilities by following the software design and leveraging existing capabilities.

Lai, Chok Fung↗

Machine Learning-Based Predictive Analytics for Aircraft Engine Conceptual Design

Big data and artificial intelligence/machine learning are transforming the global business environment. Data is now the most valuable asset for enterprises in every industry. Companies are using data-driven insights for competitive advantage. With that, the adoption of machine learning-based data analytics is rapidly taking hold across various industries, producing autonomous systems that support human decision-making. This work explored the application of machine learning to aircraft engine conceptual design. Supervised machine-learning algorithms for regression and classification were employed to study patterns in an existing, open-source database of production and research turbofan engines, and resulting in predictive analytics for use in predicting performance of new turbofan designs. Specifically, the author developed machine learning-based analytics to predict cruise thrust specific fuel consumption (TSFC) and core sizes of high-efficiency turbofan engines, using engine design parameters as the input. The predictive analytics were trained and deployed in Keras, an open-source neural networks application program interface (API) written in Python, with Google’s TensorFlow (an open source library for numerical computation) serving as the backend engine. The promising results of the predictive analytics show that machine-learning techniques merit further exploration for application in aircraft engine conceptual design.

deep-learning↗

Application of remote sensing to selected problems within the state of California

Specific case studies undertaken to demonstrate the usefulness of remote sensing technology to resource managers in California are highlighted. Applications discussed include the mapping and quantization of wildland fire fuels in Mendocino and Shasta Counties as well as in the Central Valley; the development of a digital spectral/terrain data set for Colusa County; the Forsythe Planning Experiment to maximize the usefulness of inputs from LANDSAT and geographic information systems to county planning in Mendocino County; the development of a digital data bank for Big Basin State Park in Santa Cruz County; the detection of salinity related cotton canopy reflectance differences in the Central Valley; and the surveying of avocado acreage and that of other fruits and nut crops in Southern California. Special studies include the interpretability of high altitude, large format photography of forested areas for coordinated resource planning using U-2 photographs of the NASA Bucks Lake Forestry test site in the Plumas National Forest in the Sierra Nevada Mountains.

Colwell, R. N.↗

SERVIR: Leveraging the Expertise of a Space Agency and a Development Agency to Increase Impact of Earth Observation in the Developing World

SERVIR is a joint initiative of the National Aeronautics and Space Administration (NASA) and the U.S. Agency for International Development (USAID), in collaboration with leading technical organizations around the world-- called SERVIR hubs--that serve and empower developing countries to use satellite data addressing critical challenges in food security and agriculture; water and water-related disasters; land cover, land use and ecosystems; and weather and climate. Over the past fourteen years, the program has worked with stakeholders in 50 countries across the world, partnered with 390 institutions, and generated and shared over 70 products from 27 satellites and sensors. In that process, around 7,400 specialists have been trained in the application of Earth observation data and technology. In its lifetime, SERVIR has been agile and innovative in shifting from what was essentially an incubator for testing and deploying Earth observation science and technology to making co-development the hallmark of its work, exemplified by both South-South and North-South scientific collaborations. SERVIR’s approach has embodied the concept that to solve really big problems, big, creative solutions are needed. SERVIR represents the world working together to address environmental challenges using spaced-based and geospatial technologies. Aligning with the very meaning of SERVIR, i.e. “to serve,” the program continues to be demand-driven in developing and deploying services (versus one-off products) which address development challenges using geospatial tools and Earth observation science. In 2016, as part of SERVIR’s evolution, USAID and NASA released the ‘SERVIR Service Planning Toolkit,’ a guidance document which provides a framework for how geospatial services can be used to tackle development challenges in a sustained manner. Since then, the Service Planning Toolkit’s systematic approach has begun to catch on in other Earth observation efforts. To improve access and use, SERVIR launched a Service Catalogue in February 2019, a searchable collection of demand-driven geospatial services that use Earth observations to support decision making. SERVIR implementing hub partners –include SERVIR-West Africa at the Agrometeorology, Hydrology and Meteorology (AGRHYMET) Regional Center, in Niamey, Niger; SERVIR-Eastern & Southern Africa at the Regional Centre for Mapping of Resources for Development in Nairobi, Kenya; SERVIR-Hindu Kush Himalaya at the International Centre for Integrated Mountain Development in Kathmandu, Nepal; SERVIR-Mekong at the Asian Disaster Preparedness Center in Bangkok, Thailand; and SERVIR’ -Amazonia, at the International Center for Tropical Agriculture (CIAT) in Cali, Colombia.

Searby, Nancy D.↗

LEO Download Capacity Analysis for a Network of Adaptive Array Ground Stations

To lower costs and reduce latency, a network of adaptive array ground stations, distributed across the United States, is considered for the downlink of a polar-orbiting low earth orbiting (LEO) satellite. Assuming the X-band 105 Mbps transmitter of NASA s Earth Observing 1 (EO-1) satellite with a simple line-of-sight propagation model, the average daily download capacity in bits for a network of adaptive array ground stations is compared to that of a single 11 m dish in Poker Flats, Alaska. Each adaptive array ground station is assumed to have multiple steerable antennas, either mechanically steered dishes or phased arrays that are mechanically steered in azimuth and electronically steered in elevation. Phased array technologies that are being developed for this application are the space-fed lens (SFL) and the reflectarray. Optimization of the different boresight directions of the phased arrays within a ground station is shown to significantly increase capacity; for example, this optimization quadruples the capacity for a ground station with eight SFLs. Several networks comprising only two to three ground stations are shown to meet or exceed the capacity of the big dish, Cutting the data rate by half, which saves modem costs and increases the coverage area of each ground station, is shown to increase the average daily capacity of the network for some configurations.

Ingram, Mary Ann↗

A Study on the Potential Applications of Satellite Data in Air Quality Monitoring and Forecasting

In this study we explore the potential applications of MODIS (Moderate Resolution Imaging Spectroradiometer) -like satellite sensors in air quality research for some Asian regions. The MODIS aerosol optical thickness (AOT), NCEP global reanalysis meteorological data, and daily surface PM(sub 10) concentrations over China and Thailand from 2001 to 2009 were analyzed using simple and multiple regression models. The AOT-PM(sub 10) correlation demonstrates substantial seasonal and regional difference, likely reflecting variations in aerosol composition and atmospheric conditions, Meteorological factors, particularly relative humidity, were found to influence the AOT-PM(sub 10) relationship. Their inclusion in regression models leads to more accurate assessment of PM(sub 10) from space borne observations. We further introduced a simple method for employing the satellite data to empirically forecast surface particulate pollution, In general, AOT from the previous day (day 0) is used as a predicator variable, along with the forecasted meteorology for the following day (day 1), to predict the PM(sub 10) level for day 1. The contribution of regional transport is represented by backward trajectories combined with AOT. This method was evaluated through PM(sub 10) hindcasts for 2008-2009, using ohservations from 2005 to 2007 as a training data set to obtain model coefficients. For five big Chinese cities, over 50% of the hindcasts have percentage error less than or equal to 30%. Similar performance was achieved for cities in northern Thailand. The MODIS AOT data are responsible for at least part of the demonstrated forecasting skill. This method can be easily adapted for other regions, but is probably most useful for those having sparse ground monitoring networks or no access to sophisticated deterministic models. We also highlight several existing issues, including some inherent to a regression-based approach as exemplified by a case study for Beijing, Further studies will be necessa1Y before satellite data can see more extensive applications in the operational air quality monitoring and forecasting.

Li, Can↗

Ocean Wave Studies with Applications to Ocean Modeling and Improvement of Satellite Altimeter Measurements

Combining analysis of satellite data (altimeter, scatterometer, high-resolution visible and infrared images, etc.) with mathematical modeling of non-linear wave processes, we investigate various ocean wave fields (on scales from capillary to planetary), their role in ocean dynamics and turbulent transport (of heat and biogeochemical quantities), and their effects on satellite altimeter measuring accuracy. In 1998 my attention was focused on long internal gravity waves (10 to 1000 km), known also as baroclinic inertia-gravity (BIG) waves. We found these waves to be a major factor of altimeter measurements "noise," resulting in a greater uncertainty [up to 10 cm in terms of sea surface height (SSH) amplitude] in the measured SSH signal than that caused by the sea state bias variations (up to 5 cm or so). This effect still remains largely overlooked by the satellite altimeter community. Our studies of BIG waves address not only their influence on altimeter measurements but also their role in global ocean dynamics and in transport and turbulent diffusion of biogeochemical quantities. In particular, in collaboration with Prof Peter Weichman, Caltech, we developed a theory of turbulent diffusion caused by wave motions of most general nature. Applied to the problem of horizontal turbulent diffusion in the ocean, the theory yielded the effective diffusion coefficient as a function of BIG wave parameters obtainable from satellite altimeter data. This effort, begun in 1997, has been successfully completed in 1998. We also developed a theory that relates spatial fluctuations of scalar fields (such as sea surface temperature, chlorophyll concentration, drifting ice concentration, etc.) to statistical characteristics of BIG waves obtainable from altimeter measurements. A manuscript is in the final stages of preparation. In order to verify the theoretical predictions and apply them to observations, we are now analyzing Sea-viewing Wide Field of view Sensor (SeaWiFS) and Field of view Sensor (SeaWiFS) and Advanced Very High-Resolution Radiometer (AVHRR) data on sea surface temperature (SST) and chlorophyll concentration jointly with TOPEX/POSEIDON data on SSH variations.

Glazman, Roman E.↗

NASA GES DISC's Customized Services for Climatology and Meteorology

At the NASA Goddard Earth Sciences (GES) Data and Information Service Center (DISC), we have archived and distributed more than 2,400 Earth science data products, from different missions or projects containing more than 100 M data files/granules with a total volume size nearly 2 PB that broadly serve user needs in science areas such as Atmospheric Composition, Water & Energy Cycles and Climate Variability. To date, GES DISC has developed many pertinent services to facilitate the usage of data products by our research communities, represented by approximately 24,000 registered users. We are facing the big data with increasingly archival volume and data types, moreover, we also encounter increasing users' demands and the demands are more diversified. It is still a challenge for us to better understand exactly what our users' needs are, even after developing more than 70 services, including well-known online tools such as Giovanni and MERRA subsetter. In this presentation, we will try to address how we can accommodate the users' needs from two applicational user communities, Air Quality and Wind Energy, from data or service discovery to guide them properly utilize the data and services to fit their needs.

customizable services for climate and meteorology↗

SWIPE: Spectral Water Inversion Processor and Emulator

Degradation of Earth’s inland water resources due to anthropogenic perturbations and climate anomalies at both local and global scales continues to place human health at substantial risk. There is now a growing necessity to develop pragmatic approaches that allow timely and effective extrapolation of local processes, to spatially resolved global products, and to promote operational and sustainable resource policy management. This presentation will be discussing the progress made developing SWIPE: Spectral Water Inversion Processor and Emulator. SWIPE is a platform for advanced modeling of coastal and inland aquatic habitats. The goal is create a comprehensive and cohesive system to leverage recent advancements in computation and machine learning to develop a synthetic training ground for sensitivity studies and algorithm development. The four principal facets of SWIPE include: 1. Advanced two-layer coated sphere bio-optical modeling and GPU radiative transfer modeling, 2. Big Data involving massive synthetic spectral libraries of optical properties of various global aquatic particles, surface reflectance, and top-of-atmosphere reflectance, all at hyperspectral resolution leveraging high-end computing systems at NASA Ames Research Center, 3. Deep Learning for algorithm development for water quality inversion of concentrations of common biogeophysical variables as well as optics, full uncertainty characterization by water type, and forward emulation, and lastly, 4. Image Processing for application of developed retrieval algorithms for both hyperspectral and multispectral sensors with experimental corrections for global adjacency, noise, sunglint, and benthic reflectance. This presentation will demonstrate the Equivalent Algal Populations (EAP) two-layer coated sphere scattering model which has been used develop spectral libraries of hyperspectral inherent optical properties of roughly 80 species of phytoplankton, covering 15 different classes and nine taxonomic functional types. The EAP model was also used to derive spectral properties of 10 different non-algal particle functional types. Examples of how the SMART-G (Speed-up Monte-carlo Advanced Radiative Transfer using GPU) radiative transfer code is used to model optically complex aquatic signals will be presented and discussed in the context of creating a massive synthetic database which can leverage the full power of next generation machine learning techniques and high end computing for water quality inversion. We will discuss our active investigation in things like appropriate model architectures, dimensionality reduction techniques such as PCA and autoencoders, uncertainty quantification and abstaining, and which variables actually benefit most from hyperspectral information versus multispectral resolution. We are also curious about questions relating to cost/benefit analysis in terms of computation resources, neural network complexity, and data volumes. Answers to these questions will hopefully elaborate on cost efficiency for potential future sensor design considerations.

SWIPE↗