Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Sparse Data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9

Storm Chasing from Space: Detecting Severe Weather Phenomena from Satellite Platforms

Severe weather is an awe-inspiring phenomenon that affects the entire globe. Lightning, hail, damaging wind and tornadoes pose threats to society and challenges to the scientific community. Severe weather is annually responsible for tens of billions of dollars in insured losses to property, infrastructure, and agriculture. Satellite platforms offer a globally uniform approach to observing weather phenomena in remote or data-sparse regions and over the oceans. Severe convection exhibits distinct signatures in spaceborne datasets that we use to analyze severe storms. We leverage these signatures create climatologies, improve prediction, and provide a method of detection around the globe where traditional ground-based data (such as ground-based radar or human-spotter reports) are inconsistent or unavailable. Satellites in low-earth, sun-synchronous, and geostationary orbit provide a consistent, global view of severe weather from which we can examine the current global distribution, frequency, and severity of severe storms and establish a baseline to assess their future trend in a changing Earth system.

Sarah D Bang↗

Parallel Algorithms for Computing the Tensor-Train Decomposition

The tensor-train (TT) decomposition expresses a tensor in a data-sparse format used in molecular simulations, high-order correlation functions, and optimization. In this paper, we propose four parallelizable algorithms that compute the TT format from various tensor inputs: (1) Parallel-TTSVD for traditional format, (2) PSTT and its variants for streaming data, (3) Tucker2TT for Tucker format, and (4) TT-fADI for solutions of Sylvester tensor equations. We provide theoretical guarantees of accuracy, parallelization methods, scaling analysis, and numerical results. For example, for a d-dimension tensor in $\mathbb{R}$ $n\times∙∙∙$$\times$$n$ a two-sided sketching algorithm PSTT2 is shown to have a memory complexity of $O(n^{[d/2]})$, improving upon $O(n^{d—1})$ from previous algorithms.

97 MATHEMATICS AND COMPUTING↗

Potential Impacts of Assimilating All-sky Satellite Infrared Radiances on Convection-Permitting Analysis and Prediction of Tropical Convection

Geostationary infrared satellite observations are spatially dense [< 1/(20 km)2] and temporally frequent (< 1/hour). These suggest the possibility of using these observations to constrain sub-synoptic features over data sparse regions, such as tropical oceans. In this study, the potential impacts of assimilating water vapor channel brightness temperature (WV-BT) observations from the geostationary Meteorological Satellite 7 (Meteosat-7) on tropical convection analysis and prediction were systematically examined through a series of ensemble data assimilation experiments. WV-BT observations were assimilated hourly into convection-permitting ensembles using Penn State’s ensemble square root filter (EnSRF). Comparisons against independently observed Meteosat-7 window channel brightness temperature (Window-BT) show that the assimilation of WV-BT generally improved the intensities and locations of large-scale cloud patterns at spatial scales larger than 100 km. However, comparisons against independent soundings indicate that the EnSRF analysis produced a much stronger dry bias than the no data assimilation experiment. This strong dry bias is associated with the use of the simulated WV-BT from the prior mean during the EnSRF analysis step. A stochastic variant of the ensemble Kalman filter (NoMeanSF) is proposed. The NoMeanSF algorithm was able to assimilate the WV-BT without causing such a strong dry bias and the quality of the analyses’ horizontal cloud pattern is similar to EnSRF’s analyses. Finally, deterministic forecasts initiated from the NoMeanSF analyses possess better horizontal cloud patterns above 500 km than those of the EnSRF. These results suggest that it might be better to assimilate all-sky WV-BT through the NoMeanSF algorithm than EnSRF algorithm.

Chan, Man-Yau↗

The Influence of Prescribed Boundary Conditions on Near-Surface Temperature over the Arctic in the MERRA-2 Atmospheric Model

An accurate historical record of evolving Arctic conditions is integral to furthering our understanding of climate processes and to providing a foundation for predicting future climate scenarios in northern high latitudes. Atmospheric reanalyses are seen as an important source of information on the recent past for the data-sparse Arctic region. An assessment of near-surface Arctic air temperatures finds significant discrepancies among the various modern reanalyses. An important point is the treatment of surface boundary conditions: specifically, the sea ice cover and sea surface temperatures (SSTs) over the Arctic Ocean. Reanalyses use different methodologies and data sources for SSTs and sea ice concentration boundary forcing. Notably, the Modern Era Retrospective analysis for Research and Applications, version 2 (MERRA-2) and the European Centre for Medium-Range Weather Forecasts Interim Re-Analysis (ERA-Interim) both use boundary forcing derived from the Operational Sea Surface Temperature and Sea Ice Analysis (OSTIA) over an extended, overlapping period of time. This allows for an examination of differences between the two systems while both concurrently employ the same fractional sea ice coverage. To further understand these differences, an ensemble of AMIP-style simulations using the MERRA-2 atmospheric model - but without data assimilation - shows considerable differences in Arctic temperatures as compared to reanalyses, particularly in autumn and winter months. Results from the AMIP simulations suggest that the surface representation over sea ice used in the MERRA-2 model provides an intrinsic warm bias and obfuscates Arctic Amplification, an established feature present in observations and reanalyses. An additional ensemble of AMIP-style simulations using the MERRA-2 atmospheric model was performed using boundary conditions derived from the ERA-Interim reanalysis. An in-depth comparison of surface temperatures over the Arctic from the two reanalyses and two AMIP-style ensembles will be presented, along with an assessment of the effects of the varying Arctic temperature time series on the atmospheric general circulation and energy budget.

Collow, Allison B. Marquardt↗

A fuzzy clustering algorithm to detect planar and quadric shapes

In this paper, we introduce a new fuzzy clustering algorithm to detect an unknown number of planar and quadric shapes in noisy data. The proposed algorithm is computationally and implementationally simple, and it overcomes many of the drawbacks of the existing algorithms that have been proposed for similar tasks. Since the clustering is performed in the original image space, and since no features need to be computed, this approach is particularly suited for sparse data. The algorithm may also be used in pattern recognition applications.

Krishnapuram, Raghu↗

Data for: A hybrid biophysical-machine learning framework for diurnal surface energy flux estimation using proximal sensing

Thermal-based remote sensing of surface energy fluxes has traditionally relied on high spatial resolution satellite data with revisit frequencies on the order of weeks. In this study, we evaluate a biophysics-based analytical surface energy balance model for predicting latent energy (LE) and sensible heat (H) fluxes using proximal sensing observations. The Surface Temperature Initiated Closure (STIC1.2) model has been extensively validated across a wide range of spatial and temporal scales using various satellite-derived thermal datasets. Here we extend this validation by applying STIC at sub-hourly temporal resolution over multiple growing seasons for four distinct agricultural systems. We further develop and evaluate novel STIC variants that incorporate machine learning (ML) techniques to eliminate the need for specific surface energy balance observations, specifically net radiation and soil heat flux, thereby enhancing model applicability in data-sparse settings. The integration of an ML component to estimate surface available energy is shown to have strong predictive performance for both LE (R2 = 0.81-0.94) and H (R2 = 0.46-0.72) across all agricultural systems examined here, demonstrating the potential of hybrid biophysical – machine learning approaches for surface energy balance modeling with minimal data requirements. This study concludes with a novel application of explainable machine learning (exML) to diagnose sources of model error. This exML framework attributes residual prediction errors to both model input variables and environmental drivers not explicitly included in the simulation experiments. This approach provides a new pathway for improving model design and integrating previously overlooked yet influential variables into future model iterations.

Agricultural Sciences↗

Lightning Imaging Sensor on the International Space Station: Assessments and Results from First Year Operations

Over two decades, the NASA Marshall Space Flight Center, the University of Alabama in Huntsville, and their partners have demonstrated the effectiveness and value of space-based lightning observations as a remote sensing tool for Earth science research and applications, and, in the process, established a robust global lightning climatology. The Lightning Imaging Sensor (LIS) on the Tropical Rainfall Measuring Mission (TRMM) provided global observations of tropical lightning for an impressive 17 years before that mission came to a close in April 2015. Now a space-qualified LIS, built as the flight spare for TRMM, has been installed on the International Space Station (ISS) for a minimum two year mission following its SpaceX launch on February 19, 2017. The LIS, flown as a hosted payload on the Department of Defense Space Test Program-Houston 5 (STP-H5) mission, was delivered to the ISS in the Dragon trunk and robotically installed in an Earth-viewing position on the outside of the ISS. Following successful activation and checkout, LIS has continuously observed the amount, rate, and radiant energy lightning within its field-of-view as it orbits the Earth. Placing LIS on the Space Station provides a great opportunity to not only extend the 17-year TRMM LIS record of tropical lightning measurements but also to expand that coverage to higher latitudes missed by the previous mission. Furthermore, this mission continues the important science focus to better understand the processes which cause lightning, as well as the connections between lightning and subsequent severe weather events. This understanding is a key to improving weather predictions and saving lives and property here in the United States and around the world. The LIS measurements, along with observations from the new Geostationary Lightning Mapper (GLM) operating on NOAA's newest weather satellites, the Geosynchronous Operational Environmental Satellite-16/17 (GOES-16/17), are being used to cross-validate both systems. An especially unique contribution from the ISS platform is the production of real-time lightning data, especially valuable for operational forecasting and warning applications over data sparse regions such as the oceans. Finally, LIS provides simultaneous and complementary observations with other ISS payloads such as the European Space Agency's Atmosphere-Space Interaction Monitor (ASIM) that is exploring the connection between thunderstorms and lightning with terrestrial gamma-ray flashes (TGFs). Leveraging TRMM's well-established processing and data handling assures that LIS data can be quickly delivered to users.

space-based lightning↗

Monitoring Extreme Weather in the Hindu Kush Himalaya Region

Why is monitoring extreme weather events important? The HKH (Hindu Kush Himalaya region experiences many extreme weather events, such as thunderstorms, especially during monsoon season. These events can cause economic hardship and loss of life. Monitoring Extreme Weather in the HKH Region is a service in development through SERVIR-Hindu Kush Himalaya that aims to develop a customized numerical weather prediction toolkit to assess these high impact events in this relatively data-sparse region. The High Impact Weather Assessment Toolkit (HIWAT) consists of an ensemble Weather Research and Forecasting (WRF)model, threat assessments based on the Global Precipitation Measurement (GPM) missions, and impact assessments based on Landsat and the Moderate Resolution Imaging Spectroradiometer (MODIS) imagery. In spring 2019, we began validation of forecasted precipitation using station data in Bangladesh and Climate Hazards Group InfraRed with Station data (CHIRPS).

Remote Sensing↗

A Hybrid Biophysical‐Machine Learning Framework for Diurnal Surface Energy Flux Estimation Using Proximal Sensing

Thermal infrared-based remote sensing of surface energy fluxes has traditionally relied on high spatial resolution satellite data with revisit frequencies on the order of weeks. In this study, we evaluate a biophysics-based analytical surface energy balance model for predicting latent energy ( LE ) and sensible heat ( H ) fluxes using proximal sensing observations. The Surface Temperature Initiated Closure (STIC1.2) model has been extensively validated across a wide range of spatial and temporal scales using various satellite-derived thermal infrared data sets. Here we extend this validation by applying STIC at sub-hourly temporal resolution over multiple growing seasons for four distinct agricultural systems. We further develop and evaluate novel STIC variants that incorporate machine learning (ML) techniques to eliminate the need for surface energy balance observations, specifically net radiation and soil heat flux, thereby enhancing model applicability in data-sparse settings. The integration of a ML component to estimate surface available energy is shown to have strong predictive performance for both LE (R 2 = 0.81–0.94) and H (R 2 = 0.46–0.72) across all agricultural systems examined here, demonstrating the potential of hybrid biophysical-machine learning approaches for surface energy balance modeling with minimal data requirements. This study concludes with a novel application of explainable machine learning (exML) to diagnose sources of model error. This exML framework attributes residual prediction errors to both model input variables and environmental drivers not explicitly included in the simulation experiments. This approach provides a new pathway for improving model design and integrating previously overlooked yet influential variables into future model iterations.

evapotranspiration↗

The 4D Camera: An 87 kHz Direct Electron Detector for Scanning/Transmission Electron Microscopy

We describe the development, operation, and application of the 4D Camera—a 576 by 576 pixel active pixel sensor for scanning/transmission electron microscopy which operates at 87,000 Hz. The detector generates data at ~480 Gbit/s which is captured by dedicated receiver computers with a parallelized software infrastructure that has been implemented to process the resulting 10–700 Gigabyte-sized raw datasets. The back illuminated detector provides the ability to detect single electron events at accelerating voltages from 30 to 300 kV. Through electron counting, the resulting sparse data sets are reduced in size by 10--300× compared to the raw data, and open-source sparsity-based processing algorithms offer rapid data analysis. The high frame rate allows for large and complex scanning diffraction experiments to be accomplished with typical scanning transmission electron microscopy scanning parameters.

47 OTHER INSTRUMENTATION↗

The suitability of differentiable, physics-informed machine learning hydrologic models for ungauged regions and climate change impact assessment

As a genre of physics-informed machine learning, differentiable process-based hydrologic models (abbreviated as δ or delta models) with regionalized deep-network-based parameterization pipelines were recently shown to provide daily streamflow prediction performance closely approaching that of state-of-the-art long short-term memory (LSTM) deep networks. Meanwhile, δ models provide a full suite of diagnostic physical variables and guaranteed mass conservation. Here, we ran experiments to test (1) their ability to extrapolate to regions far from streamflow gauges and (2) their ability to make credible predictions of long-term (decadal-scale) change trends. We evaluated the models based on daily hydrograph metrics (Nash–Sutcliffe model efficiency coefficient, etc.) and predicted decadal streamflow trends. For prediction in ungauged basins (PUB; randomly sampled ungauged basins representing spatial interpolation), δ models either approached or surpassed the performance of LSTM in daily hydrograph metrics, depending on the meteorological forcing data used. They presented a comparable trend performance to LSTM for annual mean flow and high flow but worse trends for low flow. For prediction in ungauged regions (PUR; regional holdout test representing spatial extrapolation in a highly data-sparse scenario), δ models surpassed LSTM in daily hydrograph metrics, and their advantages in mean and high flow trends became prominent. In addition, an untrained variable, evapotranspiration, retained good seasonality even for extrapolated cases. The δ models' deep-network-based parameterization pipeline produced parameter fields that maintain remarkably stable spatial patterns even in highly data-scarce scenarios, which explains their robustness. Combined with their interpretability and ability to assimilate multi-source observations, the δ models are strong candidates for regional and global-scale hydrologic simulations and climate change impact assessment.

54 ENVIRONMENTAL SCIENCES↗

The GEMPAK Barnes interactive objective map analysis scheme

The analysis scheme and meteorological applications of the GEMPAK data analysis and display software system developed by NASA are described. The program was devised to permit objective, versatile, and practical analysis of satellite meteorological data using a minicomputer and a display system with graphics capability. A data area can be selected within the data file for the globe, and data-sparse regions can be avoided. Distances between observations and the nearest observation points are calculated in order to avoid errors when determining synoptic weather conditions. The Barnes (1973) successive correction method is employed to restore the amplitude of small yet resolvable wavelengths suppressed in an initial filtering pass. The rms deviation is then calculated in relation to available measured data. Examples are provided of treatment of VISSR data from the GOES satellite and a study of the impact of incorrect cloud height data on synoptic weather field analysis.

Koch, S. E.↗

Impact of Atmospheric Infrared Sounder (AIRS) Thermodynamic Profiles on Regional Weather Forecasting

In data sparse regions, remotely-sensed observations can be used to improve analyses and lead to better forecasts. One such source comes from the Atmospheric Infrared Sounder (AIRS), which together with the Advanced Microwave Sounding Unit (AMSU), provides temperature and moisture profiles with accuracy comparable to that of radiosondes. The purpose of this paper is to describe a procedure to assimilate AIRS thermodynamic profile data into a regional configuration of the Advanced Research Weather Research and Forecasting (WRF-ARW) model using its three-dimension variational (3DVAR) analysis component (WRF-Var). Quality indicators are used to select only the highest quality temperature and moisture profiles for assimilation in both clear and partly cloudy regions. Separate error characteristics for land and water profiles are also used in the assimilation process. Assimilation results indicate that AIRS profiles produce an analysis closer to in situ observations than the background field. Forecasts from a 37-day case study period in the winter of 2007 show that AIRS profile data can lead to improvements in 6-h cumulative precipitation forecasts due to instability added in the forecast soundings by the AIRS profiles. Additionally, in a convective heavy rainfall event from February 2007, assimilation of AIRS profiles produces a more unstable boundary layer resulting in enhanced updrafts in the model. These updrafts produce a squall line and precipitation totals that more closely reflect ground-based observations than a no AIRS control forecast. The location of available high-quality AIRS profiles ahead of approaching storm systems is found to be of paramount importance to the amount of impact the observations will have on the resulting forecasts.

Chou, Shih-Hung↗

Improving Regional Forecast by Assimilating Atmospheric InfraRed Sounder (AIRS) Profiles into WRF Model

In data sparse regions, remotely-sensed observations can be used to improve analyses and produce improved forecasts. One such source comes from the Atmospheric InfraRed Sounder (AIRS), which together with the Advanced Microwave Sounding Unit (AMSU), represents one of the most advanced space-based atmospheric sounding systems. The purpose of this paper is to describe a procedure to optimally assimilate high resolution AIRS profile data into a regional configuration of the Advanced Research WRF (ARW) version 2.2 using WRF-Var. The paper focuses on development of background error covariances for the regional domain and background type, and an optimal methodology for ingesting AIRS temperature and moisture profiles as separate overland and overwater retrievals with different error characteristics. The AIRS thermodynamic profiles are derived from the version 5.0 Earth Observing System (EOS) science team retrieval algorithm and contain information about the quality of each temperature layer. The quality indicators were used to select the highest quality temperature and moisture data for each profile location and pressure level. The analyses were then used to conduct a month-long series of regional forecasts over the continental U.S. The long-term impacts of AIRS profiles on forecast were assessed against verifying NAM analyses and stage IV precipitation data.

Chou, Shih-Hung↗

Impact of Atmospheric Infrared Sounder (AIRS) Thermodynamic Profiles on Regional Precipitation Forecasting

In data sparse regions, remotely-sensed observations can be used to improve analyses and lead to better forecasts. One such source comes from the Atmospheric Infrared Sounder (AIRS), which together with the Advanced Microwave Sounding Unit (AMSU), provides temperature and moisture profiles in clear and cloudy regions with accuracy which approaches that of radiosondes. The purpose of this paper is to describe an approach to assimilate AIRS thermodynamic profile data into a regional configuration of the Advanced Research WRF (ARW) model using WRF-Var. Quality indicators are used to select only the highest quality temperature and moisture profiles for assimilation in clear and partly cloudy regions, and uncontaminated portions of retrievals above clouds in overcast regions. Separate error characteristics for land and water profiles are also used in the assimilation process. Assimilation results indicate that AIRS profiles produce an analysis closer to in situ observations than the background field. Forecasts from a 37-day case study period in the winter of 2007 show that AIRS profile data can lead to improvements in 6-h cumulative precipitation forecasts resulting from improved thermodynamic fields. Additionally, in a convective heavy rainfall event from February 2007, assimilation of AIRS profiles produces a more unstable boundary layer resulting in enhanced updrafts in the model. These updrafts produce a squall line and precipitation totals that more closely reflect ground-based observations than a no AIRS control forecast. The location of available high-quality AIRS profiles ahead of approaching storm systems is found to be of paramount importance to the amount of impact the observations will have on the resulting forecasts.

Chou, S.-H.↗

The Transition of NASA EOS Datasets to WFO Operations: A Model for Future Technology Transfer

The collocation of a National Weather Service (NWS) Forecast Office with atmospheric scientists from NASA/Marshall Space Flight Center (MSFC) in Huntsville, Alabama has afforded a unique opportunity for science sharing and technology transfer. Specifically, the NWS office in Huntsville has interacted closely with research scientists within the SPORT (Short-term Prediction and Research and Transition) Center at MSFC. One significant technology transfer that has reaped dividends is the transition of unique NASA EOS polar orbiting datasets into NWS field operations. NWS forecasters primarily rely on the AWIPS (Advanced Weather Information and Processing System) decision support system for their day to day forecast and warning decision making. Unfortunately, the transition of data from operational polar orbiters or low inclination orbiting satellites into AWIPS has been relatively slow due to a variety of reasons. The ability to integrate these high resolution NASA datasets into operations has yielded several benefits. The MODIS (MODerate-resolution Imaging Spectrometer ) instrument flying on the Aqua and Terra satellites provides a broad spectrum of multispectral observations at resolutions as fine as 250m. Forecasters routinely utilize these datasets to locate fine lines, boundaries, smoke plumes, locations of fog or haze fields, and other mesoscale features. In addition, these important datasets have been transitioned to other WFOs for a variety of local uses. For instance, WFO Great Falls Montana utilizes the MODIS snow cover product for hydrologic planning purposes while several coastal offices utilize the output from the MODIS and AMSR-E instruments to supplement observations in the data sparse regions of the Gulf of Mexico and western Atlantic. In the short term, these datasets have benefited local WFOs in a variety of ways. In the longer term, the process by which these unique datasets were successfully transitioned to operations will benefit the planning and implementation of products and datasets derived from both NPP and NPOESS. This presentation will provide a brief overview of current WFO usage of satellite data, the transition of datasets between SPORT and the N W S , and lessons learned for future transition efforts.

Darden, C.↗

Biological Research and Space Health Enabled by Machine Learning to Support Deep Space Missions

A key science goal of the NASA “Moon to Mars” campaign is to understand how biology responds to the Lunar, Martian, and deep space environments in order to advance fundamental knowledge and support human space missions. Through artificial intelligence (AI) and machine learning (ML), a paradigm shift has begun in space biosciences and engineered astronaut health systems, to enable Earth-independence and mission operations autonomy. We describe current AI/ML methods to support 1) fundamental biology, 2) in situ analytics, 3) high performance computing, 4) automated science, 5) self-driving labs, 6) remote data management, 7) integrated mission biomonitoring, and 8) a Precision Space Health system. AI/ML approaches that can be integrated to support these domains include active learning, explainable AI, adaptive learning, causal inference, knowledge graphs, federated learning, transfer learning, and large language models. Finally, we present results from several current ML projects that are underway in the space biology field to address key challenges of small sample n, high feature count, heterogeneity, and sparse data. These include 1) connecting omics to phenotypic data using an ensemble model to infer causality of rodent liver health disruption, 2) usage of explainable ML to interrogate muscular underpinnings of muscle atrophy, 3) ML models analyzing and determining directed acyclic graphs of human health risk leveraging rodent bone datasets, 4) usage of large pre-trained models connecting biomedical knowledgebases with small spaceflight datasets to understand gene-to-gene interactions, and 5) a suite of benchmarked open science datasets enabling programmers to identify best algorithms to answer space biology questions.

space biology↗

Refining HPCToolkit for application performance analysis at exascale

As part of the US Department of Energy’s Exascale Computing Project (ECP), Rice University has been refining its HPCToolkit performance tools to better support measurement and analysis of applications executing on exascale supercomputers. To efficiently collect performance measurements of GPU-accelerated applications, HPCToolkit employs novel non-blocking data structures to communicate performance measurements between tool threads and application threads. To attribute performance information in detail to source lines, loop nests, and inlined call chains, HPCToolkit performs parallel analysis of large CPU and GPU binaries involved in the execution of an exascale application to rapidly recover mappings between machine instructions and source code. To analyze terabytes of performance measurements gathered during executions at exascale, HPCToolkit employs distributed-memory parallelism, multithreading, sparse data structures, and out-of-core streaming analysis algorithms. To support interactive exploration of profiles up to terabytes in size, HPCToolkit’s hpcviewer graphical user interface uses out-of-core methods to visualize performance data. The result of these efforts is that HPCToolkit now supports collection, analysis, and presentation of profiles and traces of GPU-accelerated applications at exascale. These improvements have enabled HPCToolkit to efficiently measure, analyze and explore terabytes of performance data for executions using as many as 64K MPI ranks and 64K GPU tiles on ORNL’s Frontier supercomputer. HPCToolkit’s support for measurement and analysis of GPU-accelerated applications has been employed to study a collection of open-science applications developed as part of ECP. This paper reports on these experiences, which provided insight into opportunities for tuning applications, strengths and weaknesses of HPCToolkit itself, as well as unexpected behaviors in executions at exascale.

Adhianto, Laksono↗