Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data diversity”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 469 records · Page 26
All sourcesNASA NTRSDOE OSTI
NASA NTRS2019

Cloud Giovanni: Reining in Costs and Improving Performance with Analytical Data Stores Using Scalable Serverless Architecture

Giovanni is the Geospatial Interactive Online Visualization ANd aNalysis Infrastructure developed at NASA GES DISC which provides a simple and intuitive way to visualize, analyze, and access vast amounts of Earth science data. It receives large number of user requests each day for a variety of analysis and visualization services, which leads to the big data challenge of serving gradually increasing large data volumes with diverse statistical algorithms. We hereby propose a multi-dimensional accumulation method which provides fast and cost-efficient cloud analysis for diverse services including both area averaging and time averaging. This method involves the weighted volume integration over multiple variable dimensions (time and space), and is implemented in AWS using Athena providing serverless and highly scalable data analysis. Compared to the standard method, this approach dramatically reduced the computational time by order of magnitude with a minimal AWS cost incurred. For example, for a benchmark of 10-year area averaging over the 1x1 degree daily variable, the computational time was reduced from minutes to seconds, and the Athena cost is only $5 for 100,000 requests.

Zhang, Hailiang↗
DOE OSTI2026

Leveraging machine learning to enhance aerosol classification using Single-Particle Mass Spectrometry

Advancing automated classification of atmospheric aerosols from Single-Particle Mass Spectrometry (SPMS) data remains challenging due to overlapping ion signatures, compositional diversity, and limited labeled data. This study evaluates supervised and semi-supervised learning frameworks to enhance aerosol identification by jointly leveraging labeled and unlabeled spectra. Four models were compared: a supervised Support Vector Machine (SVM), a self-training SVM, a stacked autoencoder classifier, and a stacked autoencoder trained using a temporal-ensembling Mean Teacher approach. All models achieved high and stable accuracies (90.0 %–91.1 %), surpassing previous results on the same dataset (87 %) and matching the performance of state-of-the-art deep learning methods. Despite small global metric differences (≤ 1 %), semi-supervised variants yielded up to 5 %–10 % improvements for compositionally rare particle types – such as soot (0.77 % of spectra, F1-score: 0.93–0.97) and hazelnut pollen (0.98 % of spectra, F1-score: 0.97–1.00) – equating to roughly ∼ 187 additional correctly classified spectra. These gains are scientifically significant, as such rare particles exert disproportionate influence on radiative absorption and ice nucleation processes; their improved detection reduces modeled uncertainties in aerosol absorption optical depth and mixed-phase cloud ice nucleation rates. The models' residual misclassifications (≈ 9 %) largely arise from true spectral overlap among chemically adjacent species (e.g., Na- vs. K-feldspar, coated vs. uncoated feldspars), reflecting physical compositional continuity rather than algorithmic error. Collectively, these findings demonstrate that leveraging unlabeled data to learn robust spectral representations and refine classification enhances both fidelity and interpretability, bridging data-driven analysis with aerosol–climate process understanding.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2019

Using Big Data Technologies with Earth Science Data in HDF5: HDF5 Scalable Solutions

HDF5 (Hierarchical Data Format 5) is open-source, high-performance software that consists of an abstract data model, library, and fileformat used for storing and managing extremely large and/or complex data collections. NASA Earth Observing System (EOS) Data and Information Systems use HDF5 as an archival format to store remote sensing data from EOS satellites. HDF5 is also used to store other types of Geoscience and Strophysical data, e.g., seismic data and data from Low-Frequency Array (LOFAR) radio telescopes. Data stored in HDF5 has reached tens of petabytes and is growing at an accelerated rate.With the growing amout of HDF5 Earth Science data to analyze and process, scientists need to adopt big data technologies including new storage paradigms such as cloud and object storage. To run models and perform data analysis they also need to utilizied efficient and diverse ways to access data, from high-performance computing's (HPC) Message Passing Interface (MPI) I/O and deep memory hierarchies (DMH) to non-HPC frameworks such as Apache Hadoop, Spark, and Drill. The HDF Group continually works to enable usage of big data technologies in HDF software.

Knox, Larry↗
NASA NTRS2018

Increasing the Statistical Rigor of Cross-Species Differential Expression Analysis

Microgravity inflicts substantial, but undercharacterized, pressure on organisms that induces metabolic responses such as increased microbial virulence and antibiotic resistance, altered organ weights in developing rats, and loss of bone tissue in astronauts. Numerous studies have analyzed the effects of microgravity on specific organisms, tissues, or test conditions, but these projects are necessarily limited by the small sample size of space research. Increasing the sample size of spaceflight studies is non-trivial; however, pooling data from numerous studies can greatly increase the statistical rigor of comparative analyses. The GeneLab houses datasets from 73 spaceflight studies that performed transcription profiling assays. These data encompass a diverse array of organisms ranging from Escherichia coli to Mus musculus to Homo sapiens and comprise studies analyzing ionizing radiation, mammalian pregnancy, etc. Collectively, the GeneLab database contains a large quantity of transcription assays and RNA sequence data analyzing Differential Gene Expression (DGE) between microand normogravity. Xspecies, a cross-species analysis method for DGE developed by Kristiansson, et al. in 2012, identifies homologous genes between species that are universally up- or downregulated in response to test conditions. Previous work by an intern at GeneLab applied Xspecies to 19 datasets containing seven different species and identified 14 homologous groups differentially expressed under spaceflight conditions including several heat shock proteins and cytoskeletal components. Unfortunately, these results may be biased by the disproportionate number of studies on Arabidopsis thaliana (5) and Mus musculus (6) and the results are not normalized by evolutionary distances. Here, we present modifications to the Xspecies algorithm that permits incorporation of multi-omic data and normalizes data for effect size, directionality, and evolutionary distances. We then apply this algorithm to all currently available GeneLab studies

Xspecies↗
NASA NTRS2014

Cultivating an Emergent Earth Observation Analytics Ecosystem in the Cloud

A diverse set of data analytics systems for Earth Observations are sprouting up in the Earth Science community, with a wealth of processing algorithms and analysis methods. There is a similar wealth of data resources available via myriad data providers and clearinghouses, including large institutional systems like the Earth Observing System Data and Information System, Comprehensive Large Scale Array-data Stewardship System, and Federated Earth Observation Missions gateway. With Earth system science driving a need to work with more datasets together, and the community developing more analysis tools (some of them dataset-specific), how can we develop analysis workflows that incorporate far-flung datasets and leverage analysis resources from multiple organizations? Cloud computing points the way toward a solution in two different respects. Firstly, the access to and abstraction of virtually unlimited storage and computing power provides an environment that enables more straightforward means of pulling datasets and analysis resources together. Just as importantly, however, cloud computing serves as an example of an "ecosystem" of interoperating services, since the essence of cloud computing is the presentation of all resources as a service, from hardware to infrastructure to platform to software. This enables the combination of off-the-shelf, diverse services to construct entire systems that emerge out of an equally diverse community of architects and developers. This approach can be similarly applied to the data and analysis resources in the Earth Observation community. By exposing these resources via well understood services, and consuming resources in the same way, different organizations can construct bespoke analysis workflows and systems for their own purposes. The key leap the community needs to make is to develop analysis systems in components that interact with other components via services. The result would be a rich ecosystem of analytics components that can be combined to analyze datasets at scale and in conjunction with other datasets from other sources.

chaos↗
NASA NTRS1995

Evaluation of phase-diversity techniques for solar-image restoration

Phase-diversity techniques provide a novel observational method for overcomming the effects of turbulence and instrument-induced aberrations in ground-based astronomy. Two implementations of phase-diversity techniques that differ with regard to noise model, estimator, optimization algorithm, method of regularization, and treatment of edge effects are described. Reconstructions of solar granulation derived by applying these two implementations to common data sets are shown to yield nearly identical images. For both implementations, reconstructions from phase-diverse speckle data (involving multiple realizations of turbulence) are shown to be superior to those derived from conventional phase-diversity data (involving a single realization). Phase-diverse speckle reconstructions are shown to achieve near diffraction-limited resolution and are validated by internal and external consistency tests, including a comparison with a reconstruction using a well-accepted speckle-imaging method.

Paxman, Richard G.↗
NASA NTRS2012

Remote Sensing and Modeling of Landslides: Detection, Monitoring and Risk Evaluation

Landslides are one of the most pervasive hazards in the world, resulting in more fatalities and economic damage than is generally recognized_ Occurring over an extensive range of lithologies, morphologies, hydrologies, and climates, mass movements can be triggered by intense or prolonged rainfall, seismicity, freeze/thaw processes, and antbropogertic activities, among other factors. The location, size, and timing of these processes are characteristically difficult to predict and assess because of their localized spatial scales, distribution, and complex interactions between rainfall infiltration, hydromechanical properties of the soil, and the underlying surface composition. However, the increased availability, accessibility, and resolution of remote sensing data offer a new opportunity to explore issues of landslide susceptibility, hazard, and risk over a variety of spatial scales. This special issue presents a series of papers that investigate the sources, behavior, and impacts of different mass movement types using a diverse set of data sources and evaluation methodologies.

Kirschbaum, Dalia↗
DOE OSTI2025

Advanced Interactive 3D Visualization Tool for Customizable Analyses of Tomography Datasets in Material Science

Current methods for visualizing and analyzing 3D tomography datasets in materials science often lack the interactivity and depth required for detailed structural insights. This limitation restricts a researchers' ability to accurately interpret complex data, which is critical for advancing material innovations and understanding structural properties. To address this issue, we have developed a novel, web-based interactive 3D visualization and analysis tool from the Trame framework that offers customizable features to enhance data interpretability. The tool allows users to adjust parameters such as visible range, slice planes, data rotation, and layering, providing a more detailed and dynamic view of complex structures. Its user-friendly web interface increases the accessibility and ease of use for both novice and experienced researchers, to visualize large volumetric datasets. The tool supports a diverse range of data formats, making it versatile for various research applications. Unique capabilities include real-time data manipulation, automated feature detection, context-sensitive feedback, and real-time volume calculations and distributions per sliced region or layer, alongside the ability to quickly generate high-quality screenshots and videos for presentations and reports. These advancements offer a comprehensive solution for enhanced 3D data exploration, significantly improving the analysis process and communication of results in materials science.

36 - MATERIALS SCIENCE↗
DOE OSTI2021

Root carbon interation with soil minerals is dynamic, leaving a legacy of microbially derived residues

Minerals preserve the oldest most persistent soil carbon, and mineral characteristics appear to play a critical role in the formation of soil organic matter (SOM) associations. To test the hypothesis that carbon source and soil microorganisms also influence mineral-SOM associations, we incubated permeable minerals bags in soil microcosms with and without plants, in a 13CO2 labelling chamber. Mineral bags contained quartz, ferrihydrite, kaolinite, or native soil minerals isolated via density separation. Using 13C-NMR, FTICR-MS, and lipidomics, we traced plant-derived carbon onto minerals harvested from microcosms at three plant growth stages, characterizing total carbon, 13C enrichment, and SOM chemistry. While C accumulation was rapid and mineral-dependent, the accumulated amount was not significantly affected by the presence of plant roots. However, the rhizosphere did shape the chemistry of mineral-associated SOM. Minerals incubated in the rhizosphere were associated with a more diverse array of compounds with different C functional groups (carbonyl, aromatics, carbohydrates, lipids) than minerals incubated in a bulk soil control. These diverse rhizosphere-derived compounds may represent a “transient fraction” of mineral SOM, rapidly exchanging with mineral surfaces. Our results also suggest that many of the lipids which persist on minerals are microbially-derived with a large fraction of fungal lipids.

Rhizosphere, soil organic matter, grassland, 13C-N↗
DOE OSTI2023

Wind Resource Data for Southeast Asia Using a Hybrid Numerical Weather Prediction with Machine Learning Super Resolution Approach

In this work we develop and present a machine learning based downscaling approach using generative adversarial networks (GANs). GANs learn to distinguish the relationships between low-resolution and high-resolution simulations and generate accurate high-resolution output from low-resolution input (Stengel, Glaws, Hettinger, & King, 2020). Low-resolution numerical weather prediction (NWP) simulations at 9-km spatial and 60-minute temporal resolution were executed over Southeast Asia to provide input to the GANs model. GANs for wind, temperature, and pressure were trained on coarsened WIND Toolkit data with a diverse sampling of terrain and meteorological conditions. After training, the NWP simulations over Southeast Asia were enhanced by 3x along each horizontal spatial dimension and 4x along the temporal dimension. This novel downscaling approach generated 15-year high-resolution wind, temperature, and pressure data from January 2007 through December 2021 at multiple hub heights over Southeast Asia at 3-km spatial and 15-minute temporal resolution with a 16x reduction in compute time over standard dynamical downscaling.

17 WIND ENERGY↗
NASA NTRS2005

The Contribution of CEOP Data to the Understanding and Modeling of Monsoon Systems

CEOP has contributed and will continue to provide integrated data sets from diverse platforms for better understanding of the water and energy cycles, and for validaintg models. In this talk, I will show examples of how CEOP has contributed to the formulation of a strategy for the study of the monsoon as a system. The CEOP data concept has led to the development of the CEOP Inter-Monsoon Studies (CIMS), which focuses on the identification of model bias, and improvement of model physics such as the diurnal and annual cycles. A multi-model validation project focusing on diurnal variability of the East Asian monsoon, and using CEOP reference site data, as well as CEOP integrated satellite data is now ongoing. Preliminary studies show that climate models have difficulties in simulating the diurnal signals of total rainfall, rainfall intensity and frequency of occurrence, which have different peak hours, depending on locations. Further more model diurnal cycle of rainfall in monsoon regions tend to lead the observed by about 2-3 hours. These model bias offer insight into lack of, or poor representation of, key components of the convective and stratiform rainfall. The CEOP data also stimulated studies to compare and contrasts monsoon variability in different parts of the world. It was found that seasonal wind reversal, orographic effects, monsoon depressions, meso-scale convective complexes, SST and land surface land influences are common features in all monsoon regions. Strong intraseasonal variability is present in all monsoon regions. While there is a clear demarcation of onset, breaks and withdrawal in the Asian and Australian monsoon region associated with climatological intraseasonal variabillity, it is less clear in the American and Africa monsoon regions. The examination of satellite and reference site data in monsoon has led to preliminary model experiments to study the impact of aerosol on monsoon variability. I will show examples of how the study of the dynamics of aerosol-water cycle interactions in the monsoon region, can be best achieved using the CEOP data and modeling strategy.

Lau, William K. M.↗
NASA NTRS2005

The contribution of CEOP data to the understanding and modeling of monsoon systems

CEOP has contributed and will continue to provide integrated data sets from diverse platforms for better understanding of the water and energy cycles, and for validating models. In this talk, I will show examples of how CEOP has contributed to the formulation of a strategy for the study of the monsoon as a system. The CEOP data concept has led to the development of the CEOP Inter-Monsoon Studies (CIMS), which focuses on the identification of model bias, and improvement of model physics such as the diurnal and annual cycles. A multi-model validation project focusing on diurnal variability of the East Asian monsoon, and using CEOP reference site data, as well as CEOP integrated satellite data is now ongoing. Similar validation projects in other monsoon regions are being started. Preliminary studies show that climate models have difficulties in simulating the diurnal signals of total rainfall, rainfall intensity and frequency of occurrence, which have different peak hours, depending on locations. Further more model diurnal cycle of rainfall in monsoon regions tend to lead the observed by about 2-3 hours. These model bias offer insight into lack of, or poor representation of key components of the convective,and stratiform rainfall. The CEOP data also stimulated studies to compare and contrasts monsoon variability in different parts of the world. It was found that seasonal wind reversal, orographic effects, monsoon depressions, meso-scale convective complexes, SST and land surface land influences are common features in all monsoon regions. Strong intraseasonal variability is present in all monsoon regions. While there is a clear demarcation of onset, breaks and withdrawal in the Asian and Australian monsoon region associated with climatological intraseasonal variability, it is less clear in the American and Africa monsoon regions. The examination of satellite and reference site data in monsoon has led to preliminary model experiments to study the impact of aerosol on monsoon variability. I will show examples of how the study of the dynamics of aerosol-water cycle interactions in the monsoon region, can be best achieved using the CEOP data and modeling strategy.

Lau, William K. M.↗
DOE OSTI2014

Cost effective nanostructured copper substrates prepared with ultrafast laser pulses for explosives detection using surface enhanced Raman scattering

Ultrafast laser pulses induced surface nanostructures were fabricated on a copper (Cu) target through ablation in acetone, dichloromethane, acetonitrile, and chloroform. Surface morphological information accomplished from the field emission scanning electron microscopic data demonstrated the diversities of ablation mechanism in each case. Fabricated Cu substrates were utilized exultantly to investigate the surface plasmon (localized and propagating) mediated enhancements of different analytes using surface enhance Raman scattering (SERS) studies. Multiple utility of these substrates were efficiently demonstrated by collecting the SERS data of Rhodamine 6G molecule and two different secondary explosive molecules such as 5-amino-3-nitro-l,2,4-triazole and trinitrotoluene on different days which were weeks apart. We achieved significant enhancement factors of >10{sup 5} through an easily adoptable cleaning procedure.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
NASA NTRS1971

A 15.3 GHz satellite-to-ground diversity propagation experiment using a terminal separation of 4 kilometers

The performance of a path diversity satellite-to-ground millimeter wave link with two ground terminals separated by 4 km is discussed. At this separation distance the duration of fades below 6 dB was decreased by at least a factor of 10 when using path diversity and the cumulative crosscorrelation between the attenuations observed at the two terminals during rain events was approximately 0.45. Narrow beam radiometers directed along the propagation paths were also utilized to relate the path radiometric temperature to the path attenuation. An analysis of downlink propagation data for generating diversity link performance statistics is included.

Grimm, K. R.↗
NASA NTRS2017

Models of Sector Flows Under Local, Regional and Airport Weather Constraints

Recently, the ATM community has made important progress in collaborative trajectory management through the introduction of a new FAA traffic management initiative called a Collaborative Trajectory Options Program (CTOP). FAA can use CTOPs to manage air traffic under multiple constraints (manifested as flow constrained areas or FCAs) in the system, and it allows flight operators to indicate their preferences for routing and delay options. CTOPs also permits better management of the overall trajectory of flights by considering both routing and departure delay options simultaneously. However, adoption of CTOPs in airspace has been hampered by many factors that include challenges in how to identify constrained areas and how to set rates for the FCAs. Decision support tools providing assistance would be particularly helpful in effective use of CTOPs. Such DSTs tools would need models of demand and capacity in the presence of multiple constraints. This study examines different approaches to using historical data to create and validate models of maximum flows in sectors and other airspace regions in the presence of multiple constraints. A challenge in creating an empirical model of flows under multiple constraints is a lack of sufficient historical data that captures diverse situations involving combinations of multiple constraints especially those with severe weather. The approach taken here to deal with this is two-fold. First, we create a generalized sector model encompassing multiple sectors rather than individual sectors in order to increase the amount of data used for creating the model by an order of magnitude. Secondly, we decompose the problem so that the amount of data needed is reduced. This involves creating a baseline demand model plus a separate weather constrained flow reduction model and then composing these into a single integrated model. A nominal demand model is a flow model (gdem) in the presence of clear local weather. This defines the flow as a function of weather constraints in neighboring regions, airport constraints and weather in locations that can cause re-routes to the location of interest. A weather constrained flow reduction model (fwx-red) is a model of reduction in baseline counts as a function of local weather. Because the number of independent variables associated with each of the two decomposed models is smaller than that with a single model, need for amount of data is reduced. Finally, a composite model that combines these two can be represented as fwx-red (gdem(e), l) where e represents non-local constraints and l represents local weather. The approaches studied to developing these models are divided into three categories: (1) Point estimation models (2) Empirical models (3) Theoretical models. Errors in predictions of these different types of models have been estimated. In situations when there is abundant data, point estimation models tend to be very accurate. In contrast, empirical models do better than theoretical models when there is some data available. The biggest benefit of theoretical models is their general applicability in wider range situations once the degree of accuracy of these has been established.

Kulkarni, Deepak↗