Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “sparse data set”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7
All sourcesNASA NTRSDOE OSTI
NASA NTRS2021

Validation of Soil Moisture Data Products from the NASA SMAP Mission

The National Aeronautics and Space Administration (NASA)Soil Moisture Active Passive(SMAP) mission has been validatingits soil moisture (SM) products since the start of data production onMarch 31, 2015. Prior to launch, the mission defined a set of criteria for core validation sites (CVS) that enable the testing of the key mission SM accuracy requirement(unbiased root-mean-square error <0.04 m3/m3). Thevalidation approach also includes other (“sparse network”) in situSM measurements, satellite SM products, model-based SM products, and field experiments. Over the past six years, the SMAP SM products have been analyzed with respect to thesereference data,and the analysis approaches themselves have been scrutinizedin an effort to best understand the products’ performance. Validation of themost recent SMAP Level 2 and 3 SMretrievalproducts (R17000) shows that the L-band (1.4 GHz) radiometer-based SM record continues to meet mission requirements. The products aregenerallyconsistentwith SM retrievals from the European Space Agency (ESA)Soil Moisture Ocean Salinity mission, althoughthere aredifferencesin some regions. The high-resolution (3-km) SM retrieval product,generated by combining Copernicus Sentinel-1 data with SMAP observations,performswithin expectations. Currently, however,there is limited availability of3-kmCVSdatato support extensive validation at this spatial scale. The most recent (version 5)SMAP Level4 SMdata assimilation productprovidingsurface and root-zone SM with complete spatio-temporal coverageat 9-km resolution also meets performance requirements. The SMAP SM validation program will continue throughout the mission life; futureplans include expanding ittoforestedand high-latituderegions

SMAP↗
DOE OSTI2024

The 3D Lyman- α forest power spectrum from eBOSS DR16

We measure the three-dimensional power spectrum (P3D) of the transmitted flux in the Lyman-α (Ly α) forest using the complete extended Baryon Oscillation Spectroscopic Survey data release 16 (eBOSS DR16). This sample consists of ~205 000 quasar spectra in the redshift range 2 ≤ z ≤ 4 at an effective redshift z = 2.334. We propose a pair-count spectral estimator in configuration space, weighting each pair by exp( i k ∙ r), for wave vector k and pixel pair separation r, effectively measuring the anisotropic power spectrum without the need for fast Fourier transforms. This accounts for the window matrix in a tractable way, avoiding artefacts found in Fourier-transform based power spectrum estimators due to the sparse sampling transverse to the line of sight of Ly α skewers. We extensively test our pipeline on two sets of mocks: (i) idealized Gaussian random fields with a sparse sampling of Ly α skewers, and (ii) log-normal LyaCoLoRe mocks including realistic noise levels, the eBOSS survey geometry and contaminants. On eBOSS DR16 data, the Kaiser formula with a non-linear correction term obtained from hydrodynamic simulations yields a good fit to the power spectrum data in the range $(0.02 ≤ k ≤ 0.35)$ h Mpc -1 at the 1–2σ level with a covariance matrix derived from LyaCoLoRe mocks. We demonstrate a promising new approach for full-shape cosmological analyses of Ly α forest data from cosmological surveys such as eBOSS, the currently observing Dark Energy Spectroscopic Instrument and future surveys such as the Prime Focus Spectrograph, WEAVE-QSO, and 4MOST.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

Quantifying uncertainty for deep learning based forecasting and flow-reconstruction using neural architecture search ensembles

Classical problems in computational physics such as data-driven forecasting and signal reconstruction from sparse sensors have recently seen an explosion in deep neural network (DNN) based algorithmic approaches. However, most DNN models do not provide uncertainty estimates, which are crucial for establishing the trustworthiness of these techniques in downstream decision making tasks and scenarios. In recent years, ensemble-based methods have achieved significant success for the uncertainty quantification in DNNs on a number of benchmark problems. However, their performance on real-world applications remains under-explored. In this work, we present an automated approach to DNN discovery and demonstrate how this may also be utilized for ensemble-based uncertainty quantification. Specifically, we propose the use of a scalable neural and hyperparameter architecture search for discovering an ensemble of DNN models for complex dynamical systems. We highlight how the proposed method not only discovers high-performing neural network ensembles for our tasks, but also quantifies uncertainty seamlessly. This is achieved by using genetic algorithms and Bayesian optimization for sampling the search space of neural network architectures and hyperparameters. Subsequently, a model selection approach is used to identify candidate models for an ensemble set construction. Afterwards, a variance decomposition approach is used to estimate the uncertainty of the predictions from the ensemble. We demonstrate the feasibility of this framework for two tasks — forecasting from historical data and flow reconstruction from sparse sensors for the sea-surface temperature. In conclusion, we demonstrate superior performance from the ensemble in contrast with individual high-performing models and other benchmarks.

Deep ensembles↗
DOE OSTI2025

Survey-wide asteroid discovery with a high-performance computing enabled non-linear digital tracking framework

Modern astronomical surveys detect asteroids by linking together their appearances across multiple images taken over time. This approach faces limitations in detecting faint asteroids and handling the computational complexity of trajectory linking. Here, we present a novel method that adapts “digital tracking” – traditionally used for short-term linear asteroid motion across images – to work with large-scale synoptic surveys such as the Vera Rubin Observatory Legacy Survey of Space and Time (Rubin/LSST). Our approach combines hundreds of sparse observations of individual asteroids across their non-linear orbital paths to enhance detection sensitivity by several magnitudes. To address the computational challenges of processing massive data sets and dense orbital phase spaces, we developed a specialized high-performance computing architecture. We demonstrate the effectiveness of our method through experiments that take advantage of the extensive computational resources at Lawrence Livermore National Laboratory. This work enables the detection of significantly fainter asteroids in existing and future survey data, potentially increasing the observable asteroid population by orders of magnitude across different orbital families, from near-Earth objects (NEOs) to Kuiper belt objects (KBOs).

Asteroid discovery↗
NASA NTRS2007

Efficient Implementation of an Optimal Interpolator for Large Spatial Data Sets

Interpolating scattered data points is a problem of wide ranging interest. A number of approaches for interpolation have been proposed both from theoretical domains such as computational geometry and in applications' fields such as geostatistics. Our motivation arises from geological and mining applications. In many instances data can be costly to compute and are available only at nonuniformly scattered positions. Because of the high cost of collecting measurements, high accuracy is required in the interpolants. One of the most popular interpolation methods in this field is called ordinary kriging. It is popular because it is a best linear unbiased estimator. The price for its statistical optimality is that the estimator is computationally very expensive. This is because the value of each interpolant is given by the solution of a large dense linear system. In practice, kriging problems have been solved approximately by restricting the domain to a small local neighborhood of points that lie near the query point. Determining the proper size for this neighborhood is a solved by ad hoc methods, and it has been shown that this approach leads to undesirable discontinuities in the interpolant. Recently a more principled approach to approximating kriging has been proposed based on a technique called covariance tapering. This process achieves its efficiency by replacing the large dense kriging system with a much sparser linear system. This technique has been applied to a restriction of our problem, called simple kriging, which is not unbiased for general data sets. In this paper we generalize these results by showing how to apply covariance tapering to the more general problem of ordinary kriging. Through experimentation we demonstrate the space and time efficiency and accuracy of approximating ordinary kriging through the use of covariance tapering combined with iterative methods for solving large sparse systems. We demonstrate our approach on large data sizes arising both from synthetic sources and from real applications.

Memarsadeghi, Nargess↗
DOE OSTI2023

FAST (FAST AUTONOMOUS SCANNING TOOLKIT)

SF-23-006 FAST (FAST AUTONOMOUS SCANNING TOOLKIT)The software is deployed on an edge computing device at the beamline computer attached to a scanning microscope. It iteratively analyzes the data collected, then identifies new scan positions to scan next and directs the positioners that move the sample (or probe beam) to these positions. Overall, it identifies a sparse set of scan positions that are sufficient to image the full sample. This can reduce the scan time by >60%.

KANDEL, SAUGAT↗
NASA NTRS1987

Physical mechanisms of heat, momentum and turbulence fluxes

In a qualitative way, the physical mechanisms which generate fluxes of heat, momentum, and turbulence in the atmosphere are discussed. This material is presented to acquaint people with the Earth science aspects of turbulence as important processes in the atmosphere. To attempt to describe turbulent fluxes of heat, momentum, and moisture in precise mathematical detail becomes an intractable problem. It is burdened by an eighth order set of equations involving more variables than equations. It is a closure problem which requires complicated assumptions that are not necessarily always satisfied, variable boundary conditions, and sparse observational data. Therefore, we must approach the problem in a simplified manner to obtain any kind of solution involving the variables of shear, stress, and heat, moisture, and momentum fluxes. There are other problems, of course, in which the inclusion of the planetary boundary layer is extremely important. Air pollution studies, air-sea exchanges, mesoscale models, and so on, must account for the planetary layer in very specific terms. Some of the physical mechanisms that are involved in generating fluxes are described.

Theon, John S.↗
NASA NTRS2007

Using High Frequency Passive Microwave, A-train, and TRMM Data to Evaluate Hydrometer Structure in the NASA GEOS-5 Data Assimilation System

Validating water vapor and prognostic condensate in global models remains a challenging research task. Model parameterizations are still subject to a large number of tunable parameters; furthermore, accurate and representative in situ observations are very sparse, and satellite observations historically have significant quantitative uncertainties. Progress on improving cloud / hydrometeor fields in models stands to benefit greatly from the growing inventory ofA-Train data sets. ill the present study we are using a variety of complementary satellite retrievals of hydrometeors to examine condensate produced by the emerging NASA Modem Era Retrospective Analysis for Research and Applications, MERRA, and its associated atmospheric general circulation model GEOS5. Cloud and precipitation are generated by both grid-scale prognostic equations and by the Relaxed Arakawa-Schubert (RAS) diagnostic convective parameterization. The high frequency channels (89 to 183.3 GHz) from AMSU-B and MRS on NOAA polar orbiting satellites are being used to evaluate the climatology and variability of precipitating ice from tropical convective anvils. Vertical hydrometeor structure from the Tropical Rainfall Measuring Mission (TRMM) and CloudSat radars are used to develop statistics on vertical hydrometeor structure in order to better interpret the extensive high frequency passive microwave climatology. Cloud liquid and ice water path data retrieved from the Moderate Resolution Imaging Spectroradiometer, MODIS, are used to investigate relationships between upper level cloudiness and tropical deep convective anvils. Together these data are used to evaluate cloud / ice water path, gross aspects of vertical hydrometeor structure, and the relationship between cloud extent and surface precipitation that the MERRA reanalysis must capture.

Robertson, Franklin↗
NASA NTRS2010

An Aircraft-Based Upper Troposphere Lower Stratosphere O3, CO, and H2O Climatology for the Northern Hemisphere

We present a climatology of O3, CO, and H2O for the upper troposphere and lower stratosphere (UTLS), based on a large collection of high ]resolution research aircraft data taken between 1995 and 2008. To group aircraft observations with sparse horizontal coverage, the UTLS is divided into three regimes: the tropics, subtropics, and the polar region. These regimes are defined using a set of simple criteria based on tropopause height and multiple tropopause conditions. Tropopause ]referenced tracer profiles and tracer ]tracer correlations show distinct characteristics for each regime, which reflect the underlying transport processes. The UTLS climatology derived here shows many features of earlier climatologies. In addition, mixed air masses in the subtropics, identified by O3 ]CO correlations, show two characteristic modes in the tracer ]tracer space that are a result of mixed air masses in layers above and below the tropopause (TP). A thin layer of mixed air (1.2 km around the tropopause) is identified for all regions and seasons, where tracer gradients across the TP are largest. The most pronounced influence of mixing between the tropical transition layer and the subtropics was found in spring and summer in the region above 380 K potential temperature. The vertical extent of mixed air masses between UT and LS reaches up to 5 km above the TP. The tracer correlations and distributions in the UTLS derived here can serve as a reference for model and satellite data evaluation

Tilmes, S.↗
NASA NTRS2007

Representation-Independent Iteration of Sparse Data Arrays

An approach is defined that describes a method of iterating over massively large arrays containing sparse data using an approach that is implementation independent of how the contents of the sparse arrays are laid out in memory. What is unique and important here is the decoupling of the iteration over the sparse set of array elements from how they are internally represented in memory. This enables this approach to be backward compatible with existing schemes for representing sparse arrays as well as new approaches. What is novel here is a new approach for efficiently iterating over sparse arrays that is independent of the underlying memory layout representation of the array. A functional interface is defined for implementing sparse arrays in any modern programming language with a particular focus for the Chapel programming language. Examples are provided that show the translation of a loop that computes a matrix vector product into this representation for both the distributed and not-distributed cases. This work is directly applicable to NASA and its High Productivity Computing Systems (HPCS) program that JPL and our current program are engaged in. The goal of this program is to create powerful, scalable, and economically viable high-powered computer systems suitable for use in national security and industry by 2010. This is important to NASA for its computationally intensive requirements for analyzing and understanding the volumes of science data from our returned missions.

James, Mark↗
DOE OSTI2025

Data-Efficient Methods for Determining Flory–Huggins χ Parameters in Multicomponent Polymer Formulations

Polymer formulations are essential in diverse applications including personal care products, coatings, paints, adhesives, and plastic materials. Designing these formulations requires navigating large, complex design spaces, where phase and self-assembly behavior critically impact performance. The Flory–Huggins χ parameter, which quantifies segmental miscibility, is widely used to parametrize the excess free energy of mixing in formulation models. In this work, we introduce two data-efficient, top-down methods for estimating χ parameters using the Random Phase Approximation (RPA): (i) Boundary Nonlinear Regression (Boundary-NLR), which fits theoretical spinodal boundaries to experimental phase boundaries, and (ii) Surrogate Model Inverse Parameter Estimation (SMIPE), which uses a Gaussian Process Classifier to fit sparse phase maps via a surrogate model. Both methods allow rapid parametrization of polymer field-theoretic models without the need for additional experiments. We evaluate these approaches on data sets involving polymer–solvent–nonsolvent ternary mixtures and block copolymer–solvent systems, demonstrating their robustness to experimental noise and their relevance for real-world formulation design.

copolymers↗
DOE OSTI2024

Comparison of point cloud and image-based models for calorimeter fast simulation

Score based generative models are a new class of generative models that have been shown to accurately generate high dimensional calorimeter datasets. Recent advances in generative models have used images with 3D voxels to represent and model complex calorimeter showers. Point clouds, however, are likely a more natural representation of calorimeter showers, particularly in calorimeters with high granularity. Furthermore, point clouds preserve all of the information of the original simulation, more naturally deal with sparse datasets, and can be implemented with more compact models and data files. In this work, two state-of-the-art score based models are trained on the same set of calorimeter simulation and directly compared.

47 OTHER INSTRUMENTATION↗
DOE OSTI2021

PLANC: Parallel Low-rank Approximation with Nonnegativity Constraints

In this work, we consider the problem of low-rank approximation of massive dense nonnegative tensor data, for example, to discover latent patterns in video and imaging applications. As the size of data sets grows, single workstations are hitting bottlenecks in both computation time and available memory. We propose a distributed-memory parallel computing solution to handle massive data sets, loading the input data across the memories of multiple nodes, and performing efficient and scalable parallel algorithms to compute the low-rank approximation. We present a software package called Parallel Low-rank Approximation with Nonnegativity Constraints, which implements our solution and allows for extension in terms of data (dense or sparse, matrices or tensors of any order), algorithm (e.g., from multiplicative updating techniques to alternating direction method of multipliers), and architecture (we exploit GPUs to accelerate the computation in this work). We describe our parallel distributions and algorithms, which are careful to avoid unnecessary communication and computation, show how to extend the software to include new algorithms and/or constraints, and report efficiency and scalability results for both synthetic and real-world data sets.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2006

Flight Operations Analysis Tool

Flight Operations Analysis Tool (FLOAT) is a computer program that partly automates the process of assessing the benefits of planning spacecraft missions to incorporate various combinations of launch vehicles and payloads. Designed primarily for use by an experienced systems engineer, FLOAT makes it possible to perform a preliminary analysis of trade-offs and costs of a proposed mission in days, whereas previously, such an analysis typically lasted months. FLOAT surveys a variety of prior missions by querying data from authoritative NASA sources pertaining to 20 to 30 mission and interface parameters that define space missions. FLOAT provides automated, flexible means for comparing the parameters to determine compatibility or the lack thereof among payloads, spacecraft, and launch vehicles, and for displaying the results of such comparisons. Sparseness, typical of the data available for analysis, does not confound this software. FLOAT effects an iterative process that identifies modifications of parameters that could render compatible an otherwise incompatible mission set.

Easter, Robert↗
DOE OSTI2022

Geothermal Fault Zone and Fluid Imaging through Joint Airborne ZTEM and Ground MT Data Inversion Analysis

This project has aimed to achieve detailed electrical resistivity resolution at geothermal reservoir scales by combining airborne natural electromagnetic (EM) field surveying (ZTEM) with ground magnetotelluric (MT) measurements to approximate an airborne MT geophysical method. MT alone is relatively expensive and may have permitting challenges in sensitive areas. Airborne ZTEM field data contains only the magnetic field, requires a background assumption, and has been limited to relatively high frequencies, thus suffering uniqueness problems. Based on proto-type 2D simulations, ZTEM ambiguities may be reduced through formal incorporation with possibly sparse ground MT soundings, which we pursued in full 3D for this project. The methodology was tested at the high-temperature Roosevelt Hot Springs geothermal system, Utah, which was considered advantageous given the near total exposure of crystalline reservoir rocks across the project area. ZTEM and ground MT survey data were acquired in 2017, subcontracted to outside parties with which we have worked in the past. These included 80 remote-referenced tensor MT soundings over the Mineral Mountains and adjacent Roosevelt Hot Spring producing geothermal system. These MT stations abut later coverage of a similar number of MT stations taken for the Utah FORGE project providing excellent total data aperture to re-solve structure beneath both project areas better than either set alone. The airborne ZTEM survey covered 704 line kilometers in E-W flight lines with a 250 m line spacing. Although this survey was timed during a maintenance-related shutdown of power production at the Roosevelt Hot Springs, other noise sources difficult to identify but including two high-voltage state-scale transmission lines compromised the ZTEM survey badly leading to unusable responses. Thus, with DOE management concurrence, the project proceeded to emphasize inversion and interpretation of the joint SubTER-FORGE MT data sets with regard to the Roosevelt Hot Springs reservoir recharge and to deep heat sources for both it and the Utah FORGE EGS project area. We also investigated the joint ZTEM-MT sampling concept with data sets from the Eleven Mile Canyon prospect area donated by the U.S. Navy (A. Sabin, PoC). Inversion of the SubTER-FORGE MT data using the HexMT 3D finite element algorithm reveals a large, low-resistivity anomaly extending sub-vertically through the depth range of the crust beneath the western Mineral Mountains. The steep conductive zone connects in the lower crust to a more tabular conductor characteristic of much of the Great Basin that generally is ascribed to current mafic magmatic underplating, hybridization and fluid release. The location of the resolved anomaly relative to the recent (0.5-0.8 Ma) eruptive centers of the Mineral Mountains implicates it as remnants of the magma body which fed these centers. This structure appears to be currently feeding heat and fluids upward into the Roosevelt Hot Springs hydrothermal system, as well as heat laterally to the FORGE project area. Separate and joint inversion models were carried out for the donated Eleven Mile Canyon MT-ZTEM data set to demonstrate concept. ZTEM only inversion showed two main alteration zones in the western portion of the project area known from geological mapping. Joint inversion including an E-W profile of MT soundings sharpened these features considerably. It also resolved in much greater detail the graben related normal faulting structure of the central project area which lies at depths exceeding the sensitivity of ZTEM alone. The sparse number of MT da-ta relative to the ZTEM required upweighting the former by a factor of several, but an exact procedure awaits future research. Our final impression is that sparse MT data can improve resolution of the subsurface over that of ZTEM alone. However, well sampled MT data are to be preferred and offer the simplicity of interpreting just one data type, and possess the superior resolution capability coming with the electric field everywhere, and from their high bandwidth.

15 GEOTHERMAL ENERGY↗
DOE OSTI2022

A dictionary learning algorithm for compression and reconstruction of streaming data in preset order

There has been an emerging interest in developing and applying dictionary learning (DL) to process massive datasets in the last decade. Many of these efforts, however, focus on employing DL to compress and extract a set of important features from data, while considering restoring the original data from this set a secondary goal. On the other hand, although several methods are able to process streaming data by updating the dictionary incrementally as new snapshots pass by, most of those algorithms are designed for the setting where the snapshots are randomly drawn from a probability distribution. In this paper, we present a new DL approach to compress and denoise massive dataset in real time, in which the data are streamed through in a preset order (instances are videos and temporal experimental data), so at any time, we can only observe a biased sample set of the whole data. Here, our approach incrementally builds up the dictionary in a relatively simple manner: if the new snapshot is adequately explained by the current dictionary, we perform a sparse coding to find its sparse representation; otherwise, we add the new snapshot to the dictionary, with a Gram-Schmidt process to maintain the orthogonality. To compress and denoise noisy datasets, we apply the denoising to the snapshot directly before sparse coding, which deviates from traditional dictionary learning approach that achieves denoising via sparse coding. Compared to full-batch matrix decomposition methods, where the whole data is kept in memory, and other mini-batch approaches, where unbiased sampling is often assumed, our approach has minimal requirement in data sampling and storage: i) each snapshot is only seen once then discarded, and ii) the snapshots are drawn in a preset order, so can be highly biased. Through experiments on climate simulations and scanning transmission electron microscopy (STEM) data, we demonstrate that the proposed approach performs competitively to those methods in data reconstruction and denoising.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Particle hit clustering and identification using point set transformers in liquid argon time projection chambers

Liquid argon time projection chambers are often used in neutrino physics and dark-matter searches because of their high spatial resolution. The images generated by these detectors are extremely sparse, as the energy values detected by most of the detector are equal to 0, meaning that despite their high resolution, most of the detector is unused in a particular interaction. Instead of representing all of the empty detections, the interaction is usually stored as a sparse matrix, a list of detection locations paired with their energy values. Traditional machine learning methods that have been applied to particle reconstruction such as convolutional neural networks (CNNs), however, cannot operate over data stored in this way and therefore must have the matrix fully instantiated as a dense matrix. Operating on dense matrices requires a lot of memory and computation time, in contrast to directly operating on the sparse matrix. We propose a machine learning model using a point set neural network that operates over a sparse matrix, greatly improving both processing speed and accuracy over methods that instantiate the dense matrix, as well as over other methods that operate over sparse matrices. Compared to competing state-of-the-art methods, our method improves classification performance by 14%, segmentation performance by more than 22%, while taking 80% less time and using 66% less memory. Compared to state-of-the-art CNN methods, our method improves classification performance by more than 86%, segmentation performance by more than 71%, while reducing runtime by 91% and reducing memory usage by 61%.

calibration and fitting methods↗
DOE OSTI2019

Acoustic- and elastic-waveform inversion with total generalized p-variation regularization

Geophysical models usually contain both sharp interfaces and smooth variations, and it is difficult to accurately account for both of these two types of medium parameter variations using conventional full-waveform inversion methods. We develop a novel full-waveform inversion method for acoustic and elastic waves using a total generalized p-variation regularization scheme to address these challenging problems. We decompose the full-waveform inversion into two subproblems and solve these two minimization subproblems using an alternating-direction minimization strategy. One significant advantage of the total generalized p-variation regularization scheme is that it can simultaneously reconstruct sharp interfaces and smooth background variations of geophysical parameters. Such capability can also effectively suppress the noises in source-encoded inversion and sparse-data inversion, or inversion of noisy data. We demonstrate the advantages of our full-waveform inversion method using a checkerboard model, a modified elastic SEG/EAGE overthrust model, and a land field seismic data set. Our conclusions of synthetic and field seismic data demonstrate that our method reconstructs both smooth background variations and sharp interfaces of subsurface geophysical properties accurately, and provides a useful tool for accurate and reliable inversion of field seismic data.

58 GEOSCIENCES↗