SEARCH · Engineering Papers

Results for “sparse data set”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

All sources NASA NTRS DOE OSTI

DOE OSTI2024

Non-linear relationships between daily temperature extremes and US agricultural yields uncovered by global gridded meteorological datasets

Global agricultural commodity markets are highly integrated among major producers. Prices are driven by aggregate supply rather than what happens in individual countries in isolation. Furthermore, estimating the effects of weather-induced shocks on production, trade patterns and prices hence requires a globally representative weather data set. Recently, two data sets that provide daily or hourly records, GMFD and ERA5-Land, became available. Starting with the US, a data rich region, we formally test whether these global data sets are as good as more fine-scaled country-specific data in explaining yields and whether they estimate similar response functions. While GMFD and ERA5-Land have lower predictive skill for US corn and soybeans yields than the fine-scaled PRISM data, they still correctly uncover the underlying non-linear temperature relationship. All specifications using daily temperature extremes under any of the weather data sets outperform models that use a quadratic in average temperature. Correctly capturing the effect of daily extremes has a larger effect than the choice of weather data. In a second step, focusing on Sub Saharan Africa, a data sparse region, we confirm that GMFD and ERA5-Land have superior predictive power to CRU, a global weather data set previously employed for modeling climate effects in the region.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2022

Stellar kinematics of dwarf galaxies from multi-epoch spectroscopy: application to Triangulum II

ABSTRACT We present new MMT/Hectochelle spectroscopic measurements for 257 stars observed along the line of sight to the ultrafaint dwarf galaxy Triangulum II (Tri II). Combining results from previous Keck/DEIMOS spectroscopy, we obtain a sample that includes 16 likely members of Tri II, with up to 10 independent redshift measurements per star. To this multi-epoch kinematic data set, we apply methodology that we develop in order to infer binary orbital parameters from sparsely sampled radial velocity curves with as few as two epochs. For a previously identified (spatially unresolved) binary system in Tri II, we infer an orbital solution with period $296.0_{-3.3}^{+3.8} \rm ~ d$, semimajor axis $1.12^{+0.41}_{-0.24}\rm ~au$, and systemic velocity $-380.0 \pm 1.7 \rm ~km ~s^{-1}$ that we then use in the analysis of Tri II’s internal kinematics. Despite this improvement in the modelling of binary star systems, the current data remain insufficient to resolve the velocity dispersion of Tri II. We instead find a 95 per cent confidence upper limit of $\sigma _{v} \lesssim 3.4 \rm ~km~s^{-1}$.

Buttry, Rachel (ORCID:0000000155017609)↗

DOE OSTI2023

Recipes for when physics fails: recovering robust learning of physics informed neural networks

Abstract Physics-informed neural networks (PINNs) have been shown to be effective in solving partial differential equations by capturing the physics induced constraints as a part of the training loss function. This paper shows that a PINN can be sensitive to errors in training data and overfit itself in dynamically propagating these errors over the domain of the solution of the PDE. It also shows how physical regularizations based on continuity criteria and conservation laws fail to address this issue and rather introduce problems of their own causing the deep network to converge to a physics-obeying local minimum instead of the global minimum. We introduce Gaussian process (GP) based smoothing that recovers the performance of a PINN and promises a robust architecture against noise/errors in measurements. Additionally, we illustrate an inexpensive method of quantifying the evolution of uncertainty based on the variance estimation of GPs on boundary data. Robust PINN performance is also shown to be achievable by choice of sparse sets of inducing points based on sparsely induced GPs. We demonstrate the performance of our proposed methods and compare the results from existing benchmark models in literature for time-dependent Schrödinger and Burgers’ equations.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2018

Communication-avoiding optimization methods for distributed massive-scale sparse inverse covariance estimation

Across a variety of scientific disciplines, sparse inverse covariance estimation is a popular tool for capturing the underlying dependency relationships in multivariate data. Unfortunately, most estimators are not scalable enough to handle the sizes of modern high-dimensional data sets (often on the order of terabytes), and assume Gaussian samples. To address these deficiencies, we introduce HP-CONCORD, a highly scalable optimization method for estimating a sparse inverse covariance matrix based on a regularized pseudolikelihood framework, without assuming Gaussianity. Our parallel proximal gradient method uses a novel communication-avoiding linear algebra algorithm and runs across a multi-node cluster with up to 1k nodes (24k cores), achieving parallel scalability on problems with up to ≈819 billion parameters (1.28 million dimensions); even on a single node, HP-CONCORD demonstrates scalability, outperforming a state-of-the-art method. We also use HP-CONCORD to estimate the underlying dependency structure of the brain from fMRI data, and use the result to identify functional regions automatically. The results show good agreement with a clustering from the neuroscience literature.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2018

Communication-Avoiding Optimization Methods for Distributed Massive-Scale Sparse Inverse Covariance Estimation.

Across a variety of scientific disciplines, sparse inverse covariance estimation is a popular tool for capturing the underlying dependency relationships in multivariate data. Unfortunately, most estimators are not scalable enough to handle the sizes of modern high-dimensional data sets (often on the order of terabytes), and assume Gaussian samples. To address these deficiencies, we introduce HP-CONCORD, a highly scalable optimization method for estimating a sparse inverse covariance matrix based on a regularized pseudolikelihood framework, without assuming Gaussianity. Our parallel proximal gradient method uses a novel communication-avoiding linear algebra algorithm and runs across a multi-node cluster with up to 1k nodes (24k cores), achieving parallel scalability on problems with up to ~819 billion parameters (1.28 million dimensions); even on a single node, HP-CONCORD demonstrates scalability, outperforming a state-of-the-art method. We also use HP-CONCORD to estimate the underlying dependency structure of the brain from fMRI data, and use the result to identify functional regions automatically. The results show good agreement with a clustering from the neuroscience literature.

Koanantakool, Penporn↗

DOE OSTI2026

Prong Segmentation using Point Set Transformers in Multiple View Neutrino Detectors

NOvA is a long-baseline neutrino experiment studying neutrino oscillations by detecting neutrinos from the NuMI beam at Fermilab. Its physics analysis relies on accurate prong segmentation, which involves matching each hit to its source particle and identifying the particle type. This task has commonly been addressed using a combination of traditional clustering algorithms and convolutional neural networks (CNNs). However, NOvA’s detector design presents data as two sparse and decoupled 2D images (XZ and YZ views) rather than a native 3D representation, posing a significant challenge for traditional CNN-based models. In this talk, we propose a novel neural network based on the Point Set Transformer. By treating detector hits as sparse point clouds and implementing a cross-view attention mechanism, our model enables efficient information mixing between both views. Evaluated on NOvA simulated data, our model achieves superior accuracy while requiring significantly fewer computational resources compared to other models. Furthermore, the model demonstrates great performance when applied to Liquid Argon Time Projection Chamber (LArTPC) data, which shows its potential as a universal prong segmentation algorithm for multiple view neutrino detectors.

Liu, Jiaxi [UC, Irvine]↗

DOE OSTI2020

A Satellite-Derived Upper-Ocean Stratification Data Set for the Tropical North Atlantic With Potential Applications for Hurricane Intensity Prediction

Upper-ocean stratification strongly impacts vertical mixing and the heat flux between the ocean and atmosphere, especially under extreme conditions of tropical cyclones (TCs). Knowledge of prestorm stratification is important for accurate TC intensity prediction. In situ observations of the tropical ocean have significantly increased in the past decade. However, they are still too sparse to resolve ocean stratification variability in near-real time and on small spatial scales. Here, in this study, based on long-term observations and an ocean reanalysis data set from 2004–2017, we investigate the possibility of retrieving upper-ocean stratification from sea surface temperature (SST), sea surface salinity (SSS), and sea surface height (SSH) using a simple regression method. It is found that more than 90% of the mean seasonal cycle and about 30% to 80% of temperature and salinity stratification anomalies can be reconstructed using surface data from either observations or an ocean reanalysis. Simple regression can be used with satellite observations to create a high-resolution, near-real-time-gridded ocean stratification data set that successfully reproduces both the large and mesoscale variability of ocean stratification. When used in a simple expression for TC-induced SST cooling, the satellite-derived stratification shows improvements over an ocean analysis in terms of variance explained of SST cooling, offering promise as a near-real-time indicator of the ocean's impact on TC intensification.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2025

High-throughput spin-bath characterization of spin defects in semiconductors

Detailed knowledge of the local environments of spin defects in semiconductors, such as nitrogenvacancy (NV) centers in diamond or divacancies in silicon carbide, is crucial for optimizing control and entanglement protocols in quantum sensing and information applications. However, at present a direct experimental characterization of individual defect environments is not scalable, as conventional spin-bath measurements are time consuming and difficult to automate. Achieving high-throughput characterization requires short experiments to probe the spin bath. However, with fewer and noisier measurements, the inverse problem of recovering spin-bath properties from measured data becomes ill posed, with multiple spin baths having a high likelihood of yielding the same data. In this work, we present a set of computational tools to resolve the ill-posed inverse problem of recovering the atomic positions and hyperfine couplings of random nuclei surrounding spin defects from sparse, noisy experimental coherence data, which can be obtained in hours. Here, we use a trans-dimensional Bayesian approach that incorporates ab initio data to yield full posterior distributions over nuclear spin environments, enabling robust recovery from limited data. We also provide practical tools and guidelines to determine the limits of detectability for hyperfine couplings under specific dynamical decoupling sequences and sampling conditions. In addition, we demonstrate how the tools developed here, in combination with ab initio simulations of spin baths, can guide the design of efficient experimental protocols for application-specific high-throughput screening. To showcase the utility of our approach, we apply it to design fast dynamical decoupling experiments to characterize the spin baths often individual NV centers in diamond. While the primary focus is on accelerating spin-bath characterization of spin defects, this Bayesian approach also lays the foundation for digital-twin studies of spin defects, where a virtual model of the spin-defect system evolves in real time with ongoing experimental measurements. Together, the set of tools we designed and applied paves the way for scalable deployment of spin defects in semiconductors for quantum sensing and information applications.

Bayesian methods↗

DOE OSTI2024

Community detection in hypergraphs via mutual information maximization

Abstract The hypergraph community detection problem seeks to identify groups of related vertices in hypergraph data. We propose an information-theoretic hypergraph community detection algorithm which compresses the observed data in terms of community labels and community-edge intersections. This algorithm can also be viewed as maximum-likelihood inference in a degree-corrected microcanonical stochastic blockmodel. We perform the compression/inference step via simulated annealing. Unlike several recent algorithms based on canonical models, our microcanonical algorithm does not require inference of statistical parameters such as vertex degrees or pairwise group connection rates. Through synthetic experiments, we find that our algorithm succeeds down to recently-conjectured thresholds for sparse random hypergraphs. We also find competitive performance in cluster recovery tasks on several hypergraph data sets.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

Growth and Stability of Stratiform Carrollite (CuCo2S4) in the Tenke-Fungurume Ore District, Central African Copperbelt

Abstract Carrollite (CuCo2S4) is the main ore mineral of the critical battery metal cobalt, yet, surprisingly, detailed characterization of its thermodynamic properties and of its trace element contents remain sparse in the scientific literature. To fill these knowledge gaps, we generated the first thermodynamic data set for carrollite using well-characterized samples obtained from the stratiform sulfide mineralization style at Fungurume 88 deposit in the Democratic Republic of Congo. Detailed mineralogical investigation of these same samples suggests that carrollite has limited ability to incorporate other ‘sweetener’ elements (except Cu and Ni) by substitution mechanisms. Modeling using CHNOSZ 2.0.0. software and an integrated Cu-Co-S-O thermodynamic data set provides insights into the stability of carrollite relative to other Co-bearing minerals under physicochemical conditions relevant to the Central African Copperbelt. Further exploration of these models indicates that dropping redox potential in the mineralizing system is insufficient to explain the mineral-scale and deposit-scale zonation patterns commonly observed in Cu-Co deposits of the Central African Copperbelt. Instead, factors such as increasing pH, decreasing T, decreasing sulfur activity, or some combination of changing physicochemical parameters may need to be invoked to explain these zonation trends. At the Fungurume 88 deposit, the absence of linnaeite (Co3S4) suggests that ore precipitation occurred at temperatures cooler than 211 °C. From our pH-logfO2 diagrams, and relative to higher temperatures, such a lower T regime is marked by a greater degree of offset between the modeled Cu and Co solubility contours, thus facilitating a more differentiated and zoned distribution of these two metals at the deposit scale. Our models are offered to the geoscience community as exceptionally useful tools for modeling and visualizing the Cu-Co-S-O systematics and may be applied to a variety of geological and geometallurgical questions in the Central African Copperbelt and other deposits elsewhere.

Mineralogy↗

DOE OSTI2023

Machine learning-based inversion for acoustic impedance with large synthetic training data: Workflow and data characterization

Where wells are sparse or training data are difficult to label with high-quality wireline-derived impedance logs, machine learning (ML)-based inversion of acoustic impedance typically depends on small training data sets, leading to biased prediction. We have advanced a novel workflow that applies large synthetic seismic training data to reduce facies-related bias. Using a geologically realistic model as the truth model, we randomly select sparse seed wells to perform sequential Gaussian simulation (SGS) for impedance models of the same geometry and simulate facies variability. We implement random forest regression on 30 features extracted from the synthetic volume. We observe that more seed wells tend to reduce facies-induced bias by sampling more types of facies, resulting in a better prediction. We then focus on the responses of SGS models to facies changes, the number of seed wells necessary for a useful synthetic model, and how much a synthetic model can help ML-based inversion. Here, we observe that the SGS synthetic training model outperforms well-direct training in general. For modeled clastic shore-zone systems in Miocene Gulf of Mexico, two or more seed wells are necessary for a significant reduction of root-mean-square error and outliners, and improvement of facies imaging. In a field-data test, we apply a similar workflow to quantitatively predict acoustic impedance, which is then converted to a sand-volume map at a high-frequency sequence (10–100 m), revealing detailed facies and sandstone patterns. Such results are valuable in many geologic and engineering applications, such as hydrocarbon and CO 2 reservoir prospecting, reserve estimation, simulation, etc.

3D seismic↗

DOE OSTI2017

A Work-Efficient Parallel Sparse Matrix-Sparse Vector Multiplication Algorithm

We design and develop a work-efficient multithreaded algorithm for sparse matrix-sparse vector multiplication (SpMSpV) where the matrix, the input vector, and the output vector are all sparse. SpMSpV is an important primitive in the emerging GraphBLAS standard and is the workhorse of many graph algorithms including breadth-first search, bipartite graph matching, and maximal independent set. As thread counts increase, existing multithreaded SpMSpV algorithms can spend more time accessing the sparse matrix data structure than doing arithmetic. Our shared-memory parallel SpMSpV algorithm is work efficient in the sense that its total work is proportional to the number of arithmetic operations required. The key insight is to avoid each thread individually scan the list of matrix columns. Our algorithm is simple to implement and operates on existing column-based sparse matrix formats. It performs well on diverse matrices and vectors with heterogeneous sparsity patterns. A high-performance implementation of the algorithm attains up to 15x speedup on a 24-core Intel Ivy Bridge processor and up to 49x speedup on a 64-core Intel KNL manycore processor. In contrast to implementations of existing algorithms, the performance of our algorithm is sustained on a variety of different input types include matrices representing scale-free and high-diameter graphs.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

Numerical characterization of support recovery in sparse regression with correlated design

Sparse regression is employed in diverse scientific settings as a feature selection method. A pervasive aspect of scientific data is the presence of correlations between predictive features. These correlations hamper both feature selection and estimation and jeopardize conclusions drawn from estimated models. On the other hand, theoretical results on sparsity-inducing regularized regression have largely addressed conditions for selection consistency via asymptotics, and disregard the problem of model selection, whereby regularization parameters are chosen. In this numerical study, we address these issues through exhaustive characterization of the performance of several regression estimators, coupled with a range of model selection strategies. These estimators and selection criteria were examined across correlated regression problems with varying degrees of signal to noise, distributions of non-zero model coefficients, and model sparsity. Our results reveal a fundamental tradeoff between false positive and false negative control in all regression estimators and model selection criteria examined. Additionally, we numerically explore a transition point modulated by the signal-to-noise ratio and spectral properties of the design covariance matrix at which the selection accuracy of all considered algorithms degrades. Overall, we find that SCAD coupled with BIC or empirical Bayes model selection performs the best feature selection across the regression problems considered.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2016

SU-G-TeP1-09: Modality-Specific Dose Gradient Modeling for Prostate IMRT Using Spherical Distance Maps of PTV and Isodose Contours

Purpose: Overlapping volume histogram (OVH) and distance-to-target histogram (DTH) calculations rely on the assumption that dose gradients are symmetric with respect to primary target volume (PTV) expansion and minimum distance to PTV surface, respectively. It is desirable to lift this assumption and instead account for achievable modality-specific dose gradients (MSDG) for a given PTV shape. Methods: From a library of 96 prostate 7-beam IMRT plans, we computed spherical distance maps (SDMs) for PTVs and 3 iso-dose contours. Each SDM contains the minimum distances between plan isocenter and the object’s surface for a fixed set of azimuthal and polar angles. We performed principal component analysis (PCA)-based missing data recovery with PTV SDM as input and a single iso-dose contour SDM as output. Repeating this process for the set of iso-dose contours sparsely reconstructed the MSDG for a given PTV. DVH points were computed from the MSDG for bladder and rectum (OARs) as a natural way of casting patient-specific geometric information into modality-specific dose space (vs. OVH and DTH where data is purely geometric). For comparison, we implemented a cumulative (c)DTH-based prediction algorithm, and produced DVH for both OARs separately. We then computed linear regressions between each method and the DVH of the original patient plan for three different OAR DVH points. Results: R-squared for MSD-Gcomputed DVH vs. original plan DVH at V90%, V80% and V60% of max dose were 0.86, 0.83, and 0.78, for bladder and 0.36, 0.52, and 0.55 for rectum, respectively; R-squared for cDTH-based predicted DVH vs. original plan DVH were 0.67, 0.81, and 0.83, for bladder and 0.44, 0.50, and 0.51, for rectum, respectively. Conclusion: By simply modeling the MSDG about a given PTV, we are able to reproduce DVHs consistent with a validated cDTH-based DVH prediction. Regression analysis suggests MSDG could further improve predictions.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2021

Discrete-Direct Model Calibration and Uncertainty Propagation Method Confirmed on Multi-Parameter Plasticity Model Calibrated to Sparse Random Field Data

A discrete direct (DD) model calibration and uncertainty propagation approach is explained and demonstrated on a 4-parameter Johnson-Cook (J-C) strain-rate dependent material strength model for an aluminum alloy. The methodology's performance is characterized in many trials involving four random realizations of strain-rate dependent material-test data curves per trial, drawn from a large synthetic population. The J-C model is calibrated to particular combinations of the data curves to obtain calibration parameter sets which are then propagated to “Can Crush” structural model predictions to produce samples of predicted response variability. These are processed with appropriate sparse-sample uncertainty quantification (UQ) methods to estimate various statistics of response with an appropriate level of conservatism. This is tested on 16 output quantities (von Mises stresses and equivalent plastic strains) and it is shown that important statistics of the true variabilities of the 16 quantities are bounded with a high success rate that is reasonably predictable and controllable. The DD approach has several advantages over other calibration-UQ approaches like Bayesian inference for capturing and utilizing the information obtained from typically small numbers of replicate experiments in model calibration situations—especially when sparse replicate functional data are involved like force–displacement curves from material tests. The DD methodology is straightforward and efficient for calibration and propagation problems involving aleatory and epistemic uncertainties in calibration experiments, models, and procedures.

42 ENGINEERING↗

DOE OSTI2016

SAMBA: Sparse Approximation of Moment-Based Arbitrary Polynomial Chaos

A new arbitrary Polynomial Chaos (aPC) method is presented for moderately high-dimensional problems characterised by limited input data availability. The proposed methodology improves the algorithm of aPC and extends the method, that was previously only introduced as tensor product expansion, to moderately high-dimensional stochastic problems. The fundamental idea of aPC is to use the statistical moments of the input random variables to develop the polynomial chaos expansion. This approach provides the possibility to propagate continuous or discrete probability density functions and also histograms (data sets) as long as their moments exist, are finite and the determinant of the moment matrix is strictly positive. For cases with limited data availability, this approach avoids bias and fitting errors caused by wrong assumptions. In this work, an alternative way to calculate the aPC is suggested, which provides the optimal polynomials, Gaussian quadrature collocation points and weights from the moments using only a handful of matrix operations on the Hankel matrix of moments. It can therefore be implemented without requiring prior knowledge about statistical data analysis or a detailed understanding of the mathematics of polynomial chaos expansions. The extension to more input variables suggested in this work, is an anisotropic and adaptive version of Smolyak's algorithm that is solely based on the moments of the input probability distributions. It is referred to as SAMBA (PC), which is short for Sparse Approximation of Moment-Based Arbitrary Polynomial Chaos. It is illustrated that for moderately high-dimensional problems (up to 20 different input variables or histograms) SAMBA can significantly simplify the calculation of sparse Gaussian quadrature rules. SAMBA's efficiency for multivariate functions with regard to data availability is further demonstrated by analysing higher order convergence and accuracy for a set of nonlinear test functions with 2, 5 and 10 different input distributions or histograms.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

DOE OSTI2023

Ensemble Estimation of Historical Evapotranspiration for the Conterminous U.S.

Abstract Evapotranspiration (ET) is the largest component of the water budget, accounting for the majority of the water available from precipitation. ET is challenging to quantify because of the uncertainties associated with the many ET equations currently in use, and because observations of ET are uncertain and sparse. In this study, we combine information provided by available ET data and equations to produce a new monthly data set for ET for the conterminous U.S. (CONUS). These maps are produced from 1895 to 2018 at an 800 m spatial scale, marking a finer resolution than currently available products over this time period. In our approach, the relative performance of a suite of ET equations is assessed using water balance, flux tower, and remotely sensed ET estimates. At the observation locations, we use error distributions to quantify relative weights for the equations and use these in a modified Bayesian model averaging weighted ensemble approach. The relative weights are spatially generalized using a random forest regression, which is applied to wall‐to‐wall explanatory variable maps to generate CONUS‐wide relative weight maps and ensemble estimates. We assess the performance of the ensemble using a reserved subset of the observations and compare this performance against other national‐scale map products for historical to modern ET. The ensemble ET maps are shown to provide an improved accuracy over the alternative comparison products. These ET maps could be useful for a variety of hydrologic modeling and assessment applications that benefit from a long record, such as the study of periods of water scarcity through time.

Environmental Sciences & Ecology↗

DOE OSTI2024

Multi-Level Structural Damage Characterization Using Sparse Acoustic Sensor Networks and Knowledge Transferred Deep Learning

Standard structural health monitoring techniques face well-known difficulties for comprehensive defect diagnosis in real-world structures that have structural, material, or geometric complexity. This motivates the exploration of machine-learning-based structural health monitoring methods in complex structures. However, creating sufficient training data sets with various defects is an ongoing challenge for data-driven machine (deep) learning algorithms. The ability to transfer the knowledge of a trained neural network from one component to another or to other sections of the same component would drastically reduce the required training data set. Also, it would facilitate computationally inexpensive machine learning based inspection systems. In this work, a machine-learning-based multi-level damage characterization is demonstrated with the ability to transfer trained knowledge within the sparse sensor network. A novel network spatial assistance and an adaptive convolution technique are proposed for efficient knowledge transfer within the deep learning algorithm. Proposed structural health monitoring method is experimentally evaluated on an aluminum plate with artificially induced defects. It was observed that the method improves the performance of knowledge transferred damage characterization by 50% during localization and 24% during severity assessment. Further, experiments using time windows with and without multiple edge reflections are studied. Results reveal that multiply scattered waves contain rich and deterministic defect signatures that can be mined using deep learning neural networks, improving the accuracy of both identification and quantification. In the case of a fixed sensor network, using multiply scattered waves shows 100% prediction accuracy at all levels of damage characterization.

36 MATERIALS SCIENCE↗