SEARCH · Engineering Papers
Results for “sparse data”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
PyAMG: Algebraic Multigrid Solvers in Python
PyAMG is a Python package of algebraic multigrid (AMG) solvers and supporting tools for approximating the solution to large, sparse linear systems of algebraic equations, Ax = b, where A is an n × n sparse matrix. Sparse linear systems arise in a range of problems in science, from fluid flows to solid mechanics to data analysis. While the direct solvers available in SciPy’s sparse linear algebra package (scipy.sparse.linalg) are highly efficient, in many cases iterative methods are preferred due to overall complexity. However, the iterative methods in SciPy, such as CG and GMRES, often require an efficient preconditioner in order to achieve a lower complexity. Preconditioning is a powerful tool whereby the conditioning of the linear system and convergence rate of the iterative method are both dramatically improved. PyAMG constructs multigrid solvers for use as a preconditioner in this setting. A summary of multigrid and algebraic multigrid solvers can be found in Olson (2015a), in Olson (2015b), and in Falgout (2006); a detailed description can be found in Briggs et al. (2000) and Trottenberg et al. (2001).
Stoichiometrically-informed symbolic regression for extracting chemical reaction mechanisms from data
A data-driven computational method is introduced to extract chemical reaction mechanisms from time series chemical concentration data. It is realized through the use of dynamic symbolic regression in which a sparse analytical form for a dynamical system is discoverable from the underlying data. We specifically develop the stoichiometrically-informed symbolic regression (SISR) method to address a standing challenge in complex chemical reaction networks: given a time-series dataset of concentrations of several components, what is the mechanism and the associated rate constants? SISR finds the optimal mechanism, kinetic equations and rate constants by combining differential optimization with a genetic optimization approach that searches a symbolic space of possible reaction mechanisms. Use of SISR in several paradigmatic examples spanning linear and nonlinear reaction schemes results in excellent agreement between true and predicted mechanisms, including when the method is applied to noisy data. The advantages of a stoichiometrically-informed approach such as SISR to address reaction discovery is illustrated through comparison with the use of generic state-of-the-art data-driven approaches.
X-ray Computed Tomography Data of Dense Metallic Components
The data shared in here are X-ray computed tomography (XCT) scans of a hexagonal fuel nozzle in 3 sections with the Metrotom 800 system at the Manufacturing Demonstration Facility (MDF) at Oak Ridge National Laboratory. The data are used in the paper "Tomographic Sparse View Selection using the View Covariance Loss, by Lin et al. (doi:10.1109/TPAMI.2025.36000720), accepted to the international conference on computational imaging (ICCP 2025). Figures 4-7 in the paper describe the part/XCT scan. File name Descriptions: Bottom section: TCR- Single Channeled SRC L 2019-3-18 12-26-41.hdf5 Medium section: TCR- Single Channeled SRC M 2019-3-18 13-8-9.hdf5 Top section: TCR- Single Channeled SRC T 2019-3-18 13-45-39.hdf5 Each hdf5 file contains projection data, and all the relevant X-ray CT scan setting. The full list of included attributes: distance_unit: Units of all distances specified angle_unit : Units of the angles angles: Array of all angles used voxel_size_xy: Baseline recon (if any) has this voxel size in the in-plane direction voxel_size_z: Baseline recon (if any) has this voxel size in the cross-plane direction det_pixel_size_col: Size of the detector pixels in the column dimension det_pixel_size_row: Size of the detector pixels in the row dimension src_iso_dist: Source to iso-center distance iso_det_dist: Iso-center to detector distance det_angle: If the detector is rotated/tilted, this angle corresponds to that value det_row_offset: Center of rotation offset in the vertical direction det_col_offset: Center of rotation offset in the horizontal direction reconstruction: A baseline reconstruction stored as 3D array BHC params: Beam-hardening parameters - Van De Casteel Model - if it has been used to pre-process the projections We also provided a python script (hdf_io.py) that allows the user to read the relevant data from each hdf5 file.
Physics-Informed Generative Adversarial Networks for Stochastic Differential Equations
We developed a new class of physics-informed generative adversarial networks (PI-GANs) to solve forward, inverse, and mixed stochastic problems in a unified manner based on a limited number of scattered measurements. Unlike standard GANs relying solely on data for training, here we encoded into the architecture of GANs the governing physical laws in the form of stochastic differential equations (SDEs) using automatic differentiation. In particular, we applied Wasserstein GANs with gradient penalty (WGAN-GP) for its enhanced stability compared to vanilla GANs. We first tested WGAN-GP in approximating Gaussian processes of different correlation lengths based on data realizations collected from simultaneous reads at sparsely placed sensors. We obtained good approximation of the generated stochastic processes to the target ones even if there is a mismatch between the input noise dimensionality and the effective dimensionality of the target stochastic processes. We also studied the overfitting issue for both the discriminator and the generator, and we found that overfitting occurs also in the generator in addition to the discriminator as previously reported. Subsequently, we considered the solution of elliptic SDEs requiring approximations of three stochastic processes, namely the solution, the forcing, and the diffusion coefficient. Here again, we assumed data collected from simultaneous reads at a limited number of sensors for the multiple stochastic processes. Furthermore, three generators were used for the PI-GANs: two of them were feed forward deep neural networks (DNNs), while the other one was the neural network induced by the SDE. For the case where we have one group of data, we employed one feed forward DNN as the discriminator, while for the case of multiple groups of data we employed multiple discriminators in PI-GANs. We solved forward, inverse, and mixed problems without changing the framework of PI-GANs, obtaining both the means and the standard deviations of the stochastic solution and the diffusion coefficient in good agreement with benchmarks. In this work, we have demonstrated the effectiveness of PI-GANs in solving SDEs for about 120 dimensions. In principle, PI-GANs could tackle very high dimensional problems given more sensor data with low-polynomial growth in computational cost.
The Global Positioning System constellation as a space weather monitor. Comparison of electron measurements with Van Allen Probes data
Energetic electron observations in Earth's radiation belts are typically sparse, and multipoint studies often rely on serendipitous conjunctions. This paper establishes the scientific utility of the Combined X-ray Dosimeter (CXD), currently flown on 19 satellites in the Global Positioning System (GPS) constellation, by cross-calibrating energetic electron measurements against data from the Van Allen Probes. By breaking our cross calibration into two parts—one that removes any spectral assumptions from the CXD flux calculation and one that compares the energy spectra—we first validate the modeled instrument response functions, then the calculated electron fluxes. Unlike previous forward modeling of energetic electron spectra, we use a combination of four distributions that together capture a wide range of observed spectral shapes. Moreover, our two-step approach allowed us to identify, and correct for, small systematic offsets between block IIR and IIF satellites. Using the Magnetic Electron Ion Spectrometer and Relativistic Electron-Proton Telescope on Van Allen Probes as a “gold standard,” here we demonstrate that the CXD instruments are well understood. A robust statistical analysis shows that CXD and Van Allen Probes fluxes are similar and the measured fluxes from CXD are typically within a factor of 2 of Van Allen Probes at energies inline image4 MeV. Our team present data from 17 CXD-equipped GPS satellites covering the 2015 “St. Patrick's Day” geomagnetic storm to illustrate the scientific applications of such a high data density satellite constellation and therefore demonstrate that the GPS constellation is positioned to enable new insights in inner magnetospheric physics and space weather forecasting.
Discussion of CoSA: Clustering of Sparse Approximations
The purpose of this talk is to discuss the possible applications of CoSA (Clustering of Sparse Approximations) to the exploitation of HSI (HyperSpectral Imagery) data. CoSA is presented by Moody et al. in the Journal of Applied Remote Sensing (“Land cover classification in multispectral imagery using clustering of sparse approximations over learned feature dictionaries”, Vol. 8, 2014) and is based on machine learning techniques.
Thermal scattering law for ice based on neutron time-of-flight experiments carried out at the SEQUOIA spectrometer at the Oak Ridge National Laboratory
Precise estimation of neutron thermalization in moderators relies on high-fidelity thermal scattering cross-section data governed by the thermal scattering law (TSL). The Institut de Radioprotection et de Sûreté Nucléaire (IRSN) has been working on the development of improved TSL for light water ice. Many polymorphic phases of light water ice exist depending on the thermodynamic conditions. The most common type of ice at standard pressure and temperature below water freezing point (273.15 K) is ice-Ih. It is essential to have high-resolution experimental double differential data for developing and/or validating TSL for moderator materials.Existing experimental double-differential scattering data for ice-Ih are extremely sparse and of limited quality. New high-quality double-differential measurements for ice-Ih over multiple temperatures and incident neutron energies would directly support the validation and improvement of ice-Ih TSL models for criticality safety applications.Series of time-of-flight (TOF) inelastic neutron scattering experiments on ice-Ih at temperatures starting at 271 K and down to 6 K, have been carried out at the SEQUOIA spectrometer at the Spallation Neutron Source (SNS) at the Oak Ridge National Laboratory (ORNL), United States. The experiments have been performed for incident neutron energies, Ei= 11, 55, 160, 250, and 600 meV, to explore different excitation energies in the vibrational phonon spectrum. This paper presents the thermodynamic conditions and the details of the TOF measurements on ice-Ih and the derived phonon spectrum from the experimentally measured double differential data. A study of the variation of the phonon spectrum of ice-Ih as a function of temperature is highlighted, and a preliminary TSL evaluation is developed based on the experimental phonon spectrum.
A Climate Data Record (CDR) for the global terrestrial water budget: 1984–2010
Closing the terrestrial water budget is necessary to provide consistent estimates of budget components for understanding water resources and changes over time. Given the lack of in situ observations of budget components at anything but local scale, merging information from multiple data sources (e.g., in situ observation, satellite remote sensing, land surface model, and reanalysis) through data assimilation techniques that optimize the estimation of fluxes is a promising approach. Conditioned on the current limited data availability, a systematic method is developed to optimally combine multiple available data sources for precipitation ( P ), evapotranspiration (ET), runoff ( R ), and the total water storage change (TWSC) at 0.5° spatial resolution globally and to obtain water budget closure (i.e., to enforce P -ET- R -TWSC = 0) through a constrained Kalman filter (CKF) data assimilation technique under the assumption that the deviation from the ensemble mean of all data sources for the same budget variable is used as a proxy of the uncertainty in individual water budget variables. The resulting long-term (1984–2010), monthly 0.5° resolution global terrestrial water cycle Climate Data Record (CDR) data set is developed under the auspices of the National Aeronautics and Space Administration (NASA) Earth System Data Records (ESDRs) program. This data set serves to bridge the gap between sparsely gauged regions and the regions with sufficient in situ observations in investigating the temporal and spatial variability in the terrestrial hydrology at multiple scales. The CDR created in this study is validated against in situ measurements like river discharge from the Global Runoff Data Centre (GRDC) and the United States Geological Survey (USGS), and ET from FLUXNET. The data set is shown to be reliable and can serve the scientific community in understanding historical climate variability in water cycle fluxes and stores, benchmarking the current climate, and validating models.
Sparse recovery of undersampled intensity patterns for coherent diffraction imaging at high X-ray energies
Coherent X-ray photons with energies higher than 50 keV offer new possibilities for imaging nanoscale lattice distortions in bulk crystalline materials using Bragg peak phase retrieval methods. However, the compression of reciprocal space at high energies typically results in poorly resolved fringes on an area detector, rendering the diffraction data unsuitable for the three-dimensional reconstruction of compact crystals. To address this problem, we propose a method by which to recover fine fringe detail in the scattered intensity. This recovery is achieved in two steps: multiple undersampled measurements are made by in-plane sub-pixel motion of the area detector, then this data set is passed to a sparsity-based numerical solver that recovers fringe detail suitable for standard Bragg coherent diffraction imaging (BCDI) reconstruction methods of compact single crystals. The key insight of this paper is that sparsity in a BCDI data set can be enforced by recognising that the signal in the detector, though poorly resolved, is band-limited. This requires fewer in-plane detector translations for complete signal recovery, while adhering to information theory limits. Lastly, we use simulated BCDI data sets to demonstrate the approach, outline our sparse recovery strategy, and comment on future opportunities.
Latent Twins
Over the past decade, scientific machine learning has transformed the development of mathematical and computational frameworks for analyzing, modeling, and predicting complex systems. From inverse problems to numerical partial differential equations (PDEs), dynamical systems, and model reduction, these advances have pushed the boundaries of what can be simulated. Yet they have often progressed in parallel, with representation learning and algorithmic solution methods evolving largely as separate pipelines. With Latent Twins, we propose a unifying mathematical framework that creates a hidden surrogate in latent space for the underlying equations. Whereas digital twins mirror physical systems in the digital world, Latent Twins mirror mathematical systems in a learned latent space governed by operators. Through this lens, classical modeling, inversion, model reduction, and operator approximation all emerge as special cases of a single principle. We establish the fundamental approximation properties of Latent Twins for both ordinary differential equations (ODEs) and PDEs and demonstrate the framework across three representative settings: (i) canonical ODEs, capturing diverse dynamical regimes; (ii) a PDE benchmark using the shallow-water equations, contrasting Latent Twin simulations with deep operator network and forecasts with a four-dimensional variational method baseline; and (iii) a challenging real-data geopotential reanalysis dataset, reconstructing and forecasting from sparse, noisy observations. Latent Twins provide a compact, interpretable surrogate for solution operators that evaluate across arbitrary time gaps in a single-shot, while remaining compatible with scientific pipelines such as assimilation, control, and uncertainty quantification. Looking forward, this framework offers scalable, theory-grounded surrogates that bridge data-driven representation learning and classical scientific modeling across disciplines.
Improved microgrid resiliency through distributionally robust optimization under a policy-mode framework
Critical energy infrastructure are constantly under stress due to the ever increasing disruptions caused by wildfires, hurricanes, other weather related extreme events and cyber-attacks. Hence it becomes important to make critical infrastructure resilient to threats from such cyber-physical events. However, such events are hard to predict and numerous in nature and type and it becomes infeasible to make a system resilient to every possible such cyber-physical event. Such an approach can make the system operation overly conservative and impractical to operate. Furthermore, distributions of such events are hard to predict and historical data available on such events can be very sparse, making the problem even harder to solve. To deal with these issues, in this paper we present a policy-mode framework that enumerates and predicts the probability of various cyber-physical events and then a distributionally robust optimization (DRO) formulation that is robust to the sparsity of the available historical data. The proposed algorithm is illustrated on an islanded microgrid example: a modified IEEE 123-node feeder with distributed energy resources (DERs) and energy storage. Simulations are carried to validate the resiliency metrics under the sampled disruption events.
Knowledge-guided graph machine learning for spatially distributed prediction of daily discharge and nitrogen export dynamics
Spatially distributed prediction of streamflow and nitrogen export dynamics is essential for precision management of agricultural watersheds. While temporal deep learning models such as Long Short-Term Memory (LSTM) have shown strong performance at basin scales, their ability to generalize spatially is limited by insufficient representation of spatial dependencies and flow paths, particularly under data-scarce conditions. To address this gap, we propose HydroGraphNet, a knowledge-guided graph machine learning framework that integrates process-based knowledge and explicit spatial learning into temporal modeling. This framework incorporates directed graph topology to encode watershed connectivity and upstream inflows, with mass balance constraints to improve physical consistency. To enhance generalization in sparsely monitored regions, HydroGraphNet is pretrained on synthetic data generated by the SWAT+ (Soil and Water Assessment Tool Plus) model. We evaluated HydroGraphNet in the Upper Sangamon River Basin (44 HUC-12 subwatersheds, 2001–2020) against two LSTM baselines: a lumped basin-level model and a distributed variant. When benchmarked on SWAT+ simulations in pretraining, HydroGraphNet improved test NSEs by 8.9% (discharge) and 13.7% (NO₃–N load) in temporal extrapolation, and by 27.1% and 34.7% in spatial extrapolation, relative to the Lumped LSTM baseline. After fine-tuning with USGS monitoring data, the model achieved mean test NSE (KGE) scores of 0.768 (0.861) for discharge and 0.626 (0.664) for NO₃–N load, substantially outperforming baselines. Attribution analysis further highlighted the importance of upstream inflow representation and graph-based spatial learning in capturing cross-subwatershed dependencies. The model also reproduced seasonal hydrological and biogeochemical patterns consistent with known processes, demonstrating its robustness and process fidelity for spatially distributed prediction. Altogether, HydroGraphNet advances the integration of physical knowledge and spatially explicit learning in hydrological modeling, offering a generalizable framework for distributed modeling to support spatially targeted water quality management in data-scarce watersheds.
On a framework for generating PoD curves assisted by numerical simulations
The Probability of Detection (PoD) curve method has emerged as an important tool for the assessment of the performance of NDE techniques, a topic of particular interest to the nuclear industry where inspection qualification is very important. The conventional experimental means of generating PoD curves though, can be expensive, requiring large data sets (covering defects and test conditions), and equipment and operator time. Several methods of achieving faster estimates for PoD curves using physics-based modelling have been developed to address this problem. Numerical modelling techniques are also attractive, especially given the ever-increasing computational power available to scientists today. Here we develop procedures for obtaining PoD curves, assisted by numerical simulation and based on Bayesian statistics. Numerical simulations are performed using Finite Element analysis for factors that are assumed to be independent, random and normally distributed. PoD curves so generated are compared with experiments on austenitic stainless steel (SS) plates with artificially created notches. We examine issues affecting the PoD curve generation process including codes, standards, distribution of defect parameters and the choice of the noise threshold. We also study the assumption of normal distribution for signal response parameters and consider strategies for dealing with data that may be more complex or sparse to justify this. These topics are addressed and illustrated through the example case of generation of PoD curves for pulse-echo ultrasonic inspection of vertical surface-breaking cracks in SS plates.
Resonance and intercombination lines in Mg-like ions of atomic numbers Z = 13 – 92
While prominent lines of various Na-like ions have been measured with an accuracy of better than 100 ppm and corroborate equally accurate calculations, there have been remarkably large discrepancies between calculations for Mg-like ions of high atomic number. We present ab initio calculations using the multireference Moller-Plesset approach for Mg-like ions of atomic numbers Z = 13-92 and compare the results with other calculations of this isoelectronic sequence as well as with experimental data. Our results come very close to experiment (typically 100 ppm) over a wide range. Furthermore, data at high values of Z are sparse, which calls for further accurate measurements in this range where relativistic and QED effects are large.
Gene and transcript abundances of bacterial type III secretion systems from the rumen microbiome are correlated with methane yield in sheep
Background: Ruminants are important contributors to global methane emissions via microbial fermentation in their reticulo-rumens. This study is part of a larger program, characterising the rumen microbiomes of sheep which vary naturally in methane yield (g CH4/kg DM/day) and aims to defne diferences in microbial communities, and in gene and transcript abundances that can explain the animal methane phenotype. Methods: Rumen microbiome metagenomic and metatranscriptomic data were analysed by Gene Set Enrichment, sparse partial least squares regression and the Wilcoxon Rank Sum test to estimate correlations between specifc KEGG bacterial pathways/genes and high methane yield in sheep. KEGG genes enriched in high methane yield sheep were reassembled from raw reads and existing contigs and analysed by MEGAN to predict their phylogenetic origin. Protein coding sequences from Succinivibrio dextrinosolvens strains were analysed using Efective DB to predict bacterial type III secreted proteins. The efect of S. dextrinosolvens strain H5 growth on methane formation by rumen methanogens was explored using co-cultures. Results: Detailed analysis of the rumen microbiomes of high methane yield sheep shows that gene and transcript abundances of bacterial type III secretion system genes are positively correlated with methane yield in sheep. Most of the bacterial type III secretion system genes could not be assigned to a particular bacterial group, but several genes were afliated with the genus Succinivibrio, and searches of bacterial genome sequences found that strains of S. dextrinosolvens were part of a small group of rumen bacteria that encode this type of secretion system. In co-culture experiments, S. dextrinosolvens strain H5 showed a growth-enhancing efect on a methanogen belonging to the order Methanomassiliicoccales, and inhibition of a representative of the Methanobrevibacter gottschalkii clade. Conclusions: This is the frst report of bacterial type III secretion system genes being associated with high methane emissions in ruminants, and identifes these secretions systems as potential new targets for methane mitigation research. The effects of S. dextrinosolvens on the growth of rumen methanogens in co-cultures indicate that bacteriamethanogen interactions are important modulators of methane production in ruminant animals.
Lg-Wave Cross Correlation and Epicentral Double-Difference Location in and near China
In this paper, we perform epicentral relocations for a broad area using cross-correlation measurements made on Lg waves recorded at regional distances on a sparse station network. Using a two-step procedure (pairwise locations and cluster locations), we obtain final locations for 5623 events—3689 for all of China from 1985 to 2005 and 1934 for the Wenchuan area from May to August 2008. These high-quality locations comprise 20% of a starting catalog for all of China and 25% of a catalog for Wenchuan. Of the 1934 events located for Wenchuan, 1662 (86%) were newly detected. The final locations explain the residuals 89 times better than the catalog locations for all of China (3.7302–0.0417 s) and 32 times better than the catalog locations for Wenchuan (0.8413–0.0267 s). The average semimajor axes of the 95% confidence ellipses are 420 m for all of China and 370 m for Wenchuan. The average azimuthal gaps are 205° for all of China and 266° for Wenchuan. 98% of the station distances for all of China are over 200 km. The mean and maximum station distances are 898 and 2174 km. The robustness of our location estimates and various trade-offs and sensitivities is explored with different inversion parameters for the location, such as starting locations for iterative solutions and which singular values to include. Finally, our results provide order-of-magnitude improvements in locations for event clusters, using waveforms from a very sparse far-regional network for which data are openly available.
Variations in the Cloud-Base Height over the Central Himalayas during GVAX:Association with the Monsoon Rainfall
We present the measurements of cloud-base height variations over Aryabhatta Research Institute of Observational Science, Nainital (79.45 degrees E, 29.37 degrees N, 1958 m amsl) obtained from Vaisala Ceilometer, during the nearly year-long Ganges Valley Aerosol Experiment (GVAX). The cloud-base measurements are analysed in conjunction with collocated measurements of rainfall, to study the possible contributions from different cloud types to the observed monsoonal rainfall during June to September 2011. The summer monsoon of 2011 was a normal monsoon year with total accumulated rainfall of 1035.8 mm during June-September with a maximum during July (367.0 mm) and minimum during September (222.3 mm). The annual mean monsoon rainfall over Nainital is 1440 +/- 430 mm. The total rainfall measured during other months (October 2011-March 2012) was only 9% of that observed during the summer monsoon. The first cloud-base height varied from about 31 m above ground level (AGL) to a maximum of 7.6 km AGL during the summer monsoon period of 2011. It is found that about 70% of the total rain is observed only when the first cloud-base height varies between surface and 2 km AGL, indicating that most of the rainfall at high altitude stations such as Nainital is associated with stratiform low-level clouds. However, about 25% of the total rainfall is being contributed by clouds between 2 and 6 km. The occurrences of high-altitude cumulus clouds are observed to be only 2-4%. This study is an attempt to fill a major gap of measurements over the topographically complex and observationally sparse northern Indian region providing the evaluation data for atmospheric models and therefore, have implications towards the better predictions of monsoon rainfall and the weather components over this region.