Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Distribution”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

DeepAstroUDA: semi-supervised universal domain adaptation for cross-survey galaxy morphology classification and anomaly detection

Abstract Artificial intelligence methods show great promise in increasing the quality and speed of work with large astronomical datasets, but the high complexity of these methods leads to the extraction of dataset-specific, non-robust features. Therefore, such methods do not generalize well across multiple datasets. We present a universal domain adaptation method, DeepAstroUDA , as an approach to overcome this challenge. This algorithm performs semi-supervised domain adaptation (DA) and can be applied to datasets with different data distributions and class overlaps. Non-overlapping classes can be present in any of the two datasets (the labeled source domain, or the unlabeled target domain), and the method can even be used in the presence of unknown classes. We apply our method to three examples of galaxy morphology classification tasks of different complexities (three-class and ten-class problems), with anomaly detection: (1) datasets created after different numbers of observing years from a single survey (Legacy Survey of Space and Time mock data of one and ten years of observations); (2) data from different surveys (Sloan Digital Sky Survey (SDSS) and DECaLS); and (3) data from observing fields with different depths within one survey (wide field and Stripe 82 deep field of SDSS). For the first time, we demonstrate the successful use of DA between very discrepant observational datasets. DeepAstroUDA is capable of bridging the gap between two astronomical surveys, increasing classification accuracy in both domains (up to 40 % on the unlabeled data), and making model performance consistent across datasets. Furthermore, our method also performs well as an anomaly detection algorithm and successfully clusters unknown class samples even in the unlabeled target dataset.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Charged particle tracking in real-time using a full-mesh data delivery architecture and associative memory techniques

We present a flexible and scalable approach to address the challenges of charged particle track reconstruction in real-time event filters (Level-1 triggers) in collider physics experiments. The method described here is based on a full-mesh architecture for data distribution and relies on the Associative Memory approach to implement a pattern recognition algorithm that quickly identifies and organizes hits associated to trajectories of particles originating from particle collisions. We describe a successful implementation of a demonstration system composed of several innovative hardware and algorithmic elements. The implementation of a full-size system relies on the assumption that an Associative Memory device with the sufficient pattern density becomes available in the future, either through a dedicated ASIC or a modern FPGA. We demonstrate excellent performance in terms of track reconstruction efficiency, purity, momentum resolution, and processing time measured with data from a simulated LHC-like tracking detector.

47 OTHER INSTRUMENTATION↗
DOE OSTI2025

Physicochemical and Molecular Insights into the Boundary Layer and Free Troposphere Aerosol Interactions over the Southern Great Plains

Ambient aerosols’ vertical profiles are critical for evaluating the role of aerosols in atmospheric chemistry and radiative transfer, but limited data on these profiles hinders our ability to fully assess their impact on the Earth's radiative balance. Here, in this study, we investigated the size-, time-, and altitude resolved composition of individual particles and bulk molecular composition of particle samples collected by an uncrewed aerial system–ArcticShark over the Southern Great Plains. Single particle microanalysis shows that, the free tropospheric (FT) samples are dominated (56-66%) by carbonaceous sulfate particles, while boundary layer (BL) samples are dominated (57-74%) by carbonaceous particles. Back trajectory simulations suggest that FT particles are likely influenced by long-range transport and have undergone aqueous-phase processing. Conversely, in-situ size distribution data shows evidence of particle growth in the upper BL and just below the FT. This observation may indicate vertical transport of particles from an elevated aerosol layer in the FT, possibly linked to a new particle formation event. This observation is further supported by high resolution molecular composition data, which reveals particle volatility increasing with increasing size, which aligns with the growth event. This study aids in fundamental understanding of the compositional and molecular specificity of vertically resolved organic aerosols to provide insights into particle size evolution for future atmospheric models.

ArcticShark↗
DOE OSTI2025

SIDDA: SInkhorn Dynamic Domain Adaptation

Modern neural networks (NNs) often do not generalize well in the presence of a "covariate shift"; that is, in situations where the training and test data distributions differ, but the conditional distribution of classification labels remains unchanged. In such cases, NN generalization can be reduced to a problem of learning more domain-invariant features. Domain adaptation (DA) methods include a range of techniques aimed at achieving this; however, these methods have struggled with the need for extensive hyperparameter tuning, which then incurs significant computational costs. In this work, we introduce SIDDA, an out-of-the-box DA training algorithm built upon the Sinkhorn divergence, that can achieve effective domain alignment with minimal hyperparameter tuning and computational overhead. We demonstrate the efficacy of our method on multiple simulated and real datasets of varying complexity, including simple shapes, handwritten digits, and real astronomical observations. SIDDA is compatible with a variety of NN architectures, and it works particularly well in improving classification accuracy and model calibration when paired with equivariant neural networks (ENNs). We find that SIDDA enhances the generalization capabilities of NNs, achieving up to a ≈40% improvement in classification accuracy on unlabeled target data. We also study the efficacy of DA on ENNs with respect to the varying group orders of the dihedral group DN, and find that the model performance improves as the degree of equivariance increases. Finally, we find that SIDDA enhances model calibration on both source and target data--achieving over an order of magnitude improvement in the ECE and Brier score. SIDDA's versatility, combined with its automated approach to domain alignment, has the potential to advance multi-dataset studies by enabling the development of highly generalizable models.

Pandya, Sneh [Northeastern U.]↗
DOE OSTI2022

Building an Integrated Ecosystem of Computational and Observational Facilities to Accelerate Scientific Discovery

Future scientific discoveries will rely on flexible ecosystems that incorporate modern scientific instruments, high performance computing resources, parallel distributed data storage, and performant networks across multiple, independent facilities. In addition to connecting physical resources, such an ecosystem presents many challenges in logistics and accessibility, especially in orchestrating computations and experiments that span across leadership computing systems and experimental instruments. Past efforts have typically been application-specific or limited to interfaces for computing resources. This paper proposes a general framework for integrating computation resources and instrument operations, addressing challenges in code development/execution, data staging and collection, software stack, control mechanisms, resource authorization and governance, and hardware integration. We also describe a demonstration use case wherein a Bayesian optimization algorithm running on an edge computing resource guides a scanning probe microscope to autonomously and intelligently characterize a material sample. This science edge ecosystem framework will provide a blueprint for federating multi-institutional, disparate resources and orchestrating scientific workflows across them to enable next-generation discoveries.

Somnath, Suhas↗
DOE OSTI2022

Transitioning from File-Based HPC Workflows to Streaming Data Pipelines with openPMD and ADIOS2

This paper aims to create a transition path from file-based IO to streaming-based workflows for scientific applications in an HPC environment. By using the openPMP-api, traditional workflows limited by filesystem bottlenecks can be overcome and flexibly extended for in situ analysis. The openPMD-api is a library for the description of scientific data according to the Open Standard for Particle-Mesh Data (openPMD). Its approach towards recent challenges posed by hardware heterogeneity lies in the decoupling of data description in domain sciences, such as plasma physics simulations, from concrete implementations in hardware and IO. The streaming backend is provided by the ADIOS2 framework, developed at Oak Ridge National Laboratory. This paper surveys two openPMD-based loosely-coupled setups to demonstrate flexible applicability and to evaluate performance. In loose coupling, as opposed to tight coupling, two (or more) applications are executed separately, e.g. in individual MPI contexts, yet cooperate by exchanging data. This way, a streaming-based workflow allows for standalone codes instead of tightly-coupled plugins, using a unified streaming-aware API and leveraging high-speed communication infrastructure available in modern compute clusters for massive data exchange. We determine new challenges in resource allocation and in the need of strategies for a flexible data distribution, demonstrating their influence on efficiency and scaling on the Summit compute system. The presented setups show the potential for a more flexible use of compute resources brought by streaming IO as well as the ability to increase throughput by avoiding filesystem bottlenecks.

Poeschel, Franz↗
DOE OSTI2020

Sub-micrometer random-pyramid texturing of silicon solar wafers with excellent surface passivation and low reflectance

Select emerging solar cell technologies, ranging from very narrow screen-printed fingers to perovskite/silicon tandems, would benefit from a reduction of the height of the standard pyramidal texture of monocrystalline silicon wafers to below 1 μm. However, fabricating such small surface features usually compromises the requisite low reflectance and high passivation quality. Through systematic design of experiments, we demonstrate that conventional alkaline-based wet-chemical etching processes can be tuned to produce dense, spatially homogeneous, and uniformly sized sub-micrometer pyramids. Precise texture size can be further controlled by adding potassium silicate to the texturing solution to effectively slow the pyramid growth rate and promote nucleation. With pyramid size distribution data—extracted from multiple atomic force microscopy images across each wafer—we demonstrate 16 nanotextures with average pyramid sizes ranging from 62 to 512 nm, with the largest having only 0.3% of its pyramids taller than 1 μm. Six of the nanotextures exhibit light trapping and surface defect densities comparable to state-of-the-art microtextured wafers. In conclusion, the best nanotexture achieved an AM1.5G-weighted reflectance of 11.8% and a minority-carrier lifetime of 3.4 ms after passivation with amorphous silicon, demonstrating that smaller textures need not deteriorate solar cell performance.

14 SOLAR ENERGY↗
DOE OSTI2024

Systematic engineering for production of anti-aging sunscreen compound in Pseudomonas putida

Sunscreen has been used for thousands of years to protect skin from ultraviolet radiation. However, the use of modern commercial sunscreen containing oxybenzone, ZnO, and TiO 2 has raised concerns due to their negative effects on human health and the environment. In this study, we aim to establish an efficient microbial platform for production of shinorine, a UV light absorbing compound with anti-aging properties. First, we methodically selected an appropriate host for shinorine production by analyzing central carbon flux distribution data from prior studies alongside predictions from genome-scale metabolic models (GEMs). We enhanced shinorine productivity through CRISPRi-mediated downregulation and utilized shotgun proteomics to pinpoint potential competing pathways. Simultaneously, we improved the shinorine biosynthetic pathway by refining its design, optimizing promoter usage, and altering the strength of ribosome binding sites. Finally, we conducted amino acid feeding experiments under various conditions to identify the key limiting factors in shinorine production. The study combines meta-analysis of 13 C-metabolic flux analysis, GEMs, synthetic biology, CRISPRi-mediated gene downregulation, and omics analysis to improve shinorine production, demonstrating the potential of Pseudomonas putida KT2440 as platform for shinorine production.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

GPU-Accelerated Solution of the Bethe–Salpeter Equation for Large and Heterogeneous Systems

We present a massively parallel GPU-accelerated implementation of the Bethe–Salpeter equation (BSE) for the calculation of the vertical excitation energies (VEEs) and optical absorption spectra of condensed and molecular systems, starting from single-particle eigenvalues and eigenvectors obtained with density functional theory. The algorithms adopted here circumvent the slowly converging sums over empty and occupied states and the inversion of large dielectric matrices through a density matrix perturbation theory approach and a low-rank decomposition of the screened Coulomb interaction, respectively. Further computational savings are achieved by exploiting the nearsightedness of the density matrix of semiconductors and insulators to reduce the number of screened Coulomb integrals. We scale our calculations to thousands of GPUs with a hierarchical loop and data distribution strategy. The efficacy of our method is demonstrated by computing the VEEs of several spin defects in wide-band-gap materials, showing that supercells with up to 1000 atoms are necessary to obtain converged results. We discuss the validity of the common approximation that solves the BSE with truncated sums over empty and occupied states. In conclusion, we then apply our GW-BSE implementation to a diamond lattice with 1727 atoms to study the symmetry breaking of triplet states caused by the interaction of a point defect with an extended line defect.

Absorption spectra↗
DOE OSTI2026

Regularization via f -Divergence: An Application to Multi-Oxide Spectroscopic Analysis

In this paper, we explore the application of convolutional neural networks (CNNs) for predicting the chemical composition of complex geologic samples in a simulated Martian atmospheric environment. Specifically, we aim to characterize oxide weight percentages (wt.%) of rock samples analyzed by remote Laser-Induced Breakdown Spectroscopy (LIBS), framing the problem as a multi-target regression task . Neural networks trained on LIBS spectra are prone to overfitting due to high spectral complexity, limited labeled data, and measurement noise. While regularization is critical for improving generalization, common methods (e.g., ℓ 2 regularization) impose constraints not directly tied to data distribution properties. We propose a novel regularization method based on a specific ƒ-divergence induced by a graph-based estimator, designed to constrain the distributional discrepancy between predictions and targets. This regularizer serves a dual purpose: (a) mitigating overfitting by enforcing a constraint on the distributional difference between predictions and noisy targets, and (b) acting as an auxiliary loss that penalizes large divergences. To enable backpropagation, we develop a differentiable approximation of this particular ƒ-divergence, making the method feasible for neural networks. Experiments on ChemCam and SuperCam LIBS calibration spectra show that mathematical equation-divergence regularization outperforms or matches standard regularization methods (ℓ 1 , ℓ 2 , dropout) and the classical baseline, partial least squares (PLS). Combining ƒ-divergence regularization with standard regularization yields further performance gains, indicating that distributional regularization is useful in this context giving a promising direction for robust model training in planetary science applications. Source code is publicly available at Klein and Li (2025), https://doi.org/10.11578/dc.20250530.7.

58 GEOSCIENCES↗
DOE OSTI2023

Learning local equivariant representations for large-scale atomistic dynamics

Abstract A simultaneously accurate and computationally efficient parametrization of the potential energy surface of molecules and materials is a long-standing goal in the natural sciences. While atom-centered message passing neural networks (MPNNs) have shown remarkable accuracy, their information propagation has limited the accessible length-scales. Local methods, conversely, scale to large simulations but have suffered from inferior accuracy. This work introduces Allegro, a strictly local equivariant deep neural network interatomic potential architecture that simultaneously exhibits excellent accuracy and scalability. Allegro represents a many-body potential using iterated tensor products of learned equivariant representations without atom-centered message passing. Allegro obtains improvements over state-of-the-art methods on QM9 and revMD17. A single tensor product layer outperforms existing deep MPNNs and transformers on QM9. Furthermore, Allegro displays remarkable generalization to out-of-distribution data. Molecular simulations using Allegro recover structural and kinetic properties of an amorphous electrolyte in excellent agreement with ab-initio simulations. Finally, we demonstrate parallelization with a simulation of 100 million atoms.

74 ATOMIC AND MOLECULAR PHYSICS↗
DOE OSTI2022

Variance-aware weight quantization of multi-level resistive switching devices based on Pt/LaAlO3/SrTiO3 heterostructures

Abstract Resistive switching devices have been regarded as a promising candidate of multi-bit memristors for synaptic applications. The key functionality of the memristors is to realize multiple non-volatile conductance states with high precision. However, the variation of device conductance inevitably causes the state-overlap issue, limiting the number of available states. The insufficient number of states and the resultant inaccurate weight quantization are bottlenecks in developing practical memristors. Herein, we demonstrate a resistive switching device based on Pt/LaAlO 3 /SrTiO 3 (Pt/LAO/STO) heterostructures, which is suitable for multi-level memristive applications. By redistributing the surface oxygen vacancies, we precisely control the tunneling of two-dimensional electron gas (2DEG) through the ultrathin LAO barrier, achieving multiple and tunable conductance states (over 27) in a non-volatile way. To further improve the multi-level switching performance, we propose a variance-aware weight quantization (VAQ) method. Our simulation studies verify that the VAQ effectively reduces the state-overlap issue of the resistive switching device. We also find that the VAQ states can better represent the normal-like data distribution and, thus, significantly improve the computing accuracy of the device. Our results provide valuable insight into developing high-precision multi-bit memristors based on complex oxide heterostructures for neuromorphic applications.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2023

TAMM: Tensor algebra for many-body methods

Tensor algebra operations such as contractions in computational chemistry consume a significant fraction of the computing time on large-scale computing platforms. The widespread use of tensor contractions between large multi-dimensional tensors in describing electronic structure theory has motivated the development of multiple tensor algebra frameworks targeting heterogeneous computing platforms. In this paper, we present Tensor Algebra for Many-body Methods (TAMM), a framework for productive and performance-portable development of scalable computational chemistry methods. TAMM decouples the specification of the computation from the execution of these operations on available high-performance computing systems. With this design choice, the scientific application developers (domain scientists) can focus on the algorithmic requirements using the tensor algebra interface provided by TAMM, whereas high-performance computing developers can direct their attention to various optimizations on the underlying constructs, such as efficient data distribution, optimized scheduling algorithms, and efficient use of intra-node resources (e.g., graphics processing units). The modular structure of TAMM allows it to support different hardware architectures and incorporate new algorithmic advances. We describe the TAMM framework and our approach to the sustainable development of scalable ground- and excited-state electronic structure methods. We present case studies highlighting the ease of use, including the performance and productivity gains compared to other frameworks.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2026

Maximizing efficiency of dataset compression for machine learning potentials with information theory

Machine learning interatomic potentials (MLIPs) balance high accuracy and lower costs compared to density functional theory calculations, but their performance often depends on the size and diversity of training datasets. Large datasets improve model accuracy and generalization but are computationally expensive to produce and train on, while smaller datasets risk discarding rare but important atomic environments and compromising MLIP accuracy/reliability. Here, we develop an information-theoretical framework to quantify the efficiency of dataset compression methods and propose an algorithm that maximizes this efficiency. By framing atomistic dataset compression as an instance of the minimum set cover (MSC) problem over atom-centered environments, our method identifies the smallest subset of structures that contains as much information as possible from the original dataset while pruning redundant information. The approach is extensively demonstrated on the GAP-20 and TM23 datasets and validated on 64 varied datasets from the ColabFit repository. Across all cases, MSC consistently retains outliers, preserves dataset diversity, and reproduces the long-tail distributions of forces even at high compression rates, outperforming other subsampling methods. Furthermore, MLIPs trained on MSC-compressed datasets exhibit reduced error for out-of-distribution data even in low-data regimes. We explain these results using an outlier analysis and show that such quantitative conclusions could not be achieved with conventional dimensionality reduction methods. The algorithm is implemented in the open-source QUESTS package and can be used for several tasks in atomistic modeling, from data subsampling, outlier detection, and training improved MLIPs at a lower cost.

36 MATERIALS SCIENCE↗
DOE OSTI2021

Mind the gap: the power of combining photometric surveys with intensity mapping

The long wavelength modes lost to bright foregrounds in the interferometric 21-cm surveys can partially be recovered using a forward modeling approach that exploits the non-linear coupling between small and large scales induced by gravitational evolution. In this work, we build upon this approach by considering how adding external galaxy distribution data can help to fill in these modes. We consider supplementing the 21-cm data at two different redshifts with a spectroscopic sample (good radial resolution but low number density) loosely modeled on DESI-ELG at z=1 and a photometric sample (high number density but poor radial resolution) similar to LSST sample at z=1 and z=4 respectively. We find that both the galaxy samples are able to reconstruct the largest modes better than only using 21-cm data, with the spectroscopic sample performing significantly better than the photometric sample despite much lower number density. Here, we demonstrate the synergies between surveys by showing that the primordial initial density field is reconstructed better with the combination of surveys than using either of them individually. Methodologically, we also explore the importance of smoothing the density field when using bias models to forward model these tracers for reconstruction.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Measurement of the small-scale 3D Lyman- α forest power spectrum

Small-scale correlations measured in the Lyman-α (Lyα) forest encode information about the intergalactic medium and the primordial matter power spectrum. In this article, we present and implement a simple method to measure the 3-dimensional power spectrum, P 3D , of the Lyα forest at wavenumbers k corresponding to small, ~ Mpc scales. In order to estimate P 3D from sparsely and unevenly distributed data samples, we rely on averaging 1-dimensional Fourier Transforms, as previously carried out to estimate the 1-dimensional power spectrum of the Lyα forest, P 1D . Further, this methodology exhibits a very low computational cost. We confirm the validity of this approach through its application to Nyx cosmological hydrodynamical simulations. Subsequently, we apply our method to the eBOSS DR16 Lyα forest sample, providing as a proof of principle, a first P 3D measurement averaged over two redshift bins z = 2.2 and z = 2.4. This work highlights the potential for forthcoming P 3D measurements, from upcoming large spectroscopic surveys, to untangle degeneracies in the cosmological interpretation of P 1D .

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2020

Urbanization drives convergence in soil profile texture and carbon content

Urban development has driven extensive modification of the global landscape. This shift in land use and land cover alters ecological functioning, and thereby affects sustainable management agendas. Urbanization fundamentally reshapes the soils that underlay landscapes, and throughout the soil profile, extends impacts of urbanization far below the landscape surface. The impacts of urbanization on deeper soils that are beyond the reach of regular land management are largely unknown, and validation of general theories of convergent ecosystem properties are thwarted by a dearth of both level of measurement effort and the substantial heterogeneity in soils and urban landscapes. Here, we examined two soil properties with strong links to ecological functioning—carbon and mineral-fraction particle size—measured in urban soils, and compared them to their pre-urbanization conditions across a continental gradient encompassing global soil diversity. We hypothesized that urbanization drove convergence of soils properties from heterogeneous pre-urban conditions towards homogeneous urban conditions. Based on our observations, we confirm the hypothesis. Both soil carbon and particle size converged toward an intermediate value in the full data distribution, from pre-urban to urban conditions. These outcomes in urban soils were observed to uniformly be fine textured soils with overall lower carbon content. Although these properties are desirable for supporting urban infrastructure (e.g. buildings, pipes), they constrain the potential to render ecosystem services. Since soil profile texture and carbon content were convergent and observed across 11 cities, we suggest that these property profiles can be used as a universal urban soil profile to: 1) provide a clear prediction for how urbanization will shift soil properties from pre-urban conditions, 2) facilitate the adoption of commonly-accepted soil profiles for process models, and 3) offer a reference point to test against urban management strategies and how they impact soil resources.

54 ENVIRONMENTAL SCIENCES↗