SEARCH · Engineering Papers

Results for “Missing data problem”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

All sources NASA NTRS DOE OSTI

DOE OSTI2026

Merged Aerosol Value-Added Product Report

The Merged Aerosol Value-Added Product (VAP) simplifies scientists’ use of Atmospheric Radiation Measurement (ARM) User Facility aerosol data by performing several tedious, time-consuming tasks for the users. First, the VAP identifies the best data available when multiple datastreams exist for a single geophysical quantity so that ARM users do not have to research this for themselves. Second, the VAP consolidates multiple ARM aerosol datastreams into a single file for ARM data users so that they do not have to download, open, and read multiple files for their analysis. Next, the VAP transforms all measurements onto a common one-hour timestamp. The one-hour resolution matches the time resolution of the slowest instrument. Instruments with faster sampling rates than one measurement per hour are averaged over the time interval. Finally, the VAP reads the QA/QC variables and marks data with known issues as missing, so that users do not have to spend excessive time cleaning data. This includes incorporating Data Quality Reports (DQRs) that exist at the time when the VAP data is generated. DQRs are reports filed by instrument mentors or data users that indicate a problem with the output data of individual instruments.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2021

Challenging problems of quality assurance and quality control (QA/QC) of meteorological time series data

Abstract Representativeness and quality of collected meteorological data impact accuracy and precision of climate, hydrological, and biogeochemical analyses and predictions. We developed a comprehensive Quality Assurance (QA) and Quality Control (QC) statistical framework, consisting of three major phases: Phase I—Preliminary data exploration, i.e., processing of raw datasets, with the challenging problems of time formatting and combining datasets of different lengths and different time intervals; Phase II—QA of the datasets, including detecting and flagging of duplicates, outliers, and extreme data; and Phase III—the development of time series of a desired frequency, imputation of missing values, visualization and a final statistical summary. The paper includes two use cases based on the time series data collected at the Billy Barr meteorological station (East River Watershed, Colorado), and the Barro Colorado Island (BCI, Panama) meteorological station. The developed statistical framework is suitable for both real-time and post-data-collection QA/QC analysis of meteorological datasets.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2020

White Paper to Justify the Use of Strip Effluent in the Defense Waste Processing Facility During Slurry Mix Evaporator Processing

At the request of Savannah River Remediation (SRR), a white paper was written to assess whether it would be acceptable to process Strip Effluent (SE) in both the Sludge Receipt and Adjustment Tank (SRAT) and Slurry Mix Evaporator (SME) in the Defense Waste Processing Facility (DWPF) for the nitric-glycolic acid flowsheet. Savannah River National Laboratory (SRNL) experimental data and DWPF Sludge Batch 9 (SB9) process data was reviewed, looking at batches which included Precipitate Reactor Feed Tank (PRFT) and/or Strip Effluent Feed Tank (SEFT) feeds that led to long processing times. These batches were reviewed looking for processing problems such as melter feed trips, foamovers, heating rod or steam coil fouling, missed Reduction/Oxidation (REDOX) targets, and other process anomalies.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

DOE OSTI2025

Missing baryons recovered: A measurement of the gas fraction in galaxies and groups with the kinematic Sunyaev-Zel’dovich effect and CMB lensing

We present new constraints on the halo masses and matter density profiles of DESI galaxy groups by cross-correlating samples of luminous red galaxies (LRGs) and bright galaxy survey (BGS) galaxies with the publicly available CMB lensing convergence map from ACT DR6. This provides an independent, lensing-based calibration of halo masses, complementary to methods relying on clustering or dynamics. We derive constraints on the mean halo mass for three DESI-selected samples, finding log⁡(𝑀 halo /(𝑀 ⊙ /ℎ)) ≈ 13.18, 13.03 and 13.02 for the main LRG, extended LRG, and BGS samples, respectively. Using a halo model approach, we also compare the projected galaxy-matter density profiles with previously reported gas profiles inferred from measurements of the kinematic Sunyaev-Zel’dovich (kSZ) effect. This work addresses one of the key uncertainties in interpreting kSZ signals—the unknown host halo mass distribution—by providing an independent and consistent mass calibration. The agreement between the gas and total mass profiles at large aperture suggests that sufficiently far from the group center (2–3 virial radii), we recover all the baryons, offering a resolution to the missing baryon problem. We further study the cumulative gas fractions for all galaxies as well as for the most massive galaxy groups in the sample [log⁡(𝑀 halo /(𝑀 ⊙ /ℎ)) ≈ 13.5], finding values that are physically sensible and in agreement with previous findings using kSZ and x-ray data: compared to the TNG300 simulation, the observed gas fractions are systematically lower at fixed radius by ≳ 4⁢𝜎, providing compelling, independent evidence for stronger baryonic feedback in the real Universe. These findings highlight the power of combining CMB lensing with galaxy surveys to probe the interplay between baryons and dark matter in group-sized halos.

Hadzhiyska, Boryana [Lawrence Berkeley National La↗

DOE OSTI2026

Multi-Modal Bayesian Neural Network Surrogates with Conjugate Last-Layer Estimation

As data collection and simulation capabilities advance, multi-modal learning, the task of learning from multiple modalities and sources of data, is becoming an increasingly important area of research. Surrogate models that learn from data of multiple auxiliary modalities to support the modeling of a highly expensive quantity of interest have the potential to aid outer loop applications such as optimization, inverse problems, or sensitivity analyses when multi-modal data are available. We develop two multi-modal Bayesian neural network surrogate models and leverage conditionally conjugate distributions in the last layer to estimate model parameters using stochastic variational inference (SVI). We provide a method to perform this conjugate SVI estimation in the presence of partially missing observations. Here, we demonstrate improved prediction accuracy and uncertainty quantification compared to unimodal surrogate models for both scalar and time series data.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

Search for Nearly Mass-Degenerate Higgsinos Using Low-Momentum Mildly Displaced Tracks in 𝑝⁢𝑝 Collisions at $\sqrt{s}$ = 13 TeV with the ATLAS Detector

Higgsinos with masses near the electroweak scale can solve the hierarchy problem and provide a dark matter candidate, while detecting them at the LHC remains challenging if their mass splitting is 𝒪⁡(1 GeV). This Letter presents a novel search for nearly mass-degenerate Higgsinos in events with an energetic jet, missing transverse momentum, and a low-momentum track with a significant transverse impact parameter using 140 fb −1 of proton-proton collision data at $\sqrt{s}$ = 13 TeV collected by the ATLAS experiment. For the first time since LEP, a range of mass splittings between the lightest charged and neutral Higgsinos from 0.3 to 0.9 GeV is excluded at 95% confidence level, with a maximum reach of approximately 170 GeV in the Higgsino mass.

Hadron colliders↗

DOE OSTI2021

Performance Study of Distance-Weighting Approach with Loopy Sum-Product Algorithm for Multi-Object Tracking in Clutter

In this paper, we explore the performance of the distance-weighting probabilistic data association (DWPDA) approach in conjunction with the loopy sum-product algorithm (LSPA) for tracking multiple objects in clutter. First, we discuss the problem of data association (DA), which is to infer the correspondence between targets and measurements. DA plays an important role when tracking multiple targets using measurements of uncertain origin. Second, we describe three methods of data association: probabilistic data association (PDA), joint probabilistic data association (JPDA), and LSPA. We then apply these three DA methods for tracking multiple crossing targets in cluttered environments, e.g., radar detection with false alarms and missed detections. We are interested in two performance metrics: tracking accuracy and computation time. LSPA is known to be superior to PDA in terms of the former and to dominate JPDA in terms of the latter. Last, we consider an additional DA method that is a modification of PDA by incorporating a weighting scheme based on distances between position estimates and measurements. This distance-weighting approach, when combined with PDA, has been shown to enhance the tracking accuracy of PDA without significant change in the computation burden. Since PDA constitutes a crucial building block of LSPA, we hypothesize that DWPDA, when integrated with LSPA, would perform better under the two performance metrics above. Contrary to expectations, the distance-weighting approach does not enhance the performance of LSPA, whether in terms of tracking accuracy or computation time.

47 OTHER INSTRUMENTATION↗

DOE OSTI2023

Peri-Net-Pro: the neural processes with quantified uncertainty for crack patterns

Abstract This paper develops a deep learning tool based on neural processes (NPs) called the Peri-Net-Pro, to predict the crack patterns in a moving disk and classifies them according to the classification modes with quantified uncertainties. In particular, image classification and regression studies are conducted by means of convolutional neural networks (CNNs) and NPs. First, the amount and quality of the data are enhanced by using peridynamics to theoretically compensate for the problems of the finite element method (FEM) in generating crack pattern images. Second, case studies are conducted with the prototype microelastic brittle (PMB), linear peridynamic solid (LPS), and viscoelastic solid (VES) models obtained by using the peridynamic theory. The case studies are performed to classify the images by using CNNs and determine the suitability of the PMB, LBS, and VES models. Finally, a regression analysis is performed on the crack pattern images with NPs to predict the crack patterns. The regression analysis results confirm that the variance decreases when the number of epochs increases by using the NPs. The training results gradually improve, and the variance ranges decrease to less than 0.035. The main finding of this study is that the NPs enable accurate predictions, even with missing or insufficient training data. The results demonstrate that if the context points are set to the 10th, 100th, 300th, and 784th, the training information is deliberately omitted for the context points of the 10th, 100th, and 300th, and the predictions are different when the context points are significantly lower. However, the comparison of the results of the 100th and 784th context points shows that the predicted results are similar because of the Gaussian processes in the NPs. Therefore, if the NPs are employed for training, the missing information of the training data can be supplemented to predict the results.

Mathematics↗

DOE OSTI2023

Suppressing Quantum Circuit Errors Due to System Variability

We present a quantum circuit optimization technique that takes into account the variability in error rates that is inherent across present-day noisy quantum computing platforms. This method can be run after qubit routing or postcompilation and consists of computing isomorphic subgraphs to input circuits and scoring each using heuristic cost functions derived from system calibration data. Using an independent standard algorithmic test suite, we show that it is possible to recover on average nearly 40% of missing fidelity using better qubit selection via efficient to compute cost functions. We demonstrate additional performance gains by considering qubit placement over multiple quantum processors. The overhead from these tools is minimal with respect to other compilation steps, such as qubit routing, as the number of qubits increases. As such, our method can be used to find qubit mappings for problems at the scale of quantum advantage and beyond.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

DOE OSTI2026

ELVES-Dwarf. I. Satellite Systems of Eight Isolated Dwarf Galaxies in the Local Volume

The satellite populations of Milky Way (MW)–mass systems have been extensively studied, significantly advancing our understanding of galaxy formation and dark matter physics. In contrast, the satellites of lower-mass dwarf galaxies remain largely unexplored, despite hierarchical structure formation predicting that dwarf galaxies should host their own satellites. We present the first results of the ELVES-Dwarf survey, which aims to statistically characterize the satellite populations of isolated dwarf galaxies in the Local Volume (4 < D < 10 Mpc). We identify satellite candidates in integrated light using Legacy Surveys data and achieve completeness down to M g ≈ −9 mag. We then confirm the association of satellite candidates with host galaxies using surface-brightness fluctuation distances measured from Hyper Suprime-Cam data. We surveyed eight isolated dwarf galaxies with stellar masses ranging from sub-Small Magellanic Cloud to Large Magellanic Cloud scales $(10^{7.8} < M^{\textrm{host}}_{\star} < 10^{9.5} M_⊙)$, and confirmed six satellites with stellar masses between 10 5.6 and 10 8 M ⊙ . Most confirmed satellites are star-forming, in contrast to the primarily quiescent satellites observed around MW-mass hosts. By comparing observed satellite abundances and stellar mass functions with theoretical predictions, we find no evidence of a “missing satellite problem” in the dwarf galaxy regime.

Li, Jiaxuan 嘉轩李 [Princeton University, NJ (United ↗

DOE OSTI2022

Nanoscale defect evaluation framework combining real-time transmission electron microscopy and integrated machine learning-particle filter estimation

Observation of dynamic processes by transmission electron microscopy (TEM) is an attractive technique to experimentally analyze materials’ nanoscale phenomena and understand the microstructure-properties relationships in nanoscale. Even if spatial and temporal resolutions of real-time TEM increase significantly, it is still difficult to say that the researchers quantitatively evaluate the dynamic behavior of defects. Images in TEM video are a two-dimensional projection of three-dimensional space phenomena, thus missing information must be existed that makes image’s uniquely accurate interpretation challenging. Therefore, even though they are still a clustering high-dimensional data and can be compressed to two-dimensional, conventional statistical methods for analyzing images may not be powerful enough to track nanoscale behavior by removing various artifacts associated with experiment; and automated and unbiased processing tools for such big-data are becoming mission-critical to discover knowledge about unforeseen behavior. We have developed a method to quantitative image analysis framework to resolve these problems, in which machine learning and particle filter estimation are uniquely combined. The quantitative and automated measurement of the dislocation velocity in an Fe-31Mn-3Al-3Si autunitic steel subjected to the tensile deformation was performed to validate the framework, and an intermittent motion of the dislocations was quantitatively analyzed. The framework is successfully classifying, identifying and tracking nanoscale objects; these are not able to be accurately implemented by the conventional mean-path based analysis.

36 MATERIALS SCIENCE↗

DOE OSTI2023

Floating Block Method for Quantum Monte Carlo Simulations

Quantum Monte Carlo simulations are powerful and versatile tools for the quantum many-body problem. In addition to the usual calculations of energies and eigenstate observables, quantum Monte Carlo simulations can in principle be used to build fast and accurate many-body emulators using eigenvector continuation or design time-dependent Hamiltonians for adiabatic quantum computing. Furthermore, these new applications require something that is missing from the published literature, an efficient quantum Monte Carlo scheme for computing the inner product of ground state eigenvectors corresponding to different Hamiltonians. In this work, we introduce an algorithm called the floating block method, which solves the problem by performing Euclidean time evolution with two different Hamiltonians and interleaving the corresponding time blocks. We use the floating block method and nuclear lattice simulations to build eigenvector continuation emulators for energies of 4 He, 8 Be, 12 C, and 16 O nuclei over a range of local and nonlocal interaction couplings. From the emulator data, we identify the quantum phase transition line from a Bose gas of alpha particles to a nuclear liquid.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2024

Hydra: Computer Vision for Online Data Quality Monitoring

Hydra is a system utilizing computer vision for near real-time data quality monitoring. Currently operational across all of Jefferson Lab’s experimental halls, it reduces the workload of shift takers by autonomously monitoring diagnostic plots during experiments. Hydra uses "off-the-shelf" supervised learning technologies and is supported by a comprehensive MySQL database. To simplify access, web apps have been developed to facilitate both labeling and monitoring of Hydra’s inferences. Hydra can connect with the alarm system and incorporates complete historical tracking, enabling it to identify issues that shift takers could miss. When issues are detected, a natural first question is: "Why does Hydra think there is a problem?" To answer, Hydra employs Gradient-weighted Class Activation Maps (GradCAM) to identify regions of the image that are important for the specific classification. This interpretive layer enhances transparency and trustworthiness, which is essential for integration with experiment workflows and operation. The Hydra system, results, and sociological considerations for deployment will be discussed.

Jeske, Torri↗

DOE OSTI2022

Provider Perspectives: Identification and Follow-up of Infants who Are Deaf or Hard of Hearing

Objective Without timely screening, diagnosis, and intervention, hearing loss can cause significant delays in a child's speech, language, social, and emotional development. In 2019, Texas had nearly twice the average rate of loss to follow-up (LFU) or loss to documentation (LTD; i.e., missing documentation of services received) among infants who did not pass their newborn hearing screening compared to the United States overall (51.1 vs. 27.5%). We aimed to identify factors contributing to LFU/LTD among infants who do not pass their newborn hearing screening in Texas. Study Design Data were collected through semistructured qualitative interviews with 56 providers along the hearing care continuum, including hospital newborn hearing screening program staff, audiologists, primary care physicians, and early intervention (EI) program staff located in three rural and urban public health regions in Texas. Following recording and transcription of the interviews, we used qualitative data analysis software to analyze themes using a conventional content analysis approach. Results Frequently cited barriers included problems with family access to care, difficulty contacting patients, problems with communication between providers and referrals, lack of knowledge among providers and parents, and problems using the online reporting system. Providers in rural areas more often mentioned problems with family access to care and contacting families compared to providers in urban areas. Conclusion These findings provide insight into strategies that public health professionals and health care providers can use to work together to help further increase the number of children identified early who may benefit from EI services. Key Points

Obstetrics & Gynecology↗

DOE OSTI2021

Decentralized Low-Rank State Estimation for Power Distribution Systems

This article considers the low-observability state estimation problem in power distribution networks and develops a decentralized state estimation algorithm leveraging the matrix completion methodology. Matrix completion has been shown to be an effective technique in state estimation that exploits the low dimensionality of the power system measurements to recover missing information. This technique can utilize an approximate (linear) load flow model, or it can be used with no physical models in a network where no information about the topology or line admittance is available. The direct application of matrix completion algorithms requires solving a semi-definite programming (SDP) problem, which becomes computationally challenging for large networks. We therefore develop a decentralized algorithm that capitalizes on the popular proximal alternating direction method of multipliers (proximal ADMM). The method allows us to distribute the computation among different areas of the network, leading to a scalable algorithm. By doing all computations at individual control areas and only communicating with neighboring areas, the algorithm eliminates the need for data to be sent to a central processing unit and thus increases efficiency and contributes to the goal of autonomous control of distribution networks. We illustrate the advantages of the proposed algorithm numerically using standard IEEE test cases.

41 EE - Solar Energy Technologies Office (EE-4S)↗

DOE OSTI2021

Matrix Completion for Improved Observability in Low-Voltage Distribution Grids

This paper considers the problem of recovering missing entries in a partially observed matrix from relatively few measurements (i.e., the so-called matrix completion problem) with the aim of increasing the presently limited observability of low-voltage distribution grids. To this end, the partially observed matrix is formed using scarce voltage magnitude measurements while accounting for their spatial information. Voltage readings are assumed to be collected from distribution utility sensors and/or geographically-distributed cable television network sensors located in immediate proximity to distribution grid nodes. A matrix completion approach built on the parameter-less singular value shrinkage technique is used to estimate voltage magnitudes at otherwise non-observable low-voltage nodes using a small number of single- or multiple-snapshot data. The effectiveness of the proposed approach is demonstrated using a U.S.-style distribution test system from the synthetic SMART- DS data set under very low- to moderate-observability conditions.

low-rank matrix completion↗

DOE OSTI2022

Photo- and Electrocouplings of Nucleon Resonances

Advances in the exploration of the spectrum and structure of the excited states of the nucleon from experiments with electromagnetic probes on proton targets are presented here. Impressive progress has been achieved in the studies of exclusive meson photoproduction in experiments with continuous electron beams and with detectors of almost 4π acceptance. The high-quality data, coupled with the advances in the amplitude analyses of exclusive photo- and hadroproduction data, allow for the observation of several long-awaited new baryon states known previously as the “missing” resonances. Studies of exclusive meson electroproduction in the resonance region with the CLAS detector at JLab have provided the dominant part of the available world information on exclusive meson electroproduction observables. These data offer unique information on the structure of most well-established excited nucleon states in the mass region up to 1.8 GeV in terms of the evolution of their electroexcitation amplitudes with momentum transfer of the virtual photon. We discuss the impact of these results on the insight into the strong interaction dynamics that underlie the generation of the full spectrum of nucleon resonances of distinctively different structure. These results shed light on the emergence of hadron mass, which is one of the most important and still open problems in the Standard Model. The extension of the nucleon resonance stud

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2021

Limited-angle computed tomography with deep image and physics priors

Computed tomography is a well-established x-ray imaging technique to reconstruct the three-dimensional structure of objects. It has been used extensively in a variety of fields, from diagnostic imaging to materials and biological sciences. One major challenge in some applications, such as in electron or x-ray tomography systems, is that the projections cannot be gathered over all the angles due to the sample holder setup or shape of the sample. This results in an ill-posed problem called the limited angle reconstruction problem. Typical image reconstruction in this setup leads to distortion and artifacts, thereby hindering a quantitative evaluation of the results. To address this challenge, we use a generative model to effectively constrain the solution of a physics-based approach. Our approach is self-training that can iteratively learn the nonlinear mapping from partial projections to the scanned object. Because our approach combines the data likelihood and image prior terms into a single deep network, it is computationally tractable and improves performance through an end-to-end training. We also complement our approach with total-variation regularization to handle high-frequency noise in reconstructions and implement a solver based on alternating direction method of multipliers. We present numerical results for various degrees of missing angle range and noise levels, which demonstrate the effectiveness of the proposed approach.

47 OTHER INSTRUMENTATION↗