Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Missing data problem”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

A Kriging based spatiotemporal approach for traffic volume data imputation

Along with the rapid development of Intelligent Transportation Systems, traffic data collection technologies have progressed fast. The emergence of innovative data collection technologies such as remote traffic microwave sensor, Bluetooth sensor, GPS-based floating car method, and automated license plate recognition, has significantly increased the variety and volume of traffic data. Despite the development of these technologies, the missing data issue is still a problem that poses great challenge for data based applications such as traffic forecasting, real-time incident detection, dynamic route guidance, and massive evacuation optimization. A thorough literature review suggests most current imputation models either focus on the temporal nature of the traffic data and fail to consider the spatial information of neighboring locations or assume the data follow a certain distribution. These two issues reduce the imputation accuracy and limit the use of the corresponding imputation methods respectively. As a result, this paper presents a Kriging based data imputation approach that is able to fully utilize the spatiotemporal correlation in the traffic data and that does not assume the data follow any distribution. A set of scenarios with different missing rates are used to evaluate the performance of the proposed method. The performance of the proposed method was compared with that of two other widely used methods, historical average and K-nearest neighborhood. Comparison results indicate that the proposed method has the highest imputation accuracy and is more flexible compared to other methods.

Science & Technology - Other Topics↗

Case studies in bias reduction and inference for electronic health record data with selection bias and phenotype misclassification

Electronic health records (EHR) are not designed for population‐based research, but they provide easy and quick access to longitudinal health information for a large number of individuals. Many statistical methods have been proposed to account for selection bias, missing data, phenotyping errors, or other problems that arise in EHR data analysis. However, addressing multiple sources of bias simultaneously is challenging. We developed a methodological framework (R package, SAMBA ) for jointly handling both selection bias and phenotype misclassification in the EHR setting that leverages external data sources. These methods assume factors related to selection and misclassification are fully observed, but these factors may be poorly understood and partially observed in practice. As a follow‐up to the methodological work, we demonstrate how to apply these methods for two real‐world case studies, and we evaluate their performance. In both examples, we use individual patient‐level data collected through the University of Michigan Health System and various external population‐based data sources. In case study (a), we explore the impact of these methods on estimated associations between gender and cancer diagnosis. In case study (b), we compare corrected associations between previously identified genetic loci and age‐related macular degeneration with gold standard external summary estimates. These case studies illustrate how to utilize diverse auxiliary information to achieve less biased inference in EHR‐based research.

60 APPLIED LIFE SCIENCES↗

Inpainting radar missing data regions with deep learning

Abstract. Missing and low-quality data regions are a frequent problem for weather radars. They stem from a variety of sources: beam blockage, instrument failure, near-ground blind zones, and many others. Filling in missing data regions is often useful for estimating local atmospheric properties and the application of high-level data processing schemes without the need for preprocessing and error-handling steps – feature detection and tracking, for instance. Interpolation schemes are typically used for this task, though they tend to produce unrealistically spatially smoothed results that are not representative of the atmospheric turbulence and variability that are usually resolved by weather radars. Recently, generative adversarial networks (GANs) have achieved impressive results in the area of photo inpainting. Here, they are demonstrated as a tool for infilling radar missing data regions. These neural networks are capable of extending large-scale cloud and precipitation features that border missing data regions into the regions while hallucinating plausible small-scale variability. In other words, they can inpaint missing data with accurate large-scale features and plausible local small-scale features. This method is demonstrated on a scanning C-band and vertically pointing Ka-band radar that were deployed as part of the Cloud Aerosol and Complex Terrain Interactions (CACTI) field campaign. Three missing data scenarios are explored: infilling low-level blind zones and short outage periods for the Ka-band radar and infilling beam blockage areas for the C-band radar. Two deep-learning-based approaches are tested, a convolutional neural network (CNN) and a GAN that optimize pixel-level error or combined pixel-level error and adversarial loss respectively. Both deep-learning approaches significantly outperform traditional inpainting schemes under several pixel-level and perceptual quality metrics.

54 ENVIRONMENTAL SCIENCES↗

24.1.3.1.1 – Frisch Grid CZT Spectrometer

During FY18, BNL continued helping FLIR to integrate the instrument prototype, improving the readout software and testing electronics. BNL and LLNL carried out more measurements with the 4x4 arrays in laboratory conditions using several Cs sources and reanalyzed the raw data from the field measurements. LLNL processed the old recalibrated data and new data collected in the lab. These efforts allowed us to understand the cause of the discrepancies observed during the measurements at the Westinghouse Fuel Fabrication Facility (WFFF). The problem was traced back to the original ASIC and DAQ firmware provided by the ASIC developers. A program error and some other deficiencies were identified in the original firmware. These problems, which caused missing some detected events from the data stream, never showed up in the lab environment. So, they were not spotted before the field test. The ASIC designers are currently working to address these problems and will provide the updated firmware for the next field measurements.

36 MATERIALS SCIENCE↗

24.1.3.1.1 – Frisch Grid CZT Spectrometer

During FY18, BNL continued helping FLIR to integrate the instrument prototype, improving the readout software and testing electronics. BNL and LLNL carried out more measurements with the 4x4 arrays in laboratory conditions using several Cs sources and reanalyzed the raw data from the field measurements. LLNL processed the old recalibrated data and new data collected in the lab. These efforts allowed us to understand the cause of the discrepancies observed during the measurements at the Westinghouse Fuel Fabrication Facility (WFFF). The problem was traced back to the original ASIC and DAQ firmware provided by the ASIC developers. A program error and some other deficiencies were identified in the original firmware. These problems, which caused missing some detected events from the data stream, never showed up in the lab environment. So, they were not spotted before the field test. The ASIC designers are currently working to address these problems and will provide the updated firmware for the next field measurements.

97 MATHEMATICS AND COMPUTING↗

An Approach for Setting Priorities for Experimental Studies in Support of Safety Assessments - 19393

Many experiments have been conducted, in both academia and industry, to support the understanding of the disposal of radioactive waste. For the disposal of low-level radioactive waste (LLW) into radioactive waste repositories, it is important to develop an approach to tackle which datasets are able to directly underpin safety assessments, and what future research experiments could address the gaps in this understanding. In this paper, we present an approach to identify which future experiments should be considered to underpin safety assessments for disposals of LLW in a near-surface repository. The approach involves 6-steps: define the disposal problem (e.g. the disposal of LLW in engineered vaults); determine what key processes may affect the disposal of the waste; establish the assumptions underpinning the key processes; determine how sensitive the key process parameters are to variation; determine whether key data are available and assess the quality of that data with respect to the disposal problem; and determine attributes for experiments needed to supply missing key data. We show, in this paper, how this approach can be used to develop a better understanding of C-14 gas generation, under varying degrees of groundwater saturation, for the disposal of LLW into engineered vaults, in the near-surface. Furthermore, we have developed an approach that identifies the types of experiments and their attributes to aid researchers to help select appropriate types of experiments to underpin safety assessments. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Effective Missing Value Imputation Methods for Building Monitoring Data

To understand behaviors of natural and man-made events, such as energy consumption of buildings, which accounts for 40% of energy uses in the US, we deploy automated monitoring devices to record periodic observations. However, such experimental and observation data often contains problems and irregularities that have to be cleaned up before analyses. Due to various conditions affecting sensor operations, the communication channels, recording steps, or the recording media, the recorded data might have missing values, errors, or anomalous values. An effective way to clean up these problems is to replace these missing values, errors and anomalous values with expected values, a process generally known as imputation. In this work, we survey commonly used missing value imputation techniques and compare their performance on a set of building monitoring data. To compare the different types of sensor measurements with widely varying characteristics, we use normalized root mean squared error (NRMSE) as the key metric for the effectiveness of the imputation methods. We additionally consider periodicity and run time when considering comparing methods. Through extensive testing, we find that for small gap sizes, up to 8 consecutive missing values, linear interpolation performs the best; for larger gaps stretching up to 48 consecutive missing values, K-nearest neighbors provides the most accurate imputations; for even larger gaps, more computational intensive methods, such as matrix factorization, achieve the smallest NRMSE. Additionally, we observe that these computationally intensive algorithms not only provide accurate imputations for large gaps, but are also more robust across all types of sensors.

Cho, B↗

Identifying recharge sources and their impacts on a North Central New Mexico shallow aquifer using unsupervised machine learning

In this article, shallow aquifers are important but highly variable resources in arid to semi-arid regions. Limited shallow aquifer volume results in high sensitivity to recharge fluctuations, which can impact the local fauna and flora, and transport of contaminants in the aquifer or vadose zone. Aquifer response to external forcing (e.g., precipitation) is usually solved by estimating aquifer parameters and running physics-based models to match known fluctuations of hydraulic head. However, this technique is time and computationally expensive. Furthermore, high aquifer complexity decreases precision in physics-based models. Alternatively supervised machine learning is used to predict aquifer dynamics. However, these techniques rely on input data and struggle to interpret aquifer response for missing sources (i.e., snowpack data). To counter these problems, we propose an unsupervised machine learning technique (NMFk) to estimate the impact of different sources on aquifer recharge. NMFk is used to understand the influence of external forcing on shallow aquifer recharge in the Pajarito Plateau (Los Alamos, NM, USA). The results show how NMFk can be used to reduce the data dimension in a complex field dataset to three recharge signals that cause fluctuations within the field data. Here, the source signals are interpreted as rainfall, snowmelt, and a delayed aquifer response to the previous two signals. These results evidence how heterogeneous aquifers delimited by canyons incised into the Pajarito Plateau respond in similar ways to the source signals identified by NMFk. Furthermore, results show the importance of the local geology where faults act as sinks, and anthropogenic disturbances can facilitate infiltration amplifying the interpreted signal.

54 ENVIRONMENTAL SCIENCES↗

Hyperon production and polarization with CLAS and CLAS12 at Jlab

Many baryon resonances are predicted studying the channels with π, but very few established. This is the so-called "missing resonance" problem. It is important to provide data to investigate the spectrum of baryon (N* and Δ) resonances, with the decay in KY. A comprehensive study of the electromagnetic strangeness production has been undertaken at Jefferson Lab using circularly and linearly polarized photon beam and polarized hydrogen and deuterium target. Recent results in strangeness photoproduction on protons, including cross sections and spin observables, are discussed in this paper, as well preliminary results on photoproduction on bound neutron. Also an overview of the future perspectives for the 12 GeV upgrade.

08 HYDROGEN↗

Polarization observables in deuteron photodisintegration below 360 MeV.

High precision measurements of induced and transferred recoil proton polarization in d({rvec y}, {rvec p})n have been performed for photon energies of 277-357 MeV and {theta}cm = 20{sup o}-120{sup o}. The measurements were motivated by a longstanding discrepancy between meson-baryon model calculations and data at higher energies. At the low energies of this experiment, theory continues to fail to reproduce the data, indicating that either something is missing in the calculations and/or there is a problem with the accuracy of the nucleon-nucleon potential being used.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Missing Wedge Completion via Unsupervised Learning with Coordinate Networks

Cryogenic electron tomography (cryoET) is a powerful tool in structural biology, enabling detailed 3D imaging of biological specimens at a resolution of nanometers. Despite its potential, cryoET faces challenges such as the missing wedge problem, which limits reconstruction quality due to incomplete data collection angles. Recently, supervised deep learning methods leveraging convolutional neural networks (CNNs) have considerably addressed this issue; however, their pretraining requirements render them susceptible to inaccuracies and artifacts, particularly when representative training data is scarce. To overcome these limitations, we introduce a proof-of-concept unsupervised learning approach using coordinate networks (CNs) that optimizes network weights directly against input projections. This eliminates the need for pretraining, reducing reconstruction runtime by 3–20× compared to supervised methods. Our in silico results show improved shape completion and reduction of missing wedge artifacts, assessed through several voxel-based image quality metrics in real space and a novel directional Fourier Shell Correlation (FSC) metric. Our study illuminates benefits and considerations of both supervised and unsupervised approaches, guiding the development of improved reconstruction strategies.

42 ENGINEERING↗

WE-G-18A-03: Cone Artifacts Correction in Iterative Cone Beam CT Reconstruction

Purpose: For iterative reconstruction (IR) in cone-beam CT (CBCT) imaging, data truncation along the superior-inferior (SI) direction causes severe cone artifacts in the reconstructed CBCT volume images. Not only does it reduce the effective SI coverage of the reconstructed volume, it also hinders the IR algorithm convergence. This is particular a problem for regularization based IR, where smoothing type regularization operations tend to propagate the artifacts to a large area. It is our purpose to develop a practical cone artifacts correction solution. Methods: We found it is the missing data residing in the truncated cone area that leads to inconsistency between the calculated forward projections and measured projections. We overcome this problem by using FDK type reconstruction to estimate the missing data and design weighting factors to compensate the inconsistency caused by the missing data. We validate the proposed methods in our multi-GPU low-dose CBCT reconstruction system on multiple patients' datasets. Results: Compared to the FDK reconstruction with full datasets, while IR is able to reconstruct CBCT images using a subset of projection data, the severe cone artifacts degrade overall image quality. For head-neck case under a full-fan mode, 13 out of 80 slices are contaminated. It is even more severe in pelvis case under half-fan mode, where 36 out of 80 slices are affected, leading to inferior soft-tissue delineation. By applying the proposed method, the cone artifacts are effectively corrected, with a mean intensity difference decreased from ∼497 HU to ∼39HU for those contaminated slices. Conclusion: A practical and effective solution for cone artifacts correction is proposed and validated in CBCT IR algorithm. This study is supported in part by NIH (1R01CA154747-01)

60 APPLIED LIFE SCIENCES↗

Estimation of missing values in solar radiation data using piecewise interpolation methods: Case study at Penang city

The solar radiation values have been composed by automatic weather station using the device that namely pyranometer. The device is functions to records all the radiation values that have been dispersed, and these data are very useful for it experimental works and solar device’s development. In addition, for modeling and designing on solar radiation system application is needed for complete data observation. Unfortunately, lack for obtained the complete solar radiation data frequently occur due to several technical problems, which mainly contributed by monitoring device. Into encountering this matter, estimation missing values in an effort to substitute absent values with imputed data. This paper aimed to evaluate several piecewise interpolation techniques likes linear, splines, cubic, and nearest neighbor into dealing missing values in hourly solar radiation data. Then, proposed an extendable work into investigating the potential used of cubic Bezier technique and cubic Said-ball method as estimator tools. As result, methods for cubic Bezier and Said-ball perform the best compare to another piecewise imputation technique.

14 SOLAR ENERGY↗

miss-SNF: a multimodal patient similarity network integration approach to handle completely missing data sources

Abstract Motivation Precision medicine leverages patient-specific multimodal data to improve prevention, diagnosis, prognosis, and treatment of diseases. Advancing precision medicine requires the non-trivial integration of complex, heterogeneous, and potentially high-dimensional data sources, such as multi-omics and clinical data. In the literature, several approaches have been proposed to manage missing data, but are usually limited to the recovery of subsets of features for a subset of patients. A largely overlooked problem is the integration of multiple sources of data when one or more of them are completely missing for a subset of patients, a relatively common condition in clinical practice. Results We propose miss-Similarity Network Fusion (miss-SNF), a novel general-purpose data integration approach designed to manage completely missing data in the context of patient similarity networks. miss-SNF integrates incomplete unimodal patient similarity networks by leveraging a non-linear message-passing strategy borrowed from the SNF algorithm. miss-SNF is able to recover missing patient similarities and is “task agnostic”, in the sense that can integrate partial data for both unsupervised and supervised prediction tasks. Experimental analyses on nine cancer datasets from The Cancer Genome Atlas (TCGA) demonstrate that miss-SNF achieves state-of-the-art results in recovering similarities and in identifying patients subgroups enriched in clinically relevant variables and having differential survival. Moreover, amputation experiments show that miss-SNF supervised prediction of cancer clinical outcomes and Alzheimer’s disease diagnosis with completely missing data achieves results comparable to those obtained when all the data are available. Availability and implementation miss-SNF code, implemented in R, is available at https://github.com/AnacletoLAB/missSNF.

Biochemistry & Molecular Biology↗

Autonomous Correction of Sensor Data Applied to Building Technologies Using Filtering Methods

Sensor data validity is extremely important in a number of applications, particularly building technologies where collected data are used to determine performance. An example of this is Oak Ridge National Laboratory s ZEBRAlliance research project, which consists of four single-family homes located in Oak Ridge, TN. The homes are outfitted with a total of 1,218 sensors to determine the performance of a variety of different technologies integrated within each home. Issues arise with such a large amount of sensors, such as missing or corrupt data. This paper aims to eliminate these problems using: (1) Kalman filtering and (2) linear prediction filtering techniques. Five types of data are the focus of this paper: (1) temperature; (2) humidity; (3) energy consumption; (4) pressure; and (5) airflow. Simulations show the Kalman filtering method performed best in predicting temperature, humidity, pressure, and airflow data, while the linear prediction filtering method performed best with energy consumption data.

Castello, Charles C↗

AGR 5/6/7 Data Qualification Report for ATR Cycles 162B through 168A

This report provides the qualification status of experimental data for the Advanced Gas Reactor (AGR) 5/6/7 fuel irradiation. AGR-5/6/7 was conducted in the Advanced Test Reactor (ATR) at Idaho National Laboratory (INL) in support of development and qualification of tri-structural isotropic (TRISO) low-enriched fuel for use in high temperature gas-cooled reactors. The objectives of the AGR-5/6/7 experiments are to: (i) irradiate reference-design fuel particles to support fuel qualification, (ii) establish operating margins for the fuel beyond normal operating conditions, and (iii) provide irradiated-fuel performance data and irradiated-fuel samples for post-irradiation examination (PIE) and safety testing. The test train contains five separate capsules that were independently controlled and monitored. Each capsule contains multiple 12.51-mm-long compacts filled with low enriched uranium carbide/oxide (UCO) TRISO fuel particles. The primary objective of the AGR-5/6 test (Capsules 1, 2, 4, and 5) is to verify successful performance of the reference-design fuel under normal operating conditions. The AGR-7 test (Capsule 3) was designed to explore fuel performance at higher temperatures to demonstrate the capability of the fuel to withstand conditions beyond normal operating conditions in support of plant design and licensing. AGR 5/6/7 will also provide irradiated-fuel performance data on fission-gas release from failed particles during irradiation. The AGR-5/6/7 capsules were irradiated in the ATR northeast flux trap location. The experiment began on February 16, 2018 and ended on July 22, 2020, spanning nine ATR cycles over two and a half years. Thus, the AGR-5/6/7 fuel compacts were irradiated for a total of 360.9 effective full power days. The AGR 5/6/7 experiment was able to remain in the reactor core during all three Powered Axial Locator Mechanism (PALM) cycles (163A, 165A, and 167A) without overheating its fuel compacts. This report includes irradiation monitoring data from nine ATR Cycles: 162B, 163A, 164A, 164B, 165A, 166A, 166B, 167A, and 168A, as stored in the Nuclear Data Management and Analysis System (NDMAS). During irradiation, data records consisted of instantaneous measurements recorded every minute and provided by text files automatically every 2 hours. The AGR 5/6/7 data streams addressed in this report include thermocouple (TC) temperatures, sweep gas data (flow rates [capsule inlet, outlet, and downstream at detector], pressure, and moisture content), and Fission Product Monitoring System (FPMS) data (release rates and release to birth rate ratios [R/Bs]) for each of the five capsules. A total of 94,989,908 TC temperature and sweep gas data records were received and processed by NDMAS for AGR 5/6/7 irradiation. Of these records, 41,593,387 (or 43.7% of the total) met data collection and accuracy requirements and are labeled as Qualified. A total of 57,746,693 TC temperature readings were captured from 54 installed TCs. Among them, 10,034,676 TC temperature records (only 17.4%) were Qualified and 47,701,371 TC temperatures (or 82.6%) are Failed due to 48 TC failures (63.5%) and due to missing values (19.1%). To assess performance of the operational TCs, analysis of daily correlations between TCs found no evidence of virtual junction failure for any TCs. Analyses on control charts of TC temperature differences revealed trending in TC readings for TC2, 4, 5, and 13 in Capsule 3, but there is no conclusive indication of TC drift failure that caused those trends. Therefore, TC control charts are not used to disqualify TC data, but only for users’ consideration. For sweep gas flow rates, a total of 31,519,747 gas flow records (84.4%) are Qualified for use for AGR-5/6/7 experiment; 5,723,468 gas flow records (15.4%) are Failed due mostly to missing values; and 74,641 high sweep gas flow rates (0.2 %) are Trend. A large number of Failed missing TC temperature and gas flow values were caused by an error in the data output script that outputted a ‘NULL’ value when values were unchanged. This problem was fixed during the outage of Cycle 166B, which led to a substantially decreased number of missing values during the last three cycles. Nonetheless, a large amount of non-missing data remained because of the high data acquisition frequency (1-minute) and still provided sufficient data to effectively monitor the experiment as designed. For FPMS data, NDMAS received and processed fission product release and R/B data for nine ATR cycles, when ATR core reached full power during AGR 5/6/7 irradiation. These data consist of 110,388 release rate records and 110,388 R/B records for the twelve radionuclides (Kr 85m, Kr 87, Kr 88, Kr 89, Kr 90, Xe 131m, Xe 133, Xe 135, Xe 135m, Xe 137, Xe 138, and Xe 139) for each of the five capsules. Equivalent numbers of uncertainty records associated the release rates and R/B values were provided. To date, qualification status of the FPMS data stored in the NDMAS dat

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Machine learning for seismic low-frequency extrapolation

The cycle-skipping problem that plagues full waveform inversion (FWI) can be at least partially mitigated if low frequencies (which encode the kinematics of wave propagation in seismic data) are recorded. However, seismic sources and receivers are band-limited, so seismic data does not generally include signals down to 0 Hz. To improve our ability to solve the seismic inverse problem, one can synthesize this missing low-frequency (LF) content from the recorded high-frequency (HF) data using machine learning (ML) models. Deep learning models such as convolutional neural networks (CNNs) demonstrate impressive ability to perform low frequency extrapolation. However, such models require powerful hardware (GPU machines) and careful training. We assess the extrapolation capabilities of three different ML models that do not require GPU machines, namely, random forest, Gaussian process regression and gradient boosting, on both synthetic and real data. Experimental results on two synthetic data sets (generated from a low velocity lens embedded in a homogeneous medium, and the Marmousi model) demonstrate that FWI applied to the extrapolated data consistently improves inversion accuracy relative to FWI applied to the original data sets that do not contain low frequencies. Application of low-frequency extrapolation to real data from the Northwest Shelf of Australia demonstrates that tree-based ML models such as gradient boosting can outperform CNNs in terms of both accuracy and computational cost on non-GPU architectures.

58 GEOSCIENCES↗