Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Missing Data Recovery”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

A Deep Learning Approach for In-Network Synchrophasor Missing Data Recovery Using Programmable Network Switches

Phasor measurement unit (PMU) networks deliver accurate and timely measurements, which is essential for managing today’s electric power systems. To ensure data quality and enhance the cyber-resilience of PMU networks against malicious attacks and data errors, this study presents an online PMU missing data recovery scheme by leveraging P4 programmable switches. The data plane incorporates a customized PMU protocol parser that abstracts the necessary payload data for recovery. Recovery processes are executed in the control plane using a pre-trained machine learning model. Both traditional and advanced ML models, such as transformer and TimeGPT, are explicitly employed for data prediction. This approach ensures rapid and precise data recovery. Performance evaluations focus on recovery speed and accuracy, using a real dataset from a campus microgrid. With 20% missing PMU data, the mean absolute percentage error for voltage magnitude is 0.0384%, and the phase angle error discrepancy is approximately 0.4064%.

Phasor Measurement Unit, Machine Learning, Program↗

NLSI Focus Group on Missing ALSEP Data Recovery: Progress and Plans

On the six Apollo landed missions, the Astronauts deployed the Apollo Lunar Surface Experiments Package (ALSEP) science stations which measured active and passive seismic events, magnetic fields, charged particles, solar wind, heat flow, the diffuse atmosphere, meteorites and their ejecta, lunar dust, etc. Today's scientists are able to extract new information and make new discoveries from the old ALSEP data utilizing recent advances in computer capabilities and new analysis techniques. However, current-day investigators are encountering problems trying to use the ALSEP data. In 2007 archivists from NASA Goddard Space Flight Center (GSFC) National Space Science Data Center (NSSDC) estimated only about 50 percent of the processed ALSEP lunar surface data-of-interest to current lunar science investigators were in the NSSDC archives. The current-day lunar science investigators found most of the ALSEP data, then in the NSSDC archives. were extremely difficult to use. The data were in forms often not well described in the published reports and rerecording anomalies existed in the data which could only be resolved by tape experts. To resolve this problem, the DPS Lunar Data Node was established in 2008 at NSSDC and is in the process of successfully making the existing archived ALSEP data available to current-day investigators in easily useable forms. In July of 2010 the NASA Lunar Science Institute (NLSI) at Ames Research Center established the Recovery of Missing ALSEP Data Focus Group in recognition of the importance of the current activities to find the raw and processed ALSEP data missing from the NSSDC archives.

Lewis, L. R.↗

NLSI Focus Group on Recovery of Missing ALSEP Data: Status Update for 2012 NLSI Science Forum

On the six Apollo lunar landed missions, the Astronauts deployed the Apollo Lunar Surface Experiments Package (ALSEP) science stations which measured active and passive seismic events, magnetic fields, charged particles, solar wind, heat flow, the diffuse atmosphere, meteorites and their ejecta, lunar dust, etc. Today s investigators are able to extract new information and make new discoveries from the old ALSEP data utilizing recent advances in computer capabilities and new analysis techniques. However, current-day investigators are encountering problems in trying to use the ALSEP data. The data were in formats often not well described in the published reports and contained rerecording anomalies which required tape experts to resolve. To solve these problems the DPS Lunar Data Node was established at NASA Goddard Space Flight Center (GSFC) NASA Space Science Data Center (NSSDC) in 2008 and is currently in the process of making the existing archived ALSEP data available to current-day investigators in easily useable forms. However, current estimates by NSSDC archivists are that only about 60 percent of the PI processed ALSEP data and less than 30 percent of the raw experiment ALSEP data-of-interest to current lunar science investigators are currently in the NSSDC archives.

Lewis, Lyach R.↗

A Robust Event Diagnostics Platform: Integrating Tensor Analytics and Machine Learning into Real-time Grid Monitoring

The objective of this project is to develop a robust event diagnostics (RED) platform by integrating state-of-the-art tensor analytics and machine learning into real-time grid monitoring. The proposed platform can effectively analyze and discover the information hiding within the provided PMU data for effective real-time grid monitoring. The proposed RED platform provides a set of robust diagnostics tools for grid operation and management, including 1) data quality assessment, 2) data completion, 3) event detection, and 4) robust event classification. All the functionalities of the RED platform can help the operator to make informed decisions and respond in a timely manner. The developed RED platform will serve as an innovative advisory tool to reliably identify key events and discover new insights about the events and grid characteristics in the PMU data, and contribute to the efficient, safe, reliable operation and design of the nation’s electric system.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Using an Informative Missing Data Model to Predict the Ability to Assess Recovery of Balance Control after Spaceflight

Astronauts show degraded balance control immediately after spaceflight. To assess this change, astronauts' ability to maintain a fixed stance under several challenging stimuli on a movable platform is quantified by "equilibrium" scores (EQs) on a scale of 0 to 100, where 100 represents perfect control (sway angle of 0) and 0 represents data loss where no sway angle is observed because the subject has to be restrained from falling. By comparing post- to pre-flight EQs for actual astronauts vs. controls, we built a classifier for deciding when an astronaut has recovered. Future diagnostic performance depends both on the sampling distribution of the classifier as well as the distribution of its input data. Taking this into consideration, we constructed a predictive ROC by simulation after modeling P(EQ = 0) in terms of a latent EQ-like beta-distributed random variable with random effects.

Feiveson, Alan H.↗

miss-SNF: a multimodal patient similarity network integration approach to handle completely missing data sources

Abstract Motivation Precision medicine leverages patient-specific multimodal data to improve prevention, diagnosis, prognosis, and treatment of diseases. Advancing precision medicine requires the non-trivial integration of complex, heterogeneous, and potentially high-dimensional data sources, such as multi-omics and clinical data. In the literature, several approaches have been proposed to manage missing data, but are usually limited to the recovery of subsets of features for a subset of patients. A largely overlooked problem is the integration of multiple sources of data when one or more of them are completely missing for a subset of patients, a relatively common condition in clinical practice. Results We propose miss-Similarity Network Fusion (miss-SNF), a novel general-purpose data integration approach designed to manage completely missing data in the context of patient similarity networks. miss-SNF integrates incomplete unimodal patient similarity networks by leveraging a non-linear message-passing strategy borrowed from the SNF algorithm. miss-SNF is able to recover missing patient similarities and is “task agnostic”, in the sense that can integrate partial data for both unsupervised and supervised prediction tasks. Experimental analyses on nine cancer datasets from The Cancer Genome Atlas (TCGA) demonstrate that miss-SNF achieves state-of-the-art results in recovering similarities and in identifying patients subgroups enriched in clinically relevant variables and having differential survival. Moreover, amputation experiments show that miss-SNF supervised prediction of cancer clinical outcomes and Alzheimer’s disease diagnosis with completely missing data achieves results comparable to those obtained when all the data are available. Availability and implementation miss-SNF code, implemented in R, is available at https://github.com/AnacletoLAB/missSNF.

Biochemistry & Molecular Biology↗

AlphaFold -assisted structure determination of a bacterial protein of unknown function using X-ray and electron crystallography

Macromolecular crystallography generally requires the recovery of missing phase information from diffraction data to reconstruct an electron-density map of the crystallized molecule. Most recent structures have been solved using molecular replacement as a phasing method, requiring an a priori structure that is closely related to the target protein to serve as a search model; when no such search model exists, molecular replacement is not possible. New advances in computational machine-learning methods, however, have resulted in major advances in protein structure predictions from sequence information. Methods that generate predicted structural models of sufficient accuracy provide a powerful approach to molecular replacement. Taking advantage of these advances, AlphaFold predictions were applied to enable structure determination of a bacterial protein of unknown function (UniProtKB Q63NT7, NCBI locus BPSS0212) based on diffraction data that had evaded phasing attempts using MIR and anomalous scattering methods. Using both X-ray and micro-electron (microED) diffraction data, it was possible to solve the structure of the main fragment of the protein using a predicted model of that domain as a starting point. The use of predicted structural models importantly expands the promise of electron diffraction, where structure determination relies critically on molecular replacement.

molecular replacement↗

Sequential Image Recovery Using Joint Hierarchical Bayesian Learning

Abstract Recovering temporal image sequences (videos) based on indirect, noisy, or incomplete data is an essential yet challenging task. We specifically consider the case where each data set is missing vital information, which prevents the accurate recovery of the individual images. Although some recent (variational) methods have demonstrated high-resolution image recovery based on jointly recovering sequential images, there remain robustness issues due to parameter tuning and restrictions on the type of sequential images. Here, we present a method based on hierarchical Bayesian learning for the joint recovery of sequential images that incorporates prior intra- and inter-image information. Our method restores the missing information in each image by “borrowing” it from the other images. More precisely, we couple sequential images by penalizing their pixel-wise difference. The corresponding penalty terms (one for each pixel and pair of subsequent images) are treated as weakly-informative random variables that favor small pixel-wise differences but allow occasional outliers. As a result, all of the individual reconstructions yield improved accuracy. Our method can be used for various data acquisitions and allows for uncertainty quantification. Some preliminary results indicate its potential use for sequential deblurring and magnetic resonance imaging.

Xiao, Yao↗

Functional Sensory-Motor Performance Following Long Term Space Flight: The First Results of "Field Test" Experiment

The effect that extended-duration space flights may have on human space travelers, including exploration missions, is widely discussed at the present time. Specifically, there is an increasing amount of evidence showing that the physical capacity of cosmonauts is significantly reduced after long-duration space flights. It is evident that the most impaired functions are those that rely on gravity, particularly up right posture and gait. Because of the sensorimotor disturbances manifested in the neurology of the posture and gait space flight and postflight changes may also be observed in debilitating motion sickness. While the severity of particular symptoms varies, disturbances in spatial orientation and alterations in the accuracy of voluntary movements are persistently observed after long-duration space flights. At this time most of the currently available data are primarily descriptive and not yet suitable for predicting operational impacts of most sensorimotor decrements observed upon landing on planetary surfaces or asteroids. In particular there are no existing data on the recovery dynamics or functionality of neurological, cardiovascular or muscle performance making it difficult to model or simulate the cosmonauts' activity after landing and develop the appropriate countermeasure that will ensure the rapid and safe recovery of crewmembers immediately after landing in what could be hostile environments. However and as a starting position, the videos we have acquired during recent data collection following the long duration flights of cosmonauts and astronauts walking and performing other tasks shortly after return from space flight speak volumes about their level of deconditioning. A joint Russian-American team has developed a new study specifically to address the changes in crewmembers performance and the recovery of performance with the intent of filling the missing data gaps. The first (pilot) phase of this study includes recording body kinematics and quantifying the coordination and timing of relatively simple basic movements - transition from seated and prone positions to standing, walking, stepping over obstacles, tandem walking, muscle compliance, as well as characteristics of postural sway and orthostatic tolerance. Testing for changes in these parameters have been initiated in the medical tent at the landing site. The first set of experiments showed that during the first hour after landing, cosmonauts and astronauts were able to execute (although slower and with more effort than preflight) simple movements such as egress from a seated or prone position and also to remain standing for 3.5 minutes without exhibiting pronounced cardiovascular changes. More challenging tests, however, demonstrated a prominent reduction in coordination - the obstacle task, for example, was performed at much slower speed and with a marked overestimation of the obstacle height and tandem walking was greatly degraded suggesting significant changes in proprioception, brainstem and vestibular function. There is some speculation that the neural changes, either from the bottom-up or top down may be long lasting; requiring compensatory responses that will modify or mask the adverse responses we have observed. Furthermore, these compensatory responses may actually be beneficial, helping achieve a more rapid adaptation to both weightlessness and a return to earth.

Tomilovskaya, E. S.↗

Bayesian High-Rank Hankel Matrix Completion for Nonlinear Synchrophasor Data Recovery

Phasor measurement units (PMUs) provide high temporal-resolution synchrophasor measurements for power system monitoring and control. The frequent data quality issues, such as missing and bad data, prevent the incorporation of synchrophasor data in real-time operations. Most existing data-driven data recovery methods assume the power system dynamics can be approximated by a linear dynamical system, and the recovery performance degrades significantly when the power system is experiencing nonlinear dynamics during significant events. Here, this paper proposes a data-driven Bayesian nonlinear synchrophasor data recovery method (Ba-NSDR) that can recover a consecutive time period of simultaneous data losses or errors across all channels, even when the underlying system is highly nonlinear. The idea is to lift the Hankel matrix of the spatial-temporal synchrophasor data to a higher dimension such that the lifted Hankel matrix is low-rank in that space and can be processed with the kernel trick. Our proposed Bayesian method then infers the probabilistic distributions of synchrophasor from the partial observations. Some distinctive features of Ba-NSDR include an uncertainty index to measure the accuracy of the recovery result and the robustness to parameter selections. Our method is verified on both synthetic and recorded event datasets.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

History and Status of ALSEP and the Apollo Lunar Data Project

A suite of automated scientific instruments (the Apollo Lunar Surface Experiment Package, or ALSEP) was installed at each of the landing sites of Apollo 12, 14, 15, 16, and 17 from 1969 to 1972. They operated from deployment until decommissioning on 30 September 1977. These data were continuously transmitted to Earth and saved on the Range Tapes, which were recorded at the Manned Space Flight Network stations. These data were also broken out by experiment and sent to the experiment Principal Investigators on what were called the P.I. Tapes. Starting in April 1973 the Range Tape data were stored in digital format on 7-track magnetic tapes, the ARCSAV Tapes. In February 1976, the handling of the Range Tapes was transferred to UT Galveston. They produced 9-track tapes referred to as the Work Tapes. Following the Apollo program the Range and ARCSAV tapes, which were never archived, were lost. The Work Tapes were archived at the National Space Science Data Center (NSSDC). Some investigators archived their individual experiment data with NSSDC as well, but much of the data had minimal documentation, were not in digital form, or were stored in difficult to translate formats. Data from many experiments were never delivered to the NSSDC. The Lunar Data Project was started to address the problem of both missing and not readily usable data. Our effort has resulted in recovery of some of the ARCSAV tapes, recovery and digitization of a large volume of Apollo scientific and technical documentation, and restoration of many ALSEP and other Apollo data collections. Restoration involves deciphering formats, assembling necessary ancillary data (metadata), and packaging data in digital format to be archived with the Planetary Data System (PDS). Recovery of the data from the ARCSAV tapes involved having the tapes read on special equipment and extracting the individual experiment data out of the integrated data stream. We will report on the history and status of the various recovery efforts.

Work Tapes↗

Calculation of power spectrums from digital time series with missing data points

Two algorithms are developed for calculating power spectrums from the autocorrelation function when there are missing data points in the time series. Both methods use an average sampling interval to compute lagged products. One method, the correlation function power spectrum, takes the discrete Fourier transform of the lagged products directly to obtain the spectrum, while the other, the modified Blackman-Tukey power spectrum, takes the Fourier transform of the mean lagged products. Both techniques require fewer calculations than other procedures since only 50% to 80% of the maximum lags need be calculated. The algorithms are compared with the Fourier transform power spectrum and two least squares procedures (all for an arbitrary data spacing). Examples are given showing recovery of frequency components from simulated periodic data where portions of the time series are missing and random noise has been added to both the time points and to values of the function. In addition the methods are compared using real data. All procedures performed equally well in detecting periodicities in the data.

Murray, C. W., Jr.↗

Single-Photon Counting Detector Scalability for High Photon Efficiency Optical Communications Links

For high photon-efficiency deep space or low power optical communications links, such as the Orion Artemis-2 Optical Communications System (O2O) project, the received optical signal is attenuated to the extent that single- photon detectors are required. For direct-detection receivers operating at 1.55 µm wavelength, single-photon detectors including Geiger-mode InGaAs avalanche photon diodes (APDs), and in particular superconducting nanowire single-photon detectors (SNSPDs) offer the highest sensitivity and fastest detection speeds. However, these photon detectors exhibit a recovery time between registered input pulses, effectively reducing the detection efficiency over the recovery interval, resulting in missed photon detections, reduced count rate, and ultimately limiting the achievable data rate. A method to overcome this limitation is to divide the received optical signal into multiple detectors in parallel. Here we analyze this approach for a receiver designed to receive a high photon efficiency serially concatenated pulse position modulation (SCPPM) input waveform. From measured count rate and efficiency data using commercial SNSPDs, we apply a model from which we determine the effective detection efficiency, or blocking loss, for different input signal rates. We analyze the scalability of adding detectors in parallel for different modulation orders and background levels to achieve desired data rates. Finally we show tradeoffs between the number of detectors and the required received optical power, useful for real link design considerations.

Vyhnalek, Brian E.↗

Introduction to Evolve Central Appalachia

The Evolve-CAPP project is developing and implementing strategies to enable the Central Appalachian basin to realize its full economic potential for producing high-value, non-fuel, carbon-ore (CO) products, rare earth elements (REE), and critical minerals (CM). The Basinal Assessment of CORE-CM Resources is a state-of-the-art geological model and comprehensive database that will be used to identify, locate, and quantify in-situ resources in the basin. Restrictive aspects of resource recovery, such as ownership and legal restrictions, are being considered in estimating recoverable resources. A gap analysis of missing or unavailable data that could not be fully integrated in the initial assessment due to budget, access, or other constraints during the assessment effort is planned. A characterization and data acquisition plan to fully characterize the basin’s CORE-CM potential will address gaps in data. The plan will be developed with input from project team members and relevant stakeholders, including land and mineral holding companies and coal operators, who maintain sampling programs as part of their normal operations.

Bishop, Richard↗

The use of airborne imaging spectrometer data to determine experimentally induced variation in coniferous canopy chemistry

It was experimentally determined whether induced differences in forest canopy chemical composition can be detected using data from the Airborne Imaging Spectrometer (AIS). Treatments were applied to an even-aged forest of Douglas fir trees. Work to date has stressed wet chemical analysis of foilage samples and correction of AIS data. Plot treatments were successful in providing a range of foliar N2 concentrations. Much time was spent investigating and correcting problems with the raw AIS data. Initial problems with groups of drop out lines in the AIS data were traced to the tape recorder and the tape drive. Custom adjustment of the tape drive led to recovery of most missing lines. Remaining individual drop out lines were replaced using average of adjacent lines. Application of a notch filter to the Fourier transform of the image in each band satisfactorily removed vertical striping. The aspect ratio was corrected by resampling the image in the line direction using nearest neighbor interpolation.

Swanberg, Nancy A.↗

Assessment of Turbulence Models for a Single-Injector Cooling Flow

This work assesses the ability of Reynolds-averaged Navier-Stokes turbulence models to predict a fundamental thermal cooling flow configuration. This geometry consists of a plate with a single cooling hole positioned downstream of a heated exhaust nozzle. The linear Shear Stress Transport (SST), non-linear Quadratic Constitutive Relation (QCR), and differential SSG/LRR Reynolds Stress Model (RSM) turbulence models were investigated for flow conditions with a blowing ratio of 1.0 and 2.0. Despite significant differences in turbulence model complexity, the results miss important details observed in the data and large eddy simulation (LES). Injection penetration is over-predicted. Mixing and centerline wake recovery is too slow. Temperature predictions are worse than velocity, suggesting that perhaps the simple gradient diffusion approximation used to model the turbulent heat flux may be to blame. These observations are put into context using other studies to identify shortcomings in the modeling of both the turbulent stress and turbulent heat flux.

Film Cooling↗

Assessment of Turbulence Models for a Single-Injector Cooling Flow

This work assesses the ability of Reynolds-averaged Navier-Stokes turbulence models to predict a fundamental thermal cooling flow configuration. This geometry consists of a plate with a single cooling hole positioned downstream of a heated exhaust nozzle. The linear Shear Stress Transport (SST), non-linear Quadratic Constitutive Relation (QCR), and differential SSG/LRR Reynolds Stress Model (RSM) turbulence models were investigated for flow conditions with a blowing ratio of 1.0 and 2.0. Despite significant differences in turbulence model complexity, the results miss important details observed in the data and large eddy simulation (LES). Injection penetration is over-predicted. Mixing and centerline wake recovery is too slow. Temperature predictions are worse than velocity, suggesting that perhaps the simple gradient diffusion approximation used to model the turbulent heat flux may be to blame. These observations are put into context using other studies to identify shortcomings in the modeling of both the turbulent stress and turbulent heat flux.

Film Cooling↗

Data about data – when, why and how metadata can support the digital plant

A structured approach for recording data quality and contextual information about how and why a signal exists – i.e. metadata – is central to interpret and use sensor data correctly. This is becoming increasingly important with the global trend with data-driven applications such as digital twins and AI-models. But a structured metadata collection and organization of sensor data is not routine in most plants, which can result in lost information and missed opportunities to make use of the investments made in the data collection. Therefore, the IWA task group on Metadata Collection and Organization in wastewater resource recovery systems (MetaCO) was initiated in 2020 and recently delivered the IWA scientific and technical report number 31. The report gives and in-depth description about metadata in water resources recovery facilities (WRRFs) and is available as open access at IWA publishing. The report is the outcome of the collaboration between more than 80 water professionals with the intention to serve WRRF data users with a guide on how to structure and make use of metadata throughout the data pipeline in order to maximize the value of sensor data.

Alferes, Janelcy [VITO, Belgium]↗