Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Missing data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 361 records · Page 20
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

A Visual Analytics Framework for Reviewing Streaming Performance Data

Understanding and tuning the performance of extreme-scale parallel computing systems demands a streaming approach due to the computational cost of applying offline algorithms to vast amounts of performance log data. Analyzing large streaming data is challenging because the rate of receiving data and limited time to comprehend data make it difficult for the analysts to sufficiently examine the data without missing important changes or patterns. To support streaming data analysis, we introduce a visual analytic framework comprising of three modules: data management, analysis, and interactive visualization. The data management module collects various computing and communication performance metrics from the monitored system using streaming data processing techniques and feeds the data to the other two modules. The analysis module automatically identifies important changes and patterns at the required latency. In particular, we introduce a set of online and progressive analysis methods for not only controlling the computational costs but also helping analysts better follow the critical aspects of the analysis results. Finally, the interactive visualization module provides the analysts with a coherent view of the changes and patterns in the continuously captured performance data. Through a multi-faceted case study on performance analysis of parallel discrete-event simulation, we demonstrate the effectiveness of our framework for identifying bottlenecks and locating outliers.

Human-centered computing↗
NASA NTRS2006

Managing the On-Board Data Storage, Acknowledgment and Retransmission System for Spitzer

The Spitzer Space Telescope has a two-phase downlink system. Data are transmitted during one telecom session. Then commands are sent during the next session to delete those data that were received and to retransmit those data that were missed. We must build sequences that are as efficient as possible to make the best use of our finite supply of liquid helium, One way to improve efficiency is to use only the minimum time needed during telecom sessions to transmit the predicted volume of data. But, we must also not fill the onboard storage and must allow enough time margin to retransmit missed data. We describe tools and procedures that allow us to build observatory sequences that are single-fault tolerant in this regard and that allow us to recover quickly and safely from anomalies that affect the receipt or acknowledgment of data.

Spitzer↗
NASA NTRS2006

Managing the On-Board Data Storage, Acknowledgement and Retransmission System for Spitzer

The Spitzer Space Telescope has a two-phase downlink system. Recorded data are transmitted during one telecom session. Then commands are sent during the next session to delete those data that were received on the ground and to retransmit those data that were missed. We must build science sequences that are as efficient as possible to make the best use of our supply of liquid helium. One way to improve efficiency is to use only the minimum time needed during telecom sessions to transmit the predicted volume of data. But, we must also not fill the on-board storage and must allow enough time margin to retransmit missed data. We describe tools and procedures that allow us to build science sequences that are single-fault tolerant in this regard and that allow us to recover quickly and safely from anomalies that affect the receipt or acknowledgment (i.e. deletion) of data.

on-board storage↗
DOE OSTI2015

Performance of the CMS missing transverse momentum reconstruction in pp data at $\sqrt{s}$ = 8 TeV

The performance of missing transverse energy reconstruction algorithms is presented by our team using√s=8 TeV proton-proton (pp) data collected with the CMS detector. Events with anomalous missing transverse energy are studied, and the performance of algorithms used to identify and remove these events is presented. The scale and resolution for missing transverse energy, including the effects of multiple pp interactions (pileup), are measured using events with an identified Z boson or isolated photon, and are found to be well described by the simulation. Novel missing transverse energy reconstruction algorithms developed specifically to mitigate the effects of large numbers of pileup interactions on the missing transverse energy resolution are presented. These algorithms significantly reduce the dependence of the missing transverse energy resolution on pileup interactions. Furthermore, an algorithm that provides an estimate of the significance of the missing transverse energy is presented, which is used to estimate the compatibility of the reconstructed missing transverse energy with a zero nominal value.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2020

Probing Few-Body Nuclear Dynamics via H 3 and He 3 ( e , e ' p ) pn Cross-Section Measurements

Here, we report the first measurement of the (e, e' p) three-body breakup reaction cross sections in helium-3 ( 3 He) and tritium ( 3 H) at large momentum transfer ($\langle{Q^2}\rangle$ ≈ 1.9 (GeV/c) 2 ) and x B > 1 kinematics, where the cross section should be sensitive to quasielastic (QE) scattering from single nucleons. The data cover missing momenta 40 ≤ p miss ≤ 500 MeV/c that, in the QE limit with no rescattering, equals the initial momentum of the probed nucleon. The measured cross sections are compared with state-of-the-art ab-initio calculations. Overall good agreement, within ±20%, is observed between data and calculations for the full p miss range for 3 H and for 100 ≤ p miss ≤ 350 MeV/c for 3 He. Including the effects of rescattering of the outgoing nucleon improves agreement with the data at p miss > 250 MeV/c and suggests contributions from charge-exchange (SCX) rescattering. The isoscalar sum of 3 He plus 3 H, which is largely insensitive to SCX, is described by calculations to within the accuracy of the data over the entire pmiss range. This validates current models of the ground state of the three-nucleon system up to very high initial nucleon momenta of 500 MeV/c.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2016

AGC-4 Experiment Irradiation Monitoring Data Qualification Interim Report

The Graphite Technology Development Program is running a series of six experiments to quantify the effects of irradiation on nuclear grade graphite. The fourth experiment, Advanced Graphite Creep 4 (AGC 4), began with Advanced Test Reactor (ATR) cycle 157D on May 30, 2015, and has been irradiated for two cycles. The capsule was removed from the reactor after ATR cycle 158A, which ended on January 2, 2016, due to interference with another experiment. Irradiation will resume when the interfering experiment is removed from the reactor. This report documents qualification of AGC 4 experiment irradiation monitoring data for use by the Advanced Reactor Technologies (ART) Technology Development Office (TDO) Program for research and development activities required to design and license the first HTR nuclear plant. Qualified data meet the requirements for use as described in the experiment planning and quality assurance documents. Failed data do not meet the requirements and provide no useable information. Trend data may not meet all requirements, but still provide some useable information. Use of Trend data requires assessment of how any deficiencies affect a particular use of the data. All thermocouples (TCs) have functioned throughout the AGC-4 experiment. All temperature data are Qualified for use by the ART TDO Program. Argon, helium, and total gas flow data were within expected ranges and are Qualified for use by the ART TDO Program. Discharge gas line moisture values were consistently low during cycle 157D. At the start of cycle 158A, gas moisture briefly spiked to over 600 ppmv and then declined throughout the cycle. Moisture values are within the measurement range of the instrument and are Qualified for use by the ART TDO Program. Graphite creep specimens were subjected to one of three loads, 393, 491, or 589 lbf. For a brief period during cycle 157D between 12:19 on June 2, 2015 and 08:23 on June 11, 2015 the load cells were wired incorrectly resulting in missing stack load data. Missing stack loads were estimated from measured ram pressures using regression equations developed from the existing data from cycle 157D. Estimated stack loads during this period are considered to be an accurate representation of actual load applied to the stacks. These loads deviate slightly from the planned loads. This deviation does not prevent the data from being Qualified for use, but must be taken into account when analyzing the effect of load on creep. Stack displacement increased consistently throughout the first two cycles with total displacement ranging from 0.4 to 0.8 in. During ATR outages, a set of pneumatic rams raised the stacks of graphite creep specimens to ensure the specimens were not stuck within the test train. This stack raising was performed twice. All stacks were raised successfully each time. The load and displacement data are Qualified for use by the ART TDO Program.

36 MATERIALS SCIENCE↗
DOE OSTI2015

Estimation of missing values in solar radiation data using piecewise interpolation methods: Case study at Penang city

The solar radiation values have been composed by automatic weather station using the device that namely pyranometer. The device is functions to records all the radiation values that have been dispersed, and these data are very useful for it experimental works and solar device’s development. In addition, for modeling and designing on solar radiation system application is needed for complete data observation. Unfortunately, lack for obtained the complete solar radiation data frequently occur due to several technical problems, which mainly contributed by monitoring device. Into encountering this matter, estimation missing values in an effort to substitute absent values with imputed data. This paper aimed to evaluate several piecewise interpolation techniques likes linear, splines, cubic, and nearest neighbor into dealing missing values in hourly solar radiation data. Then, proposed an extendable work into investigating the potential used of cubic Bezier technique and cubic Said-ball method as estimator tools. As result, methods for cubic Bezier and Said-ball perform the best compare to another piecewise imputation technique.

14 SOLAR ENERGY↗
NASA NTRS1994

Kriging in the Shadows: Geostatistical Interpolation for Remote Sensing

It is often useful to estimate obscured or missing remotely sensed data. Traditional interpolation methods, such as nearest-neighbor or bilinear resampling, do not take full advantage of the spatial information in the image. An alternative method, a geostatistical technique known as indicator kriging, is described and demonstrated using a Landsat Thematic Mapper image in southern Chiapas, Mexico. The image was first classified into pasture and nonpasture land cover. For each pixel that was obscured by cloud or cloud shadow, the probability that it was pasture was assigned by the algorithm. An exponential omnidirectional variogram model was used to characterize the spatial continuity of the image for use in the kriging algorithm. Assuming a cutoff probability level of 50%, the error was shown to be 17% with no obvious spatial bias but with some tendency to categorize nonpasture as pasture (overestimation). While this is a promising result, the method's practical application in other missing data problems for remotely sensed images will depend on the amount and spatial pattern of the unobscured pixels and missing pixels and the success of the spatial continuity model used.

Rossi, Richard E.↗
NASA NTRS2000

Quantitative Estimation of Some Possible Errors Associated With TRMM GV Monthly Rain Maps and a Scheme for Spatial Averaging of Radar Accumulations

The Tropical Rainfall Measuring Mission (TRMM) Ground Validation (GV) rain products are generated from ground-based radar observations using a bulk adjusted FACE radar reflectivity (Ze) - rain rate (R) relationship. Monthly Ze-R relationships are obtained by extracting the Ze data with a horizontal resolution of 2 km from an altitude of 1.5 km (from Constant Altitude PPIS, CAPPIS) over the locations of the gauges. The gauge and radar data are merged, and a quality control procedure is applied that eliminates poorly correlated gauge-radar data. Bulk adjusted Ze-R relations are then derived from this quality-controlled merged data base. These Z-R relations are applied to the base scan reflectivity to produce rain maps. The present study examines the effect of these different planes on the total accumulations from the radar and the associated effect on the bulk adjustment coefficient. In order to evaluate the credibility of these GV products, the effect of the different planes involved in the bulk adjustment method will be discussed, The gauge data will also be used to assess the impact of missing radar data (i.e., data gaps) upon the monthly radar-derived rainfall estimates. Experiments were also performed varying both the area of the radar data extracted over the gauges and the use of different weighting coefficients. Preliminary results from these sensitivity tests indicate improved agreement and consistency between the gauge and radar data.

Datta, Saswati↗
DOE OSTI2025

Denoising Autoencoder for Reconstructing Sensor Observation Data and Predicting Evapotranspiration: Noisy and Missing Values Repair and Uncertainty Quantification

Abstract Machine learning (ML) methods applied in scientific research often deal with interrelated features in high‐dimensional data. Reducing data noise and redundancy is needed to increase prediction accuracy and efficiency especially when dealing with data from field sensors. We explored an unsupervised learning method, the denoising autoencoder (DAE), to extract the underlying data structure from noisy raw data in the context of predicting hydrologic quantities from multiple field sensors. These sensors have intrinsic instrumental noise and occasional malfunctions that cause missing values. Our DAE neural network reconstructed meteorological sensor data containing noise and missing values to predict evapotranspiration in a mountainous watershed. The DAE reconstructed the sensor variables with a mean coefficient of determination value of 0.77 across 15 dimensions representing individual sensors. It reduced variance and bias uncertainties compared to a classical autoencoder model. The reconstruction quality varied across dimensions depending on their cross‐correlation and alignment with the underlying data structure. Uncertainties arising from the model structure were overall higher than those resulting from data corruption. We attached the DAE structure to a downstream ET‐prediction neural network in three formats and achieved reasonably accurate ET predictions . The use of the DAE notably reduced variance uncertainty in ET prediction. However, excessive variance reduction may be accompanied by an increase in bias due to the intrinsic bias‐variance tradeoff. Our method of evaluating and reducing uncertainties in aggregated data from different sources can be used to improve predictive models, process understanding, and uncertainty quantification for better water resource management. Plain Language Summary We present a machine learning method, namely the denoising autoencoder, which reduces the effects of data noise and missing values typically present in scientific data sets collected through sensor measurements. This method selects the most relevant information from noisy raw data collected by the instruments and fills in missing values. To demonstrate the effectiveness of our method, we applied it to predict evapotranspiration, a hydrologic variable that represents the water moved from the land surface to the atmosphere through a combination of evaporation and plant water use (transpiration). We also used a random sampling technique (the Monte Carlo method) to compare the uncertainty in the predictions when using the raw and noisy data versus the reconstructed data. The denoising process produced more accurate predictions of evapotranspiration with less uncertainty. Improved predictions of evapotranspiration can lead to a better understanding and accounting of water budgets. This ML approach is broadly suitable for a wide variety of applications that involve noisy sensor data with missing values. Key Points We used a denoising autoencoder (DAE) neural network to reduce noise in meteorological and soil sensor observations by on average We used Monte Carlo sampling to estimate the bias and variance of all model outputs, including uncertainty sources from data and the model We attached the DAE component to a downstream neural network to predict ET with the variance reduced by , compared to that without the DAE

denoising autoencoder↗
NASA NTRS2005

Fluctuations in Conjunction Miss Distance Projections as Time Approaches Time of Closest Approach

A responsibility of the Trajectory Operations Officer is to ensure that the International Space Station (ISS) avoids colliding with debris. United States Space Command (USSPACECOM) tracks and catalogs a portion of the debris in Earth orbit, but only objects with a perigee less than 600 km and a radar cross section (RCS) greater than 10 cm-objects that, in fact, represent only a small fraction of the objects in Earth orbit. To accommodate for this, the ISS uses shielding to protect against collisions with smaller objects. This study provides a better understanding of how quickly, and to what degree, USSPACECOM projections tend to converge to the final, true miss distance. The information included is formulated to better predict the behavior of miss distance data during real-time operations. It was determined that the driving components, in order of impact on miss distance fluctuations, are energy dissipation rate (EDR), RCS, and inclination. Data used in this analysis, calculations made, and conclusions drawn are stored in Microsoft Excel log sheets. A separate log sheet, created for each conjunction, contains information such as predicted miss distances, apogee and perigee of debris orbit, EDR, RCS, inclination, tracks and observations, statistical data, and other evaluation/orbital parameters.

Christian, John A., III↗
NASA NTRS2004

Clustering with Missing Values: No Imputation Required

Clustering algorithms can identify groups in large data sets, such as star catalogs and hyperspectral images. In general, clustering methods cannot analyze items that have missing data values. Common solutions either fill in the missing values (imputation) or ignore the missing data (marginalization). Imputed values are treated as just as reliable as the truly observed data, but they are only as good as the assumptions used to create them. In contrast, we present a method for encoding partially observed features as a set of supplemental soft constraints and introduce the KSC algorithm, which incorporates constraints into the clustering process. In experiments on artificial data and data from the Sloan Digital Sky Survey, we show that soft constraints are an effective way to enable clustering with missing values.

constraints↗
DOE OSTI2018

Using gap-filled MAIAC AOD and WRF-Chem to estimate daily PM 2.5 concentrations at 1 km resolution in the Eastern United States

To link short-term exposures of air pollutants to health outcomes, scientists must use high temporal and spatial resolution estimates of PM 2.5 concentrations. We develop a daily PM 2.5 product at 1 × 1 km 2 spatial resolution across the eastern United States (east of 90° W) with the aid of 1 × 1 km 2 MAIAC aerosol optical depth (AOD) data, 36 × 36 km 2 WRF-Chem output, 1 × 1 km 2 land-use type from the National Land Cover Database, and 0.125° × 0.125° ERA-Interim re-analysis meteorology. A gap-filling technique is applied to MAIAC AOD data to construct robust daily estimates of AOD when the satellite data are missing (e.g., areas obstructed by clouds or snow cover). The input data are incorporated into a multiple-linear regression model trained to surface observations of PM 2.5 from the EPA Air Quality System (AQS) monitoring network. The model generates a high-fidelity estimate (r 2 = 0.75 using a 10-fold random cross-validation) of daily PM 2.5 throughout the eastern United States. Of the inputs to the statistical model, WRF-Chem output (r 2 = 0.66) is the most important contributor to the skill of the model. MAIAC AOD is also a strong contributor (r 2 = 0.52). Daily PM 2.5 output from our statistical model can be easily integrated into county-level epidemiological studies. The novelty of this project is that we are able to simulate PM 2.5 in a computationally efficient manner that is constrained to ground monitors, satellite data, and chemical transport model output at high spatial resolution (1 × 1 km 2 ) without sacrificing the temporal resolution (daily) or spatial coverage (>2,000,000 km 2 ).

47 OTHER INSTRUMENTATION↗
DOE OSTI2023

Leveraging Application Data Constraints to Optimize Database-Backed Web Applications

Exploiting the relationships among data is a classical query optimization technique. As persistent data is increasingly being created and maintained programmatically, prior work that infers data relationships from data statistics misses an important opportunity. We present Coco, the first tool that identifies data relationships by analyzing database-backed applications. Once identified, Coco leverages the constraints to optimize the application's physical design and query execution. Instead of developing a fixed set of predefined rewriting rules, Coco employs an enumerate-test-verify technique to automatically exploit the discovered data constraints to improve query execution. Each resulting rewrite is provably equivalent to the original query. Using 14 real-world web applications, our experiments show that Coco can discover numerous data constraints from code analysis and improve real-world application performance significantly.

Computer Science↗
NASA NTRS1980

Efficient telemetry format

Format would simplify ground processing of telemetry data. Also, missing minor frame would create error in only one set of source data instead of disrupting all sets. Format organizes data from various sources into autonomous blocks. Data are pre-processed, in effect, so main computer only needs to determine block type and process data set as batch.

Greenberg, E.↗
NASA NTRS2011

Modeling Ionosphere Environments: Creating an ISS Electron Density Tool

The International Space Station (ISS) maintains an altitude typically between 300 km and 400 km in low Earth orbit (LEO) which itself is situated in the Earth's ionosphere. The ionosphere is a region of partially ionized gas (plasma) formed by the photoionization of neutral atoms and molecules in the upper atmosphere of Earth. It is important to understand what electron density the spacecraft is/will be operating in because the ionized gas along the ISS orbit interacts with the electrical power system resulting in charging of the vehicle. One instrument that is already operational onboard the ISS with a goal of monitoring electron density, electron temperature, and ISS floating potential is the Floating Potential Measurement Unit (FPMU). Although this tool is a valuable addition to the ISS, there are limitations concerning the data collection periods. The FPMU uses the Ku band communication frequency to transmit data from orbit. Use of this band for FPMU data runs is often terminated due to necessary observation of higher priority Extravehicular Activities (EVAs) and other operations on ISS. Thus, large gaps are present in FPMU data. The purpose of this study is to solve the issue of missing environmental data by implementing a secondary electron density data source, derived from the COSMIC satellite constellation, to create a model of ISS orbital environments. Extrapolating data specific to ISS orbital altitudes, we model the ionospheric electron density along the ISS orbit track to supply a set of data when the FPMU is unavailable. This computer model also provides an additional new source of electron density data that is used to confirm FPMU is operating correctly and supplements the original environmental data taken by FPMU.

Gurgew, Danielle N.↗
NASA NTRS2007

Thermal Control Materials on MISSE-5 with Comparison to Earlier Flight Data

A variety of thermal control materials were flown on the Materials on International Space Station Experiment (MISSE)-5. Several types of beta cloth, as used in multi-layer insulation blankets, were flown, including samples from the same batch as used on the International Space Station. Two candidate sunshade materials for the James Webb Space Telescope were also exposed on MISSE-5. The white thermal control coating AZ93 was applied to Kapton instead of aluminum; this sample maintained good solar absorptance and did not indicate any significant level of contamination to the MISSE-5 experiment. Marker coatings maintained their color. Thermo-optical properties are discussed, along with comparable data from MISSE-2 and the Passive Optical Sample Assembly (POSA) - I experiments.

Finckenor, Miria↗