Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Missing data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7
All sourcesNASA NTRSDOE OSTI
DOE OSTI2018

A Kriging based spatiotemporal approach for traffic volume data imputation

Along with the rapid development of Intelligent Transportation Systems, traffic data collection technologies have progressed fast. The emergence of innovative data collection technologies such as remote traffic microwave sensor, Bluetooth sensor, GPS-based floating car method, and automated license plate recognition, has significantly increased the variety and volume of traffic data. Despite the development of these technologies, the missing data issue is still a problem that poses great challenge for data based applications such as traffic forecasting, real-time incident detection, dynamic route guidance, and massive evacuation optimization. A thorough literature review suggests most current imputation models either focus on the temporal nature of the traffic data and fail to consider the spatial information of neighboring locations or assume the data follow a certain distribution. These two issues reduce the imputation accuracy and limit the use of the corresponding imputation methods respectively. As a result, this paper presents a Kriging based data imputation approach that is able to fully utilize the spatiotemporal correlation in the traffic data and that does not assume the data follow any distribution. A set of scenarios with different missing rates are used to evaluate the performance of the proposed method. The performance of the proposed method was compared with that of two other widely used methods, historical average and K-nearest neighborhood. Comparison results indicate that the proposed method has the highest imputation accuracy and is more flexible compared to other methods.

Science & Technology - Other Topics↗
NASA NTRS2015

Astronaut Preflight Cardiovascular Variables Associated with Vascular Compliance are Highly Correlated with Post-Flight Eye Outcome Measures in the Visual Impairment Intracranial Pressure (VIIP) Syndrome Following Long Duration Spaceflight

The detection of the first VIIP case occurred in 2005, and adequate eye outcome measures were available for 31 (67.4%) of the 46 long duration US crewmembers who had flown on the ISS since its first crewed mission in 2000. Therefore, this analysis is limited to a subgroup (22 males and 9 females). A "cardiovascular profile" for each astronaut was compiled by examining twelve individual parameters; eleven of these were preflight variables: systolic blood pressure, pulse pressure, body mass index, percentage body fat, LDL, HDL, triglycerides, use of anti‐lipid medication, fasting serum glucose, and maximal oxygen uptake in ml/kg. Each of these variables was averaged across three preflight annual physical exams. Astronaut age prior to the long duration mission, and inflight salt intake was also included in the analysis. The group of cardiovascular variables for each crew member was compared with seven VIIP eye outcome variables collected during the immediate post‐flight period: anterior-posterior axial length of the globe measured by ultrasound and optical biometry; optic nerve sheath diameter, optic nerve diameter, and optic nerve to sheath ratio‐ each measured by ultrasound and magnetic resonance imaging (MRI), intraocular pressure (IOP), change in manifest refraction, mean retinal nerve fiber layer (RNFL) on optical coherence tomography (OCT), and RNFL of the inferior and superior retinal quadrants. Since most of the VIIP eye outcome measures were added sequentially beginning in 2005, as knowledge of the syndrome improved, data were unavailable for 22.0% of the outcome measurements. To address the missing data, we employed multivariate multiple imputation techniques with predictive mean matching methods to accumulate 200 separate imputed datasets for analysis. We were able to impute data for the 22.0% of missing VIIP eye outcomes. We then applied Rubin's rules for collapsing the statistical results across our 200 multiply imputed data sets to assess the canonical correlation between the eye outcomes and the twelve astronaut cardiovascular variables available for all 31 subjects. Results: A highly significant canonical correlation was observed among the canonical solutions (p<.00001), with an average best canonical correlation of.97. The results suggest a strong association between astronauts' measures of cardiovascular health and the seven eye outcomes of the VIIP syndrome used in this analysis. Furthermore, the "joint test" revealed a significant difference in cardiovascular profile between male and female astronauts (Prob > F = 0.00001). Overall, female astronauts demonstrated a significantly healthier cardiovascular status. Individually, the female astronauts had significantly healthier profiles on seven of twelve cardiovascular variables than the men (p values ranging from <0.0001 to <0.05). Male astronauts did not demonstrate significantly healthier values on any of the twelve cardiovascular variables measured

Otto, Christian↗
DOE OSTI2015

SHELXT – Integrated space-group and crystal-structure determination

SHELXT automates routine small-molecule structure determination starting from single-crystal reflection data, the Laue group and a reasonable guess as to which elements might be present. The new computer program SHELXT employs a novel dual-space algorithm to solve the phase problem for single-crystal reflection data expanded to the space group P1. Missing data are taken into account and the resolution extended if necessary. All space groups in the specified Laue group are tested to find which are consistent with the P1 phases. After applying the resulting origin shifts and space-group symmetry, the solutions are subject to further dual-space recycling followed by a peak search and summation of the electron density around each peak. Elements are assigned to give the best fit to the integrated peak densities and if necessary additional elements are considered. An isotropic refinement is followed for non-centrosymmetric space groups by the calculation of a Flack parameter and, if appropriate, inversion of the structure. The structure is assembled to maximize its connectivity and centred optimally in the unit cell. SHELXT has already solved many thousand structures with a high success rate, and is optimized for multiprocessor computers. It is, however, unsuitable for severely disordered and twinned structures because it is based on the assumption that the structure consists of atoms.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2022

A Review of the Use of Wearables in Indoor Environmental Quality Studies and an Evaluation of Data Accessibility from a Wearable Device

An understanding of indoor environmental quality (IEQ) and its effects on occupant well-being can inform building system design and operation. The use of wearables in field studies to collect subjective and objective health performance indicators (HPIs) from a large number of occupants could deliver important improvements in IEQ. To facilitate the use of wearables in IEQ studies, there is a need to identify which HPIs should be collected and to evaluate data accessibility from these devices. To address this issue, a literature review of previous IEQ studies was conducted to identify relationships between different IEQ factors and HPIs, with a focus on HPIs that were collected using wearables. A preliminary assessment of data accessibility from a selected wearable device (Fitbit Versa 2) was performed and documented. The review suggested the need to further investigate and collect sleep quality parameters, heart rate, stress response, as well as subjective ratings of comfort using wearables. The data accessibility assessment revealed issues related to missing data points and data resolution from the examined device. A set of recommendations is outlined to inform future studies.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2018

AAF_UHSAS-air_CACTI

The UHSAS-air is the bench version of the UHSAS that measures particle size from 0.06 to 1.0 micro-meters. Aerosol passes through a laser beam and the resulting intensity of scattered light is directly correlated to the diameter of the particle passing through the beam. This isntrument has been modified to volumetricaly control the Sheath Flow using an Alicat flow controller. Data is reported at 1.000s and missing data is reported as -9999 Multiple data flags are present in this version. Counts?: The coincident error is deemed minimal until the counts exceed 4000. Counts between 3000 and 4000 are flagged with a caution flag (1). After 4000 counts a bad flag (2) is triggered and the data should not be used. Noise?: Instrument noise detected in 65-75nm size range. 1: Caution slight noise observed. 2: High amount of noise observed data should not be used. Total? Sum of all the flags triggered Max Flag: Max flag triggered by the above QC test CloudFlag: Warns of the presence of a cloud. Clouds can cause shattering on the probe and result in questionable measurements. Data during these periods of thin clouds (1) or heavy clouds (2) should be used with extreme caution.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS1972

Data acquisition and processing history for the Explorer 33 (AIMP-D) satellite

The quality control monitoring system, using accounting and quality control data bases, made it possible to perform an in-depth analysis. Results show that the percentage of useable data files for experimenter analysis was 97.7%; only 0.4% of the data sequences supplied to the experimenter exhibited missing data. The 50 percentile probability delay values (referenced to station record data) indicate that the analog tapes arrived within 11 days, the data were digitized within 4.2 weeks, and the experimenter tapes were delivered in 8.95 weeks or less.

Karras, T. J.↗
NASA NTRS1992

An interactive environment for the analysis of large Earth observation and model data sets

We propose to develop an interactive environment for the analysis of large Earth science observation and model data sets. We will use a standard scientific data storage format and a large capacity (greater than 20 GB) optical disk system for data management; develop libraries for coordinate transformation and regridding of data sets; modify the NCSA X Image and X Data Slice software for typical Earth observation data sets by including map transformations and missing data handling; develop analysis tools for common mathematical and statistical operations; integrate the components described above into a system for the analysis and comparison of observations and model results; and distribute software and documentation to the scientific community.

Bowman, Kenneth P.↗
NASA NTRS1993

An interactive environment for the analysis of large Earth observation and model data sets

We propose to develop an interactive environment for the analysis of large Earth science observation and model data sets. We will use a standard scientific data storage format and a large capacity (greater than 20 GB) optical disk system for data management; develop libraries for coordinate transformation and regridding of data sets; modify the NCSA X Image and X DataSlice software for typical Earth observation data sets by including map transformations and missing data handling; develop analysis tools for common mathematical and statistical operations; integrate the components described above into a system for the analysis and comparison of observations and model results; and distribute software and documentation to the scientific community.

Bowman, Kenneth P.↗
DOE OSTI2018

3D Visualisation of Un-georeferenced, Coordinate-poor Spatial Data in Complex D and D and WM Environments - 18294

This paper describes the development of an algorithm-based utility tool using Visual Basic that examines un-georeferenced or GPS-deprived spatial datasets (3D datasets lacking one or more of the key x, y and z components). Taking into account any known geometrical parameters which are input via a user-friendly front-end, the new code automatically evaluates attributes, including any spatial coordinate components, detects invalid survey data points and fills in bad or missing data points utilising data-specific interpolation (e.g. cubic or linear spline) of valid data points, such that local extrema are not created. Previously troublesome 2D data can be easily converted to realistic 3D contour display, integrated with other time-series data to create 4D movies, and incorporated into other existing 3D datasets. More importantly those processed and 'more complete' data can be easily stored in 3D or 4D spatial database management systems for future interrogation by using clear, realistic and intuitive 3D visualisation. Test cases are presented to demonstrate the value and versatility of the newly developed utility tool which include the 3D visualisation of: surveyed wall thicknesses of a helical coil, radiation mapping of a plant and aerial survey mapping of radiation in a region near Fukushima. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2023

Pair distribution function analysis of nano-object assemblies

The atomic pair distribution function (aPDF) analysis technique, also known as the total scattering method, which considers both Bragg and diffuse scattering, has been used extensively to probe local atomic arrangements in crystalline and disordered materials. In contrast, there have been limited applications of the PDF in self-assembled nanomaterials, which represent a class of materials built from nanoscale objects, such as nano-colloids, micelles and proteins. As distinguished from atoms, nano-objects have polydispersity in size and shape, and such form-factor effects complicate the application of PDF analysis to nano-systems. Here, in this paper, the application of the PDF is extended to spherical nano-object assemblies and the formulae for the nano-PDF (nPDF) are derived, showing some differences from the aPDF. By numerical simulations, the properties of the nPDF (peak broadening and pattern profile) are studied systematically as a function of structural features, such as nano-object parameters (size and size polydispersity) and assembly structural features (size, shape, structure type and lattice disorder), and of data processing parameters ( q cut-off and `missing' data in ultra-small-angle regions). The nPDF analysis method is found to provide an effective route to revealing not only nanoscale but also mesoscale structural properties, for example the morphology of a nano-assembly.

77 NANOSCIENCE AND NANOTECHNOLOGY↗
DOE OSTI2015

Tall Tower Wind Energy Monitoring and Numerical Model Validation in Northern Nevada

The main objectives of this project were to conduct a tall-tower and sodar field campaign in complex terrain, investigate wind properties relevant to wind energy assessment, and evaluate high-resolution models with fixed and adaptive grid structures. Two 60-m towers at Virginia Peak ridges near Washoe Valley, Nevada, were instrumented with cup and vane anemometers as well as sonic anemometers, and an acoustic sounder (hereafter sodar) was installed near one of the towers. The towers were located 2,700 m apart with a vertical distance of 140 m elevation between their bases. Each tower had a downhill exposure of rolling complex terrain, with the nearby valley floor 3,200 m to the west and 800 m below the summit. Cup anemometers were installed at both towers at 20, 40, and 60 m, wind vanes at 20 and 60 m, and sonic anemometers at 20 and 60 m. The sodar measurements were nominally provided every 10 m in vertical distance from 40 to 200 m with the quality of the data generally decreasing with height. Surface air temperature, atmospheric pressure, and radiation measurements were conducted at 1.5 m AGL at both of the towers. Although the plan was to conduct a 1-year period of data collection, we extended the period (October 5, 2012 through February 24, 2014) to cover for possible data loss from instrument or communication problems. We also present a preliminary analysis of the towers and sodar data, including a detailed inventory of available and missing data as well as outliers. The analysis additionally includes calculation of the Weibull parameters, turbulence intensity, and initial computation of wind power density at various heights.

17 WIND ENERGY↗
DOE OSTI2022

Accounting for uncertainty sources in pyrometric measurements: Variables affecting Surface Temperature measurements using the LLNL Streak Spectrometers

Here we present a detailed analysis of the error sensitivities of the LLNL temperature analysis package used in analyzing streak spectrometer data. These sensitivities include noise, temperature, wavelength, emissivity, missing bands, and missing data points. The analysis package was tested for its ability to accurately reconstruct randomly generated noisy data and found to perform within specifications.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
NASA NTRS1981

Holographic interferometry of transparent media with reflection from imbedded test objects

In applying holographic interferometry, opaque objects blocking a portion of the optical beam used to form the interferogram give rise to incomplete data for standard computer tomography algorithms. An experimental technique for circumventing the problem of data blocked by opaque objects is presented. The missing data are completed by forming an interferogram using light backscattered from the opaque object, which is assumed to be diffuse. The problem of fringe localization is considered.

Prikryl, I.↗
NASA NTRS1996

Variance Analysis of Unevenly Spaced Time Series Data

We have investigated the effect of uneven data spacing on the computation of delta (sub chi)(gamma). Evenly spaced simulated data sets were generated for noise processes ranging from white phase modulation (PM) to random walk frequency modulation (FM). Delta(sub chi)(gamma) was then calculated for each noise type. Data were subsequently removed from each simulated data set using typical two-way satellite time and frequency transfer (TWSTFT) data patterns to create two unevenly spaced sets with average intervals of 2.8 and 3.6 days. Delta(sub chi)(gamma) was then calculated for each sparse data set using two different approaches. First the missing data points were replaced by linear interpolation and delta (sub chi)(gamma) calculated from this now full data set. The second approach ignored the fact that the data were unevenly spaced and calculated delta(sub chi)(gamma) as if the data were equally spaced with average spacing of 2.8 or 3.6 days. Both approaches have advantages and disadvantages, and techniques are presented for correcting errors caused by uneven data spacing in typical TWSTFT data sets.

Hackman, Christine↗
NASA NTRS2008

Potential Use of a Bayesian Network for Discriminating Flash Type from Future GOES-R Geostationary Lightning Mapper (GLM) data

Continuous monitoring of the ratio of cloud flashes to ground flashes may provide a better understanding of thunderstorm dynamics, intensification, and evolution, and it may be useful in severe weather warning. The National Lighting Detection Network TM (NLDN) senses ground flashes with exceptional detection efficiency and accuracy over most of the continental United States. A proposed Geostationary Lightning Mapper (GLM) aboard the Geostationary Operational Environmental Satellite (GOES-R) will look at the western hemisphere, and among the lightning data products to be made available will be the fundamental optical flash parameters for both cloud and ground flashes: radiance, area, duration, number of optical groups, and number of optical events. Previous studies have demonstrated that the optical flash parameter statistics of ground and cloud lightning, which are observable from space, are significantly different. This study investigates a Bayesian network methodology for discriminating lightning flash type (ground or cloud) using the lightning optical data and ancillary GOES-R data. A Directed Acyclic Graph (DAG) is set up with lightning as a "root" and data observed by GLM as the "leaves." This allows for a direct calculation of the joint probability distribution function for the lighting type and radiance, area, etc. Initially, the conditional probabilities that will be required can be estimated from the Lightning Imaging Sensor (LIS) and the Optical Transient Detector (OTD) together with NLDN data. Directly manipulating the joint distribution will yield the conditional probability that a lightning flash is a ground flash given the evidence, which consists of the observed lightning optical data [and possibly cloud data retrieved from the GOES-R Advanced Baseline Imager (ABI) in a more mature Bayesian network configuration]. Later, actual GLM and NLDN data can be used to refine the estimates of the conditional probabilities used in the model; i.e., the Bayesian network is a learning network. Methods for efficient calculation of the conditional probabilities (e.g., an algorithm using junction trees), finding data conflicts, goodness of fit, and dealing with missing data will also be addressed.

Solakiewiz, Richard↗
DOE OSTI2016

A SPATIOTEMPORAL APPROACH FOR HIGH RESOLUTION TRAFFIC FLOW IMPUTATION

Along with the rapid development of Intelligent Transportation Systems (ITS), traffic data collection technologies have been evolving dramatically. The emergence of innovative data collection technologies such as Remote Traffic Microwave Sensor (RTMS), Bluetooth sensor, GPS-based Floating Car method, automated license plate recognition (ALPR) (1), etc., creates an explosion of traffic data, which brings transportation engineering into the new era of Big Data. However, despite the advance of technologies, the missing data issue is still inevitable and has posed great challenges for research such as traffic forecasting, real-time incident detection and management, dynamic route guidance, and massive evacuation optimization, because the degree of success of these endeavors depends on the timely availability of relatively complete and reasonably accurate traffic data. A thorough literature review suggests most current imputation models, if not all, focus largely on the temporal nature of the traffic data and fail to consider the fact that traffic stream characteristics at a certain location are closely related to those at neighboring locations and utilize these correlations for data imputation. To this end, this paper presents a Kriging based spatiotemporal data imputation approach that is able to fully utilize the spatiotemporal information underlying in traffic data. Imputation performance of the proposed approach was tested using simulated scenarios and achieved stable imputation accuracy. Moreover, the proposed Kriging imputation model is more flexible compared to current models.

Han, Lee↗
DOE OSTI2018

FUSRAP Case Study - Benefits and Lessons Learned from Implementation of a Modern Data Solution at Maywood - 18399

This paper focuses on data production and management practices dating back to 1999 that continue to evolve to support the project today. In 2000, the FUSRAP Maywood Project began using a software environmental data management system and laboratory information management system, developed by a single database administrator (DA) and authored in Microsoft Access{sup TM} (MS Access). The original databases included basic functionality including sample number assignment and label generation, tracking of LIMS gamma spec QC and assignment of basic qualifiers and basic database queries. The DA continued over the next fifteen years to refine the EDMS/LIMS by integrating project-specific data evaluation guidance document processes and Quality Assurance/Quality Control (QA/QC) charting and excursion warning tools. With the retirement of the original DA, the U.S. Army Corps of Engineers (USACE)-Cabrera Team procured and implemented a commercial-leaning solution using modern database tools and an established EDMS vendor. For the database transition process, the project team established and configured the new EDMS/LIMS with basic architecture and accessibility; evaluated existing workflows; established/revised LIMS and EDMS-related documentation (manuals and procedures); and conducted dry runs and validation testing. The overall team coordinated closely to facilitate a seamless transition from initial assessment and authoring, to data migration, and finally to resumption of normal ongoing operations. Key features of the new EDMS Subcontractor's software included lookup tables, an import wizard, data checking options and unique indexes. Since the database systems became operational, the technical staff gathered useful feedback from end users (e.g., lab staff, field engineers, task managers, and the third-party data validator) to refine the EDMS/LIMS operation. This feedback triggered improvements in the performance of onsite laboratory qualification forms, promoted consistency in sample ID numbers between the database sample planning screens and field chain-of-custodies, corrected bugs in the LIMS Duplicate Evaluation Report, eliminated data import issues such as blank, inconsistent or missing data from key fields. One of the advantages of the commercial database solution was that it allowed the project to cut cost over time due to the lower cost of database maintenance provided by the commercial database vendor versus the cost of an on-staff DA. Cost savings are estimated at $40,000 per year. There were also indirect cost savings resulting from implementation of the new database since it allowed better integration of the LIMS and EDMS. While there will always be challenges associated with long term records/data management for any old remediation site, the Maywood Team's approach to an upgraded EDMS/LIMS implementation serves as one element of the overall effort to enhance the integrity and reproducibility of critical data that supports the remediation program while at the same time, more effectively using available resources. These lessons-learned will enhance implementation of cost-effective solutions for similar environmental remediation sites. (authors)

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

FAIR data infrastructure and tools for AI-assisted streamflow prediction

Focal Area(s) Areas: We discuss how the integration of AI into Earth Science models can impact streamflow predictions at both the science and data levels. Doing so, we address cross-cutting needs related to the goal of making data FAIR (Findable, Accessible, Interoperable, and Re-usable [1]) for seamless use with Artificial Intelligence/Machine Learning (AI/ML) in Earth System Science at DOE. A novel idea is that AI/ML itself can help with the FAIR data goal and address issues in targeted areas e.g. missing data, data quality and reduction. In addition, the interpretability of results obtained with new AI methods is poised to impact broader scientific challenges in hydrology..

54 ENVIRONMENTAL SCIENCES↗