Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data quality analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 595 records · Page 33
All sourcesNASA NTRSDOE OSTI
NASA NTRS1996

Constructing Space-Time Views from Fixed Size Statistical Data: Getting the Best of Both Worlds

Many performance monitoring tools are currently available to the super-computing community. The performance data gathered and analyzed by these tools fall under two categories: statistics and event traces. Statistical data is much more compact but lacks the probative power event traces offer. Event traces, on the other hand, can easily fill up the entire file system during execution such that the instrumented execution may have to be terminated half way through. In this paper, we propose an innovative methodology for performance data gathering and representation that offers a middle ground. The user can trade-off tracing overhead, trace data size vs. data quality incrementally. In other words, the user will be able to limit the amount of trace collected and, at the same time, carry out some of the analysis event traces offer using spacetime views for the entire execution. Two basic ideas are employed: the use of averages to replace recording data for each instance and "formulae" to represent sequences associated with communication and control flow. With the help of a few simple examples, we illustrate the use of these techniques in performance tuning and compare the quality of the traces we collected vs. event traces. We found that the trace files thus obtained are, in deed, small, bounded and predictable before program execution and that the quality of the space time views generated from these statistical data are excellent. Furthermore, experimental results showed that the formulae proposed were able to capture 100% of all the sequences associated with 11 of the 15 applications tested. The performance of the formulae can be incrementally improved by allocating more memory at run-time to learn longer sequences.

Schmidt, Melisa↗
NASA NTRS2018

Extreme Air Pollution Events in Hokkaido, Japan Traced Back to Early Snowmelt and Large-Scale Wildfires Over East Eurasia: Case Studies

To identify the unusual climate conditions and their connections to air pollutions in a remote area due to wildfires, we examine three anomalous large-scale wildfires in May 2003, April 2008, and July 2014 over East Eurasia, as well as how products of those wildfires reached an urban city, Sapporo, in the northern part of Japan (Hokkaido), significantly affecting the air quality. NASA's MERRA-2 (the Modern-Era Retrospective analysis for Research and Applications, Version 2) aerosol re-analysis data closely reproduced the PM2.5 variations in Sapporo for the case of smoke arrival inJuly 2014. Results show that all three cases featured unusually early snowmelt in East Eurasia, accompanied by warmer and drier surface conditions in the months leading to the fires, inducing long-lasting soil dryness and producing environmental conditions conductive to active wildfires. Due to prevailing anomalous synoptic-scale atmospheric motions, smoke from those fires eventually reached a remote area, Hokkaido, and worsened the air quality in Sapporo. In futurestudies, continuous monitoring of the timing of Eurasian snowmelt and the air quality from the source regions to remote regions, coupled with the analysis of atmospheric and surface conditions, may be essential in more accurately predicting the effects of wildfires on air quality.

MERRA-2↗
DOE OSTI2020

Optimization of Energy Flow through Synthetic Metabolic Modules and Regulatory Networks in a Model Photosynthetic Eukaryotic Microbe

Photosynthetic organisms have recently gained considerable attention for a role in development of renewable energy sources. Genome-enabled systems biology methods, coupled with functional and synthetic genomics, present opportunities to develop sustainable and economical applications such as fuel production within the next 10 to 15 years. However, optimization of light-driven metabolism for biomass or biofuel production will require a detailed systems biology understanding of photosynthetic processes and cellular metabolism. Genome-scale metabolic models (GEMs) are at the core of systems analysis of cellular processes and form a common organizational framework for analyses of data resulting from functional genomics experimental work and computational studies. Therefore, there is a clear demand for high quality photosynthetic model organisms and the appropriate computational tools that enable systems analysis of light-driven metabolism. Through research conducted we expanded the currently available repertoire of photosynthetic GEMs to include the commercially valuable model diatom Phaeoctylum tricornutum. Diatoms have a peculiar and distinct evolutionary footprint and represent a major eukaryotic lineage that is taxonomically and functionally distinct from green and red algae and vascular plants. Therefore, the true potential for light-driven metabolism aimed at biofuel production remains poorly understood at a systems level for a large subset of the global diversity of photosynthetic organisms. The metabolic capabilities of P. tricornutum were comparatively modeled with those from other photosynthetic groups in order to elucidate the occurrence of metabolic traits within and between phototrophs. Additionally, this research resulted in significant extension of the COnstraints Based Reconstruction and Analysis (COBRA) Toolbox to accommodate the crucial need for infrastructure required for ‘omics data integration and analysis in the context of genome-scale models. Therefore, the proposed research achieved two important goals. First, within the broad scope of photosynthetic organisms, we functionally compared and, as a result, identified cellular processes that require optimization in order to enable deployment as biofuel feedstock. Second, the proposed research resulted in development of key computational infrastructure, which can be further extended to other biological systems, that is currently lacking but necessary for multiple ‘omics data integration.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

Assessment of BQ-9000 Biodiesel Properties for 2023

This is the seventh in a series of reports documenting the quality of biodiesel from U.S.- and Canadian-based producers that participate in the BQ-9000 program, the biodiesel industry’s voluntary quality assurance program. Participants provided monthly data on critical quality parameters for calendar year 2023 with quality data provided to a team of experts, who removed any identifying company information and provided anonymized data to the National Renewable Energy Laboratory (NREL) for statistical analysis. New for 2023, data on kinematic viscosity, sulfated ash, distillation temperature, carbon residue, and cetane were collected, as well as individual levels of sodium, potassium, calcium, and magnesium. Critical quality parameters analyzed are listed in Table ES-1 with descriptive statistics.

09 BIOMASS FUELS↗
NASA NTRS2008

Atmospheric Correction at AERONET Locations: A New Science and Validation Data Set

This paper describes an AERONET-based Surface Reflectance Validation Network (ASRVN) and its dataset of spectral surface bidirectional reflectance and albedo based on MODIS TERRA and AQUA data. The ASRVN is an operational data collection and processing system. It receives 50x50 square kilometer subsets of MODIS L1B data from MODAPS and AERONET aerosol and water vapor information. Then it performs an accurate atmospheric correction for about 100 AERONET sites based on accurate radiative transfer theory with high quality control of the input data. The ASRVN processing software consists of L1B data gridding algorithm, a new cloud mask algorithm based on a time series analysis, and an atmospheric correction algorithm. The atmospheric correction is achieved by fitting the MODIS top of atmosphere measurements, accumulated for 16-day interval, with theoretical reflectance parameterized in terms of coefficients of the LSRT BRF model. The ASRVN takes several steps to ensure high quality of results: 1) cloud mask algorithm filters opaque clouds; 2) an aerosol filter has been developed to filter residual semi-transparent and sub-pixel clouds, as well as cases with high inhomogeneity of aerosols in the processing area; 3) imposing requirement of consistency of the new solution with previously retrieved BRF and albedo; 4) rapid adjustment of the 16-day retrieval to the surface changes using the last day of measurements; and 5) development of seasonal back-up spectral BRF database to increase data coverage. The ASRVN provides a gapless or near-gapless coverage for the processing area. The gaps, caused by clouds, are filled most naturally with the latest solution for a given pixels. The ASRVN products include three parameters of LSRT model (k(sup L), k(sup G), k(sup V)), surface albedo, NBRF (a normalized BRF computed for a standard viewing geometry, VZA=0 deg., SZA=45 deg.), and IBRF (instantaneous, or one angle, BRF value derived from the last day of MODIS measurement for specific viewing geometry) for MODIS 500m bands 1-7. The results are produced daily at resolution of 1 km in gridded format. We also provide cloud mask, quality flag and a browse bitmap image. The new dataset can be used for a wide range of applications including validation analysis and science research.

Wang, Yujie↗
NASA NTRS1990

Global variability of water vapor and condensate from SSM/I

In the three years since its launch, the Special Sensor Microwave Imager (SSM/I) provided the beginnings of a data set crucial to the study of the Earth's global hydrologic cycle. Investigations with SSM/I centered largely on two themes: (1) an analysis of the variability in atmospheric moisture as a signature of dynamic and kinematic processes, and (2) development of high quality data sets for determining moisture balance and, eventually, diabatic heating on regional and planetary scales. To date, most of the emphasis has been on the precipitable water field since algorithms for this quantity appear to be the most mature. In addition, researchers began work with liquid water, precipitating ice and ocean surface wind stress fields. The retrievals used in these studies are from algorithms developed by Wentz (RSS Tech Rpt. 1989) and Spencer (1988 JTECH).

Robertson, Franklin R.↗
NASA NTRS1998

Incorporating Parallel Computing into the Goddard Earth Observing System Data Assimilation System (GEOS DAS)

Atmospheric data assimilation is a method of combining actual observations with model forecasts to produce a more accurate description of the earth system than the observations or forecast alone can provide. The output of data assimilation, sometimes called the analysis, are regular, gridded datasets of observed and unobserved variables. Analysis plays a key role in numerical weather prediction and is becoming increasingly important for climate research. These applications, and the need for timely validation of scientific enhancements to the data assimilation system pose computational demands that are best met by distributed parallel software. The mission of the NASA Data Assimilation Office (DAO) is to provide datasets for climate research and to support NASA satellite and aircraft missions. The system used to create these datasets is the Goddard Earth Observing System Data Assimilation System (GEOS DAS). The core components of the the GEOS DAS are: the GEOS General Circulation Model (GCM), the Physical-space Statistical Analysis System (PSAS), the Observer, the on-line Quality Control (QC) system, the Coupler (which feeds analysis increments back to the GCM), and an I/O package for processing the large amounts of data the system produces (which will be described in another presentation in this session). The discussion will center on the following issues: the computational complexity for the whole GEOS DAS, assessment of the performance of the individual elements of GEOS DAS, and parallelization strategy for some of the components of the system.

Larson, Jay W.↗
DOE OSTI2023

Combining cosmic shear data with correlated photo-z uncertainties: constraints from DESY1 and HSC-DR1

An accurate calibration of the source redshift distribution p(z) is a key aspect in the analysis of cosmic shear data. This, one way or another, requires the use of spectroscopic or high-quality photometric samples. However, the difficulty to obtain colour-complete spectroscopic samples matching the depth of weak lensing catalogs means that the analyses of different cosmic shear datasets often use the same samples for redshift calibration. This introduces a source of statistical and systematic uncertainty that is highly correlated across different weak lensing datasets, and which must be accurately characterised and propagated in order to obtain robust cosmological constraints from their combination. In this paper we introduce a method to quantify and propagate the uncertainties on the source redshift distribution in two different surveys sharing the same calibrating sample. The method is based on an approximate analytical marginalisation of the p(z) statistical uncertainties and the correlated marginalisation of residual systematics. We apply this method to the combined analysis of cosmic shear data from the DESY1 data release and the HSC-DR1 data, using the COSMOS 30-band catalog as a common redshift calibration sample. We find that, although there is significant correlation in the uncertainties on the redshift distributions of both samples, this does not change the final constraints on cosmological parameters significantly. The same is true also for the impact of residual systematic uncertainties from the errors in the COSMOS 30-band photometric redshifts. Additionally, we show that these effects will still be negligible in Stage-IV datasets. Finally, the combination of DESY1 and HSC-DR1 allows us to constrain the "clumpiness" parameter to S 8 = ${0.768}_{-0.017}^{+0.021}$. This corresponds to a ~√(2) improvement in uncertainties with respect to either DES or HSC alone.

79 ASTRONOMY AND ASTROPHYSICS↗
NASA NTRS2004

Precipitation Analysis at Fine Time Scales Using Multiple Satellites: Real-time and Research Products and Applications

Quasi-global precipitation analyses at fine time scales (3-hr) are described. TRMM observations (radar and passive microwave) are used to calibrate polar-orbit microwave observations from SSM/I (and other satellites instruments, including AMSR and AMSU) and geosynchronous IR observations. The individual data sets are then merged using a priority order based on quality to form the Multi-satellite Precipitation Analysis (MPA). Raingauge information is used to help constrain the satellite-based estimates over land. The TRMM standard research product (Version 6 3B-42 of the TRMM products) will be available for the entire TRMM period (January 1998-present) in 2004. The real-time version of this merged product has been produced over the past two years and is available on the U.S. TRMM web site (trmm.gsfc.nasa.gov) at 0.25" latitude-longitude resolution over the latitude range from 5O"N-5O0S. Validation of daily totals indicates good results, with limitations noted in mid-latitude winter over land and regions of shallow, orographic precipitation. Various applications of these estimates are described, including: 1) detecting potential floods in near real-time; 2) analyzing Indian Ocean precipitation variations related to the initiation of El Nino; 3) determining characteristics of the African monsoon; and 4) analysis of diurnal variations.

Adler, Robert↗
NASA NTRS1997

Constructing Space-Time Views from Fixed Size Statistical Data: Getting the Best of both Worlds

Many performance monitoring tools are currently available to the super-computing community. The performance data gathered and analyzed by these tools fall under two categories: statistics and event traces. Statistical data is much more compact but lacks the probative power event traces offer. Event traces, on the other hand, can easily fill up the entire file system during execution such that the instrumented execution may have to be terminated half way through. In this paper, we propose an innovative methodology for performance data gathering and representation that offers a middle ground. The user can trade-off tracing overhead, trace data size vs. data quality incrementally. In other words, the user will be able to limit the amount of trace collected and, at the same time, carry out some of the analysis event traces offer using space-time views for the entire execution. Two basic ideas arc employed: the use of averages to replace recording data for each instance and formulae to represent sequences associated with communication and control flow. With the help of a few simple examples, we illustrate the use of these techniques in performance tuning and compare the quality of the traces we collected vs. event traces. We found that the trace files thus obtained are, in deed, small, bounded and predictable before program execution and that the quality of the space time views generated from these statistical data are excellent. Furthermore, experimental results showed that the formulae proposed were able to capture 100% of all the sequences associated with 11 of the 15 applications tested. The performance of the formulae can be incrementally improved by allocating more memory at run-time to learn longer sequences.

Schmidt, Melisa↗
NASA NTRS1980

Evaluation of SLAR and thematic mapper MSS data for forest cover mapping using computer-aided analysis techniques

Several possibilities were considered for defining the data set in which the same test areas could be used for each of the four different spatial resolutions being evaluated. The LARSYS CLUSTER was used to sort the vectors into spectral classes to reduce the within-spectral class variability in an effort to develop training statistics. A data quality test was written to determine the basic signal to noise characteristics within the data set being used. Because preliminary analysis of the LANDSAT MSS data revealed the presence of high cirrus clouds, other data sets are being sought.

Hoffer, R. M.↗
NASA NTRS1992

SkICAT: A cataloging and analysis tool for wide field imaging surveys

We describe an integrated system, SkICAT (Sky Image Cataloging and Analysis Tool), for the automated reduction and analysis of the Palomar Observatory-ST ScI Digitized Sky Survey. The Survey will consist of the complete digitization of the photographic Second Palomar Observatory Sky Survey (POSS-II) in three bands, comprising nearly three Terabytes of pixel data. SkICAT applies a combination of existing packages, including FOCAS for basic image detection and measurement and SAS for database management, as well as custom software, to the task of managing this wealth of data. One of the most novel aspects of the system is its method of object classification. Using state-of-theart machine learning classification techniques (GID3* and O-BTree), we have developed a powerful method for automatically distinguishing point sources from non-point sources and artifacts, achieving comparably accurate discrimination a full magnitude fainter than in previous Schmidt plate surveys. The learning algorithms produce decision trees for classification by examining instances of objects classified by eye on both plate and higher quality CCD data. The same techniques will be applied to perform higher-level object classification (e.g., of galaxy morphology) in the near future. Another key feature of the system is the facility to integrate the catalogs from multiple plates (and portions thereof) to construct a single catalog of uniform calibration and quality down to the faintest limits of the survey. SkICAT also provides a variety of data analysis and exploration tools for the scientific utilization of the resulting catalogs. We include initial results of applying this system to measure the counts and distribution of galaxies in two bands down to Bj is approximately 21 mag over an approximate 70 square degree multi-plate field from POSS-II. SkICAT is constructed in a modular and general fashion and should be readily adaptable to other large-scale imaging surveys.

Weir, N.↗
NASA NTRS2019

Analysis Ready Data in Analytics Optimized Data Stores for Analysis of Big Earth Data in the Cloud

Cloud computing offers the possibility of making the analysis of Big Data approachable for a wider community due to affordable access to computing power, an ecosystem of usable tools for parallel processing, and migration of many large datasets to archives in the cloud, allowing data-proximal computing. Generally, data analysis acceleration in the cloud comes from running multiple nodes in a split-combine-apply strategy. Data systems such as the Earth Observing System Data and Information System are in a position to "pre-split" the data by storing them in a data store that is optimized for data parallel computing, i.e., an Analytics-Optimized Data Store (AODS). A variety of approaches to AODS are possible, from highly scalable databases to scalable filesystems to data formats optimized for cloud access (e.g., zarr and cloud-optimized datasets), with the optimal choice dependent on both the types of analysis and the geospatial structure of the data. A key question is how much preprocessing of the data to do, both before splitting and as the first part of the apply step. Again, the geospatial structure of the data and the analysis type influence the decision, with the added complexity of the user type. Trans-disciplinary users who are not well-versed in the nuances of quality-filtering and georeferencing of remote sensing orbit/swath/scene data tend to ask for more highly processed data, relying on the data provider to make sensible decisions on preprocessing parameters. (This accounts for the popularity of "Level 3" gridded data, despite the lower spatial resolution it provides.) In this case, data can be preprocessed before the split, resulting in higher performance in the rest of the "apply" step, which can be transformative for use cases such as interactive data exploration at scale. Discipline researchers who are experienced with remote sensing data often prefer more flexibility in customizing the preprocessing data into Analysis Ready Data, resulting in more need for on-the-fly preprocessing.

Lynnes, Christopher↗
DOE OSTI2021

Cinema:Snap: Real-time tools for analysis of dynamic diamond anvil cell experiment data

We report we developed tools and a workflow for real-time analysis of data from dynamic diamond anvil cell experiments performed at user light sources. These tools allow users to determine the phases of matter observed during the compression of materials in order to make decisions during an experiment to improve the quality of experimental results and maximize the use of scarce experimental facility time. The tools fill a gap in dynamic compression data analysis tools that are real-time, are flexible to the needs of high-pressure scientists, connect to automated processing of results, can be easily incorporated into workflows with existing tools and data formats, and support remote experimental data analysis workflows. Specific analytics developed include novel automated two-peak analysis for overlapping peaks and multiple phases, coordinated views of pressure and temperature values, full-compression contour plots, and configurable views of integrated x-ray diffraction. We present an experimental use case to show how the tools produce real-time analytics that help the scientists revise parameters for the next compression

47 OTHER INSTRUMENTATION↗
DOE OSTI2024

Carbon fiber classification using raman spectroscopy

Carbon fiber characterization processes are described that include multi-condition Raman spectroscopy-based examination combined with multivariate data analyses. Methods are a nondestructive material characterization approach that can provide predictions as to carbon fiber bulk physical properties, as well as identification of unknown carbon fiber materials for quality control purposes. The framework of the multivariate analysis methods includes a principal component-based identification protocol including comparison of Raman spectral data from an unknown carbon fiber with a data library of multiple principal component spaces.

Houk, Amanda L.↗
NASA NTRS1974

Use of Whatman-41 filters in air quality sampling networks (with applications to elemental analysis)

The operation of a 16-site parallel high volume air sampling network with glass fiber filters on one unit and Whatman-41 filters on the other is reported. The network data and data from several other experiments indicate that (1) Sampler-to-sampler and filter-to-filter variabilities are small; (2) hygroscopic affinity of Whatman-41 filters need not introduce errors; and (3) suspended particulate samples from glass fiber filters averaged slightly, but not statistically significantly, higher than from Whatman-41-filters. The results obtained demonstrate the practicability of Whatman-41 filters for air quality monitoring and elemental analysis.

Neustadter, H. E.↗
NASA NTRS1983

A modal analysis of flexible aircraft dynamics with handling qualities implications

A multivariable modal analysis technique is presented for evaluating flexible aircraft dynamics, focusing on meaningful vehicle responses to pilot inputs and atmospheric turbulence. Although modal analysis is the tool, vehicle time response is emphasized, and the analysis is performed on the linear, time-domain vehicle model. In evaluating previously obtained experimental pitch tracking data for a family of vehicle dynamic models, it is shown that flexible aeroelastic effects can significantly affect pitch attitude handling qualities. Consideration of the eigenvalues alone, of both rigid-body and aeroelastic modes, does not explain the simulation results. Modal analysis revealed, however, that although the lowest aeroelastic mode frequency was still three times greater than the short-period frequency, the rigid-body attitude response was dominated by this aeroelastic mode. This dominance was defined in terms of the relative magnitudes of the modal residues in selected vehicle responses.

Schmidt, D. K.↗
NASA NTRS1986

OSAC analysis of the Far Ultraviolet Spectroscopic Explorer (FUSE) telescope

An investigation is made of the sensitivity of the image quality for the proposed FUSE telescope to mirror misalignments and a wide spatial frequency range of figure errors. Representative figure error data was obtained for the analysis from measurements made on the SEUTS (Solar Extreme Ultraviolet Telescope Spectrograph) telescope mirrors. The tolerancing analysis was carried out with the aid of the Optical Surface Analysis Code (OSAC) program.

Saha, Timo T.↗