Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Error Metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10
All sourcesNASA NTRSDOE OSTI
NASA NTRS1992

Multiple symbol partially coherent detection of MPSK

It is shown that by using the known (or estimated) value of carrier tracking loop signal to noise ratio (SNR) in the decision metric, it is possible to improve the error probability performance of a partially coherent multiple phase-shift-keying (MPSK) system relative to that corresponding to the commonly used ideal coherent decision rule. Using a maximum-likeihood approach, an optimum decision metric is derived and shown to take the form of a weighted sum of the ideal coherent decision metric (i.e., correlation) and the noncoherent decision metric which is optimum for differential detection of MPSK. The performance of a receiver based on this optimum decision rule is derived and shown to provide continued improvement with increasing length of observation interval (data symbol sequence length). Unfortunately, increasing the observation length does not eliminate the error floor associated with the finite loop SNR. Nevertheless, in the limit of infinite observation length, the average error probability performance approaches the algebraic sum of the error floor and the performance of ideal coherent detection, i.e., at any error probability above the error floor, there is no degradation due to the partial coherence. It is shown that this limiting behavior is virtually achievable with practical size observation lengths. Furthermore, the performance is quite insensitive to mismatch between the estimate of loop SNR (e.g., obtained from measurement) fed to the decision metric and its true value. These results may be of use in low-cost Earth-orbiting or deep-space missions employing coded modulations.

Simon, M. K.↗
NASA NTRSDate not supplied

Urban Air Mobility Conflict Resolution: Centralized or Decentralized?

This work begins to address one of the critical questions in the urban air mobility and small unmanned aircraft communities: Should the en-route conflict resolution function in an urban air mobility traffic system be centralized or decentralized? Three conflict resolution architectures are modeled and analyzed: centralized, decentralized with uniform rules, and decentralized with mixed rules. This study compares these architectures and investigates their robustness to communication and state information errors in terms of safety and efficiency metrics. Experiments are conducted using a high-fidelity Monte Carlo traffic simulator and a generic set of traffic scenarios with increasing traffic density. When no errors were modeled, the centralized architecture marginally outperformed the decentralized architecture. However, performance of the centralized architecture was found to be adversely affected by the modeled input errors to a greater degree than was the decentralized architecture. Performance of the centralized architecture also was degraded significantly by the modeled transmission errors of the centralized resolution maneuvers. In the decentralized architecture, uniform rules outperformed mixed rules because, in the mixed rules case, system safety performance was undermined and dominated by the poor performers.

Urban Air Mobility (UAM) traffic system↗
DOE OSTI2023

Uncertain of uncertainties? A comparison of uncertainty quantification metrics for chemical data sets

Abstract With the increasingly more important role of machine learning (ML) models in chemical research, the need for putting a level of confidence to the model predictions naturally arises. Several methods for obtaining uncertainty estimates have been proposed in recent years but consensus on the evaluation of these have yet to be established and different studies on uncertainties generally uses different metrics to evaluate them. We compare three of the most popular validation metrics (Spearman’s rank correlation coefficient, the negative log likelihood (NLL) and the miscalibration area) to the error-based calibration introduced by Levi et al. ( Sensors 2022 , 22 , 5540). Importantly, metrics such as the negative log likelihood (NLL) and Spearman’s rank correlation coefficient bear little information in themselves. We therefore introduce reference values obtained through errors simulated directly from the uncertainty distribution. The different metrics target different properties and we show how to interpret them, but we generally find the best overall validation to be done based on the error-based calibration plot introduced by Levi et al. Finally, we illustrate the sensitivity of ranking-based methods (e.g. Spearman’s rank correlation coefficient) towards test set design by using the same toy model ferent test sets and obtaining vastly different metrics (0.05 vs. 0.65).

Rasmussen, Maria H.↗
DOE OSTI2023

Detecting macroevolutionary genotype–phenotype associations using error-corrected rates of protein convergence

On macroevolutionary timescales, extensive mutations and phylogenetic uncertainty mask the signals of genotype–phenotype associations underlying convergent evolution. To overcome this problem, we extended the widely used framework of non-synonymous to synonymous substitution rate ratios and developed the novel metric ω C , which measures the error-corrected convergence rate of protein evolution. While ω C distinguishes natural selection from genetic noise and phylogenetic errors in simulation and real examples, its accuracy allows an exploratory genome-wide search of adaptive molecular convergence without phenotypic hypothesis or candidate genes. Using gene expression data, we explored over 20 million branch combinations in vertebrate genes and identified the joint convergence of expression patterns and protein sequences with amino acid substitutions in functionally important sites, providing hypotheses on undiscovered phenotypes. We further extended our method with a heuristic algorithm to detect highly repetitive convergence among computationally non-trivial higher-order phylogenetic combinations. Our approach allows bidirectional searches for genotype–phenotype associations, even in lineages that diverged for hundreds of millions of years.

59 BASIC BIOLOGICAL SCIENCES↗
NASA NTRS2018

HZETRN Radiation Transport Validation Using Balloon-Based Experimental Data

The deterministic radiation transport code HZETRN (High charge (Z) and Energy TRaNsport) was developed by NASA to study the effects of cosmic radiation on astronauts and instrumentation shielded by various materials. This work presents an analysis of computed differential flux from HZETRN compared with measurement data from three balloon-based experiments over a range of atmospheric depths, particle types, and energies. Model uncertainties were quantified using an interval-based validation metric that takes into account measurement uncertainty both in the flux and the energy at which it was measured. Average uncertainty metrics were computed for the entire dataset as well as subsets of the measurements (by experiment, particle type, energy, etc.) to reveal any specific trends of systematic over- or under-prediction by HZETRN. The distribution of individual model uncertainties was also investigated to study the range and dispersion of errors beyond just single scalar and interval metrics. The differential fluxes from HZETRN were generally well-correlated with balloon-based measurements; the median relative model difference across the entire dataset was determined to be 30%. The distribution of model uncertainties, however, revealed that the range of errors was relatively broad, with approximately 30% of the uncertainties exceeding ± 40%. The distribution also indicated that HZETRN systematically under-predicts the measurement dataset as a whole, with approximately 80% of the relative uncertainties having negative values. Instances of systematic bias for subsets of the data were also observed, including a significant underestimation of alpha particles and protons for energies below 2.5 GeV/u. Muons were found to be systematically over-predicted at atmospheric depths deeper than 50 g/cm(sup 2) but under-predicted for shallower depths. Furthermore, a systematic under-prediction of alpha particles and protons was observed below the geomagnetic cutoff, suggesting that improvements to the light ion production cross sections in HZETRN should be investigated.

James E Warner↗
DOE OSTI2024

ESM data downscaling: a comparison of super-resolution deep learning models

Abstract Climate projections at fine spatial resolutions are required to conduct accurate risk assessment for critical infrastructure and design adaptation planning. Generating these projections using advanced Earth system models (ESM) requires significant computational resources. To address this issue, various statistical downscaling techniques have been introduced to generate fine-resolution data from coarse-resolution simulations. In this study, we evaluate and compare five deep learning-based downscaling techniques, namely, super-resolution convolutional neural networks, fast super-resolution convolutional neural network ESM, efficient sub-pixel convolutional neural network, enhanced deep residual network (EDRN), and super-resolution generative adversarial network (SRGAN). These techniques are applied to a dataset generated by the Energy Exascale Earth System Model (E3SM), focusing on key surface variables such as surface temperature, shortwave heat flux, and longwave heat flux. Models are trained and validated using paired fine-resolution (0.25 $$^{\circ }$$ ∘ ) and coarse-resolution (1 $$^{\circ }$$ ∘ ) monthly data obtained from a 9-year simulation. Next, blind testing is performed using monthly data obtained from two different years outside of the training and validation set. To evaluate the efficiency of each technique, different statistical metrics are used, including mean squared error (MSE), peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), and learned perceptual image patch similarity (LPIPS). The results show that EDRN outperforms other algorithms in terms of PSNR, SSIM, and MSE, but struggles to capture fine-scale features in the data. In contrast, SRGAN, a generative model that uses perceptual loss, excels in capturing fine details at boundaries and internal structures, resulting in lower LPIPS than other methods.

Pawar, Nikhil M. (ORCID:0000000211613289)↗
DOE OSTI2022

Multimodal sensor fusion framework for residential building occupancy detection

For several years now, smart building energy systems have been a research area of intensive activity. In light of the increasing need for sustainable buildings and energy systems, this trend motivates an increasing need for a solution to reduce carbon dioxide emissions and improve energy efficiency. This work proposes a high-performing and transferable occupancy detection framework that combines sensor data from different data modalities, including time series environmental data (temperature, humidity, and illuminance), image data, and acoustic energy data using ensemble method. To draw out the best prediction performance in each modality, the proposed framework was developed, including various models that were designed to learn the occupancy patterns reflected in the physical data streams. To tackle the time series environmental data, we designed two variants of an occupancy detection spatiotemporal pattern network (Occ-STPN) that performs both feature level and decision level fusion, respectively. We also propose a new metric; the fading memory mean square error (FMMSE), that provides a fair evaluation and penalization of delayed occupancy predictions. Multiple open-sourced datasets, including the Electricity Consumption and Occupancy and the University of California, Irvine's (UCI) building occupancy detection dataset, along with our own real data collected from six different houses, were used to validate the algorithms' performance. The experimental results presented herein break down the performance for each sensing modality, and a detailed analysis of the performance is also discussed.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2025

Thermal and electric multidomain dynamic model for integration of power grid distribution with behind-the-meter devices

As renewable energy sources like solar and wind power become more integrated into the grid, coordinated control of behind-the-meter devices is crucial for enhancing grid flexibility and reliability and for meeting cost targets, with standardized models being developed to support this transition. The increasing flexibility and uncertainty of integrated renewable energy grids, along with interactions between various subsystems, make traditional steady-state modeling insufficient to capture transient and dynamic behaviors. Current models (e.g., composite load and battery equivalent models) focus on thermodynamic or electrical characteristics but overlook critical electromechanical interactions. This limits the ability to share performance information for grid services and hampers fast dynamic simulations. In addition, motor stalling is usually triggered by a fault event and attributed to the characteristics of the mechanical torque of the motor, resulting in absorption of a large amount of reactive power during the stalling period. Further, this significant withdrawal of reactive power will deteriorate the dynamic voltage stability of power grids and cause delayed voltage recovery. Therefore, an in-depth modeling of the thermodynamics or mechanical torque is essential to study the impacts of the realistic torque characteristics of those behind-the-meter devices on power system voltage stability. This study developed a dynamic multidomain model for building HVAC systems, such as air-source heat pumps, to simulate their thermal and electrical responses to grid transients. The model can accurately predict power metrics with a mean absolute percentage error of 10%, by validating against with power system computer-aided design performance data. Case studies demonstrate the model capability of capturing the transient response to sudden voltage changes, rapid load fluctuations, and system shutdowns respectively. During a sudden voltage drop (30% for 0.1s), a fully loaded heat pump’s motor speed dropped, continued declining, and shut down after 3.6s, with severe power oscillations and a torque spike. A partially loaded unit experienced temporary oscillations but stabilized. Under higher building loads, compressor speed increased from 64% to 100%, with power and torque rising before stabilizing. In safety-triggered shutdowns, power decreased after minor fluctuations, and torque briefly spiked before dropping to zero.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2022

INTERFACES. A Program for Determining the 3D Structures of Surfaces Sites Using NMR Data

Dynamic nuclear polarization surface enhanced NMR spectroscopy has enabled the determination of high-resolution structures from surface-supported molecules, including singlesite heterogeneous catalysts. Structure determinations have largely mimicked the approaches used in biomolecular NMR spectroscopy, namely, using distance measurements to constrain a conformational search. These early demonstrations made use of purpose-built software, which has limited the adoption of the technique. Herein, we describe the open-source program INTERFACES (Interpret NMR to Elucidate or Reconstruct the Full Atomistic Configurations of External Surfaces) which automates the analysis of RE(SP)DOR data as well as the structure determination for surface sites. Distances, angles, dihedral angles, complex orientation, and distance from the support can all be sampled to find all structures that agree with the experimental data. A χ 2 metric is used to define the error ranges of the REDOR fits and produce structures with an arbitrary level of confidence. Structural solutions are then provided as both overlays and ORTEP-like probability ellipsoids.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2026

Cryogenic-Refined MOSFET Modeling for Oscillator, Frequency Divider, and Amplifier Designs Below 4 K

Capturing device characteristic changes at cryogenic temperatures is crucial for cryo-CMOS circuit designs. In this work, we present an isothermal cryogenic-refined modeling approach for CMOS transistors that is simple, low overhead, and easy to implement while offering the required accuracy for predicting circuit performance at the designated temperatures. Guided by die-level measurement data and circuit design principles, the model introduces corrections to only five critical parameters: threshold voltage, carrier mobility, elevated low-frequency flicker noise, dominant high-frequency shot noise, and subthreshold swing (SS). These refinements are implemented around the foundry-provided SPICE model, which is typically validated only down to about 200 K. With these adjustments, the proposed cryogenic-refined model achieves less than 5% error in both large-signal metrics (I–V characteristics) and small-signal parameters (e.g., transconductance) when compared with device measurements at deep-cryogenic temperatures. The methodology is validated in two advanced technologies: TSMC 40-nm CMOS and GlobalFoundries (GF) 45-nm RF-SOI. We further demonstrate its applicability in three representative RF circuits: a 30-GHz LC oscillator, a high-speed current-mode-logic (CML) frequency divider (FD), and a subthreshold Gb/s amplifier, all showing close agreement between simulated predictions and measurements performed at 4 and 2.5 K. Finally, we believe that the proposed approach is implementation-friendly and can significantly accelerate the development of cryo-CMOS integrated circuits.

circuit modeling↗
NASA NTRS1985

Analyzing the test process using structural coverage

A large, commercially developed FORTRAN program was modified to produce structural coverage metrics. The modified program was executed on a set of functionally generated acceptance tests and a large sample of operational usage cases. The resulting structural coverage metrics are combined with fault and error data to evaluate structural coverage. It was shown that in the software environment the functionally generated tests seem to be a good approximation of operational use. The relative proportions of the exercised statement subclasses change as the structural coverage of the program increases. A method was also proposed for evaluating if two sets of input data exercise a program in a similar manner. Evidence was provided that implies that in this environment, faults revealed in a procedure are independent of the number of times the procedure is executed and that it may be reasonable to use procedure coverage in software models that use statement coverage. Finally, the evidence suggests that it may be possible to use structural coverage to aid in the management of the acceptance test processed.

Ramsey, James↗
NASA NTRS2006

Integrated Modeling of Advanced Opto-Mechanical Systems

The design of optical hardware for space applications is particularly challenging when developing high performance, novel systems that have no precedent. Integrated modeling and analysis of such opto-mechanical systems seeks to describe the end-to-end performance of the hardware using mission-relevant metrics. This multidisciplinary analysis might start with thermal disturbances from observation maneuvers, compute the system temperatures, compute the distorted positions and shapes of the hardware and compute the resulting optical performance. Dynamic disturbances such as reaction wheel imbalance or inertia imbalance of optical delay lines might be applied to a structural dynamic model and used in a guidance and control analysis. Mission-relevant science metrics might include wavefront quality, pointing error or imaging stability. Assembling a tool chain that can be both nimble and effective when scaled to the high fidelity models of detail design has been challenging. An integrated thermal, mechanical and optical analysis capability suitable for detail design has been developed and verified through experimental measurement. This capability was used in the design of flight-like breadboard hardware and development of a test apparatus that established both the level of performance of the hardware and the validity of the analysis. The analysis includes prediction of the thermal environment of the test chamber, detailed temperature distributions on the breadboard hardware, fine scale deformations of the optical elements, and computation of the wavefront quality using geometric optics. A battery of tests were conducted to assess the experiment data acquisition, measurement and control system and to establish the performance of the hardware design and accuracy of the integrated modeling. Thermal loads that represent operational observing maneuvers were imposed and the hardware optical performance was measured and compared to analytical predictions.

optomechanical analysis↗
NASA NTRS2007

Measuring Image Navigation and Registration Performance at the 3-Sigma Level Using Platinum Quality Landmarks

Geostationary Operational Environmental Satellite (GOES) Image Navigation and Registration (INR) performance is specified at the 3- level, meaning that 99.7% of a collection of individual measurements must comply with specification thresholds. Landmarks are measured by the Replacement Product Monitor (RPM), part of the operational GOES ground system, to assess INR performance and to close the INR loop. The RPM automatically discriminates between valid and invalid measurements enabling it to run without human supervision. In general, this screening is reliable, but a small population of invalid measurements will be falsely identified as valid. Even a small population of invalid measurements can create problems when assessing performance at the 3-sigma level. This paper describes an additional layer of quality control whereby landmarks of the highest quality ("platinum") are identified by their self-consistency. The platinum screening criteria are not simple statistical outlier tests against sigma values in populations of INR errors. In-orbit INR performance metrics for GOES-12 and GOES-13 are presented using the platinum landmark methodology.

Carr, James L.↗
NASA NTRS2009

Ares I Static Tests Design

Probabilistic engineering design enhances safety and reduces costs by incorporating risk assessment directly into the design process. In this paper, we assess the format of the quantitative metrics for the vehicle which will replace the Space Shuttle, the Ares I rocket. Specifically, we address the metrics for in-flight measurement error in the vector position of the motor nozzle, dictated by limits on guidance, navigation, and control systems. Analyses include the propagation of error from measured to derived parameters, the time-series of dwell points for the duty cycle during static tests, and commanded versus achieved yaw angle during tests. Based on these analyses, we recommend a probabilistic template for specifying the maximum error in angular displacement and radial offset for the nozzle-position vector. Criteria for evaluating individual tests and risky decisions also are developed.

Carson, William↗
NASA NTRS2008

Image Compression Algorithm Altered to Improve Stereo Ranging

A report discusses a modification of the ICER image-data-compression algorithm to increase the accuracy of ranging computations performed on compressed stereoscopic image pairs captured by cameras aboard the Mars Exploration Rovers. (ICER and variants thereof were discussed in several prior NASA Tech Briefs articles.) Like many image compressors, ICER was designed to minimize a mean-square-error measure of distortion in reconstructed images as a function of the compressed data volume. The present modification of ICER was preceded by formulation of an alternative error measure, an image-quality metric that focuses on stereoscopic-ranging quality and takes account of image-processing steps in the stereoscopic-ranging process. This metric was used in empirical evaluation of bit planes of wavelet-transform subbands that are generated in ICER. The present modification, which is a change in a bit-plane prioritization rule in ICER, was adopted on the basis of this evaluation. This modification changes the order in which image data are encoded, such that when ICER is used for lossy compression, better stereoscopic-ranging results are obtained as a function of the compressed data volume.

Kiely, Aaron↗
NASA NTRSDate not supplied

Short Circuiting the Controller – Missteps in Maintenance and Inspection of Process and Wiring in STS-93

The primary objective of the Space Transportation System mission 93 (STS-93) was to deploy the Chandra X-Ray Observatory. Chandra, the world's most powerful X-Ray telescope, allowed scientists from around the world to study some of the most distant and dynamic objects in the universe. Stripped of nearly 7,000 pounds of its own gear to make room for the payload, the orbiter assigned to this mission was Space Shuttle Columbia. Prior to STS-93, Columbia had flown 25 flights and was NASA's oldest and heaviest orbiter. On July 23, 1999, after two prior launch scrubs, Eileen Collins and her crew of four launched from Kennedy Space Center. About five seconds after launch, Mission Control at Johnson Space Center detected a voltage drop on one of Columbia's electrical buses. As a result of this power fluctuation, a primary and back-up Main Engine controller dropped offline. Given design redundancy, the two remaining controllers supported all three engines. If there had been any other AC bus issues, one engine of the three on the Orbiter would have shut down. The redundant set of digital computer units in each engine controller saved Columbia and her crew from a very risky contingency abort. Post-flight inspection revealed soot on a screw head and a hole in an adjacent Kapton insulated wire. The single strand of polyimide wire was located nearly half-way down the payload bay. The Shuttle Independent Assessment Team (SIAT) reported that the wire had rubbed and chaffed against a burred screw head. The burr was later determined to be the result of overtightening of the screw by a technician during a maintenance refurbishment. Alone, the burr may not have been problematic, but later, during another ground processing event, possibly years after, someone inadvertently stepped on the wiring harness. With the pressure and motion of unintended contact, some of the Kapton insulation rubbed off against the burred screw head. The SIAT suspected the wire damage was pre-existing and was caused 4 or 5 years prior to the flight. Finally, the intense vibrations during the launch sequence allowed contact between the exposed conductor and exposed metal area on the burred screw head, resulting in the arcing and shorting of the wire. Due to the quick turnaround times of Space Shuttle Orbiters, wiring issues caused from multiple maintenance events were often overlooked. Additionally, failing to incorporate thorough and early inclusion of human systems integration (HSI) applications as a crucial part of the decision process can result in these types of misshaps. In order to reduce human error associated with integrated manufacturing, maintenance, refurbishment and flight preparations, wire inspection criteria should be refined and standardized, visual inspection processes should be quantified, and technicians should be certified by specially trained instructors. This case, among many others, unveils why human error management and development of safety metrics is a vital piece in the development of complex systems, and why it should be supported aggressively and implemented program wide.

Human Systems Integration↗