Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Error Metrics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
NASA NTRS2018

HZETRN Radiation Transport Validation Using Balloon-Based Experimental Data

The deterministic radiation transport code HZETRN (High charge (Z) and Energy TRaNsport) was developed by NASA to study the effects of cosmic radiation on astronauts and instrumentation shielded by various materials. This work presents an analysis of computed differential flux from HZETRN compared with measurement data from three balloon-based experiments over a range of atmospheric depths, particle types, and energies. Model uncertainties were quantified using an interval-based validation metric that takes into account measurement uncertainty both in the flux and the energy at which it was measured. Average uncertainty metrics were computed for the entire dataset as well as subsets of the measurements (by experiment, particle type, energy, etc.) to reveal any specific trends of systematic over- or under-prediction by HZETRN. The distribution of individual model uncertainties was also investigated to study the range and dispersion of errors beyond just single scalar and interval metrics. The differential fluxes from HZETRN were generally well-correlated with balloon-based measurements; the median relative model difference across the entire dataset was determined to be 30%. The distribution of model uncertainties, however, revealed that the range of errors was relatively broad, with approximately 30% of the uncertainties exceeding ± 40%. The distribution also indicated that HZETRN systematically under-predicts the measurement dataset as a whole, with approximately 80% of the relative uncertainties having negative values. Instances of systematic bias for subsets of the data were also observed, including a significant underestimation of alpha particles and protons for energies below 2.5 GeV/u. Muons were found to be systematically over-predicted at atmospheric depths deeper than 50 g/cm(sup 2) but under-predicted for shallower depths. Furthermore, a systematic under-prediction of alpha particles and protons was observed below the geomagnetic cutoff, suggesting that improvements to the light ion production cross sections in HZETRN should be investigated.

James E Warner↗
NASA NTRS1985

Analyzing the test process using structural coverage

A large, commercially developed FORTRAN program was modified to produce structural coverage metrics. The modified program was executed on a set of functionally generated acceptance tests and a large sample of operational usage cases. The resulting structural coverage metrics are combined with fault and error data to evaluate structural coverage. It was shown that in the software environment the functionally generated tests seem to be a good approximation of operational use. The relative proportions of the exercised statement subclasses change as the structural coverage of the program increases. A method was also proposed for evaluating if two sets of input data exercise a program in a similar manner. Evidence was provided that implies that in this environment, faults revealed in a procedure are independent of the number of times the procedure is executed and that it may be reasonable to use procedure coverage in software models that use statement coverage. Finally, the evidence suggests that it may be possible to use structural coverage to aid in the management of the acceptance test processed.

Ramsey, James↗
NASA NTRS2006

Integrated Modeling of Advanced Opto-Mechanical Systems

The design of optical hardware for space applications is particularly challenging when developing high performance, novel systems that have no precedent. Integrated modeling and analysis of such opto-mechanical systems seeks to describe the end-to-end performance of the hardware using mission-relevant metrics. This multidisciplinary analysis might start with thermal disturbances from observation maneuvers, compute the system temperatures, compute the distorted positions and shapes of the hardware and compute the resulting optical performance. Dynamic disturbances such as reaction wheel imbalance or inertia imbalance of optical delay lines might be applied to a structural dynamic model and used in a guidance and control analysis. Mission-relevant science metrics might include wavefront quality, pointing error or imaging stability. Assembling a tool chain that can be both nimble and effective when scaled to the high fidelity models of detail design has been challenging. An integrated thermal, mechanical and optical analysis capability suitable for detail design has been developed and verified through experimental measurement. This capability was used in the design of flight-like breadboard hardware and development of a test apparatus that established both the level of performance of the hardware and the validity of the analysis. The analysis includes prediction of the thermal environment of the test chamber, detailed temperature distributions on the breadboard hardware, fine scale deformations of the optical elements, and computation of the wavefront quality using geometric optics. A battery of tests were conducted to assess the experiment data acquisition, measurement and control system and to establish the performance of the hardware design and accuracy of the integrated modeling. Thermal loads that represent operational observing maneuvers were imposed and the hardware optical performance was measured and compared to analytical predictions.

optomechanical analysis↗
NASA NTRS2007

Measuring Image Navigation and Registration Performance at the 3-Sigma Level Using Platinum Quality Landmarks

Geostationary Operational Environmental Satellite (GOES) Image Navigation and Registration (INR) performance is specified at the 3- level, meaning that 99.7% of a collection of individual measurements must comply with specification thresholds. Landmarks are measured by the Replacement Product Monitor (RPM), part of the operational GOES ground system, to assess INR performance and to close the INR loop. The RPM automatically discriminates between valid and invalid measurements enabling it to run without human supervision. In general, this screening is reliable, but a small population of invalid measurements will be falsely identified as valid. Even a small population of invalid measurements can create problems when assessing performance at the 3-sigma level. This paper describes an additional layer of quality control whereby landmarks of the highest quality ("platinum") are identified by their self-consistency. The platinum screening criteria are not simple statistical outlier tests against sigma values in populations of INR errors. In-orbit INR performance metrics for GOES-12 and GOES-13 are presented using the platinum landmark methodology.

Carr, James L.↗
NASA NTRS2009

Ares I Static Tests Design

Probabilistic engineering design enhances safety and reduces costs by incorporating risk assessment directly into the design process. In this paper, we assess the format of the quantitative metrics for the vehicle which will replace the Space Shuttle, the Ares I rocket. Specifically, we address the metrics for in-flight measurement error in the vector position of the motor nozzle, dictated by limits on guidance, navigation, and control systems. Analyses include the propagation of error from measured to derived parameters, the time-series of dwell points for the duty cycle during static tests, and commanded versus achieved yaw angle during tests. Based on these analyses, we recommend a probabilistic template for specifying the maximum error in angular displacement and radial offset for the nozzle-position vector. Criteria for evaluating individual tests and risky decisions also are developed.

Carson, William↗
NASA NTRS2008

Image Compression Algorithm Altered to Improve Stereo Ranging

A report discusses a modification of the ICER image-data-compression algorithm to increase the accuracy of ranging computations performed on compressed stereoscopic image pairs captured by cameras aboard the Mars Exploration Rovers. (ICER and variants thereof were discussed in several prior NASA Tech Briefs articles.) Like many image compressors, ICER was designed to minimize a mean-square-error measure of distortion in reconstructed images as a function of the compressed data volume. The present modification of ICER was preceded by formulation of an alternative error measure, an image-quality metric that focuses on stereoscopic-ranging quality and takes account of image-processing steps in the stereoscopic-ranging process. This metric was used in empirical evaluation of bit planes of wavelet-transform subbands that are generated in ICER. The present modification, which is a change in a bit-plane prioritization rule in ICER, was adopted on the basis of this evaluation. This modification changes the order in which image data are encoded, such that when ICER is used for lossy compression, better stereoscopic-ranging results are obtained as a function of the compressed data volume.

Kiely, Aaron↗
NASA NTRSDate not supplied

Short Circuiting the Controller – Missteps in Maintenance and Inspection of Process and Wiring in STS-93

The primary objective of the Space Transportation System mission 93 (STS-93) was to deploy the Chandra X-Ray Observatory. Chandra, the world's most powerful X-Ray telescope, allowed scientists from around the world to study some of the most distant and dynamic objects in the universe. Stripped of nearly 7,000 pounds of its own gear to make room for the payload, the orbiter assigned to this mission was Space Shuttle Columbia. Prior to STS-93, Columbia had flown 25 flights and was NASA's oldest and heaviest orbiter. On July 23, 1999, after two prior launch scrubs, Eileen Collins and her crew of four launched from Kennedy Space Center. About five seconds after launch, Mission Control at Johnson Space Center detected a voltage drop on one of Columbia's electrical buses. As a result of this power fluctuation, a primary and back-up Main Engine controller dropped offline. Given design redundancy, the two remaining controllers supported all three engines. If there had been any other AC bus issues, one engine of the three on the Orbiter would have shut down. The redundant set of digital computer units in each engine controller saved Columbia and her crew from a very risky contingency abort. Post-flight inspection revealed soot on a screw head and a hole in an adjacent Kapton insulated wire. The single strand of polyimide wire was located nearly half-way down the payload bay. The Shuttle Independent Assessment Team (SIAT) reported that the wire had rubbed and chaffed against a burred screw head. The burr was later determined to be the result of overtightening of the screw by a technician during a maintenance refurbishment. Alone, the burr may not have been problematic, but later, during another ground processing event, possibly years after, someone inadvertently stepped on the wiring harness. With the pressure and motion of unintended contact, some of the Kapton insulation rubbed off against the burred screw head. The SIAT suspected the wire damage was pre-existing and was caused 4 or 5 years prior to the flight. Finally, the intense vibrations during the launch sequence allowed contact between the exposed conductor and exposed metal area on the burred screw head, resulting in the arcing and shorting of the wire. Due to the quick turnaround times of Space Shuttle Orbiters, wiring issues caused from multiple maintenance events were often overlooked. Additionally, failing to incorporate thorough and early inclusion of human systems integration (HSI) applications as a crucial part of the decision process can result in these types of misshaps. In order to reduce human error associated with integrated manufacturing, maintenance, refurbishment and flight preparations, wire inspection criteria should be refined and standardized, visual inspection processes should be quantified, and technicians should be certified by specially trained instructors. This case, among many others, unveils why human error management and development of safety metrics is a vital piece in the development of complex systems, and why it should be supported aggressively and implemented program wide.

Human Systems Integration↗
NASA NTRS2018

Adjoint Estimation of Observation Impact Explored with an Observing System Simulation Experiment

In an Observing System Simulation Experiment, the full, true state of the simulated atmosphere is known. This knowledge allows the direct calculation of analysis and forecast errors, and also may be used in conjunction with an adjoint tool to calculate metrics that are unachievable in the real world. For example, the errors that result from the use of the analysis state as verification when running adjoint calculations of observation impact on a forecast can be quantified. The adjoint itself can also be used to operate on the analysis state rather than on the forecast state. Results from these and other adjoint experiments using the NASA/GMAO OSSE framework will be presented.

Prive, Nikki↗
NASA NTRS2016

Investigation of Optimal Digital Image Correlation Patterns for Deformation Measurement

Digital image correlation (DIC) relies on the surface texture of a specimen to measure deformation. When the specimen itself has little or no texture, a pattern is applied to the surface which deforms with the specimen and acts as an artificial surface texture. Because the applied pattern has an effect on the accuracy of DIC, an ideal pattern is sought for which the error introduced into DIC measurements is minimal. In this work, a study is performed in which several DIC pattern quality metrics from the literature are correlated to DIC measurement error. The resulting correlations give insight on the optimality of DIC patterns in general. Optimizations are then performed to produce patterns which are well suited for DIC. These patterns are tested to show their relative benefits. Chief among these benefits are a reduction in error of approximately 30 with respect to a randomly generated pattern.

Bomarito, G. F.↗
NASA NTRS2019

Adjoint-Based Anisotropic Mesh Adaptation for a Stabilized Finite-Element Flow Solver

An adjoint solver is implemented in the FUN3D stabilized finite-element flow solver. The adjoint solution is used to generate anisotropic, adapted meshes to control error in scalar out- put functionals, such as lift or drag coefficient. The inviscid and turbulent adjoints are verified with a finite-difference derivative approximation and can be used in design optimization in addition to mesh adaptation. The adjoint capability represents an extension of previous research using the FUN3D stabilized finite-element capability with metric-based mesh adaptation and interpolation-based error estimates to generate highly anisotropic adapted meshes for turbulent flows. In the present work, a metric-based approach is again utilized, where the adjoint and the primal solutions both contribute to the generation of a metric tensor field that is subsequently used to produce the required anisotropic mesh for each adaptation cycle. Adaptive results are then shown for an inviscid supersonic flow over a diamond airfoil, inviscid transonic flow over the ONERA M6 wing, and viscous laminar flow over the NACA 0012 airfoil, all using drag force as the output functional. Adjoint-based adaptation is compared with a multiscale solution-based approach that controls the L norm of Mach number interpolation error to demonstrate the effectiveness and effciency of the adjoint-based adaptive mesh technology.

Balan, Aravind↗
NASA NTRS2004

Improvement of the TOPEX and Jason Orbit Time Series: Precision Orbit Determination, Calibration, Validation and Improvement Through the Combined Reduction and Analysis of GPS, SLR, DORIS and Altimeter Data

Orbit error is a major component in the overall error budget of all altimeter satellite missions. Jason-I is no exception and a 1 cm radial orbit accuracy goal has been set, which represents a factor of two improvement over what is currently being achieved for TOPEX/Poseidon (TP). Our current analysis suggests this goal has been met and even improved upon, but the challenge is to be able to continually achieve this high accuracy, verify the performance and characterize and quantify the remaining errors over the lifetime of the mission. The computation, verification and error characterization of such high accuracy orbits requires the reduction and analysis of all available tracking data (GPS, SLR, DORIS and altimeter). Current analysis also indicates the history of TP orbits can be further improved employing new solution strategies developed and tested on Jason-I. Our research focuses on the calibration, validation and improvement of orbit accuracies using all available tracking data including altimetry. We will compute and distribute well centered Jason orbits with an accuracy of better than 1-cm in the radial component. In addition to the orbits themselves, a characterization of the orbit error will be distributed and accumulated as a time series of orbit performance metrics to track anomalies and trends. The long time series of orbit error characterization will enable a better understanding of the remaining orbit errors and its impact on the altimeter data analysis. As part of this research effort we are also significantly improving the current level of TP orbit accuracy, re-computing new high-accuracy TP orbits from the beginning of the TP mission and continuing into the future (as long as TP is healthy). Our funded research effort will result in a complete and consistent time series of improved orbits for both TP and Jason, significantly benefiting the long time series of altimeter data analysis and the TP/Jason dual mission. The resultant high accuracy orbits and the characterization of their error will allow further improvements to the accuracy and overall quality of the altimeter measurement time series making possible further strides in radar altimeter remote sensing.

Luthcke, Scott B.↗
NASA NTRSDate not supplied

Evaluating Lunar Descent and Landing Performance From a Near Rectilinear Halo Orbit Using Linear Covariance Resetting Techniques

Upcoming lunar programs are striving the achieve precision landing in a safe and robust manner. Various elements impact this mission objective ranging from on-orbit operations with ground station tracking to incorporating relative sensors with hazard detection and avoidance (HDA) to support the final approach and landing phase. Modeling the impacts of ground tracking, trajectory replanning, relative navigation sensors, and particularly a potential HDA system on the integrated closed-loop GN\&C system performance poses a unique challenge due to the complexity and interaction with multiple facets of the vehicle including the trajectory design, sensing hardware, navigation system, guidance and targeting, and the overall mission concept of operations. This paper outlines techniques to systematically analyze and compare the performance impacts of ground tracking and replanning and an HDA system where the onboard navigation errors are reset or uploaded from an external source and the vehicle's reference trajectory is regenerated requiring the system dispersions to also be reset to reflect this in-flight profile adjustment. To illustrate the application of these general techniques for analyzing the performance impacts due to incorporating these resetting events, they are demonstrated with a human lunar descent and landing scenario starting from a near rectilinear halo orbit (NRHO) until the vehicle precisely reaches its predetermined landing site on the lunar surface. Performance metrics such as inertial and relative navigation errors, trajectory dispersions, footprint dispersions, and propellant usage are provided.

GN&C↗
NASA NTRSDate not supplied

Space Needle Returns

STEP is imported into Engineering Sketch Pad. Some bodies where slightly scaled and translated in OpenCSM to create a manifold solid for the downstream meshing process. The braces at the base and columns around the core are omitted because their solids are malformed or created nonmanifold intersections. EGADS provides an initial tessellation of the surface. refine adapted the surface mesh to a curvature and feature size metric. TetGen initially filled the volume. The TetGen mesh is adapted to the Spalding Law of the Wall u+ with refine to provide the initial mesh for flow solution. Solution-based mesh adaptation is performed where FUN3D-FV computes the flow solution with the Reynolds-averaged Navier-Stokes equations coupled to the Spalart-Allmaras turbulence model. The freestream Mach number is 4 approaching 40° from the central axis of the Space Needle. The volume and surface mesh is adapted with refine to reduce estimated interpolation error in Mach number via the multiscale metric. The adapted mesh implicitly resolves the boundary layers, shocks, and expansions. The surface mesh is shown for the lee side with a slice through the volume on the left. Computational schlieren in the lower right shows density variations. A slice of the mesh is colored with Mach number in the upper right where mesh with freestream Mach number is not rendered. The volume mesh contains 64 million vertices. A NASA worm logo is sketched and extruded into a solid in OpenCSM. The worm is unioned to the Space Needle roof to produce the inset mesh image.

mesh↗
NASA NTRS2004

Robust Timing Synchronization in Aeronautical Mobile Communication Systems

This work details a study of robust synchronization schemes suitable for satellite to mobile aeronautical applications. A new scheme, the Modified Sliding Window Synchronizer (MSWS), is devised and compared with existing schemes, including the traditional Early-Late Gate Synchronizer (ELGS), the Gardner Zero-Crossing Detector (GZCD), and the Sliding Window Synchronizer (SWS). Performance of the synchronization schemes is evaluated by a set of metrics that indicate performance in digital communications systems. The metrics are convergence time, mean square phase error (or root mean-square phase error), lowest SNR for locking, initial frequency offset performance, midstream frequency offset performance, and system complexity. The performance of the synchronizers is evaluated by means of Matlab simulation models. A simulation platform is devised to model the satellite to mobile aeronautical channel, consisting of a Quadrature Phase Shift Keying modulator, an additive white Gaussian noise channel, and a demodulator front end. Simulation results show that the MSWS provides the most robust performance at the cost of system complexity. The GZCD provides a good tradeoff between robustness and system complexity for communication systems that require high symbol rates or low overall system costs. The ELGS has a high system complexity despite its average performance. Overall, the SWS, originally designed for multi-carrier systems, performs very poorly in single-carrier communications systems. Table 5.1 in Section 5 provides a ranking of each of the synchronization schemes in terms of the metrics set forth in Section 4.1. Details of comparison are given in Section 5. Based on the results presented in Table 5, it is safe to say that the most robust synchronization scheme examined in this work is the high-sample-rate Modified Sliding Window Synchronizer. A close second is its low-sample-rate cousin. The tradeoff between complexity and lowest mean-square phase error determines the rankings of the Gardner Zero-Crossing Detector and both versions of the Early-Late Gate Synchronizer. The least robust models are the high and low-sample-rate Sliding Window Synchronizers. Consequently, the recommended replacement synchronizer for NASA's Advanced Air Transportation Technologies mobile aeronautical communications system is the high-sample-rate Modified Sliding Window Synchronizer. By incorporating this synchronizer into their system, NASA can be assured that their system will be operational in extremely adverse conditions. The quick convergence time of the MSWS should allow the use of high-level protocols. However, if NASA feels that reduced system complexity is the most important aspect of their replacement synchronizer, the Gardner Zero-Crossing Detector would be the best choice.

Xiong, Fu-Qin↗
NASA NTRS2018

Optical Performance Prediction of the Thirty Meter Telescope after Initial Alignment Using Optical Modeling

We present an estimate of the optical performance of the Thirty Meter Telescope (TMT) after execution of the full telescope alignment plan. The TMT alignment is performed by the Global Metrology System (GMS) and the Alignment and Phasing System (APS). The GMS first measures the locations of the telescope optics and instruments as a function of elevation angle. These initial measurements will be used to adjust the optics positions and build initial elevation look-up tables. Then the telescope is aligned using starlight as the input for the APS at multiple elevation angles. APS measurements are used to refine the telescope alignment to build elevation and temperature dependent look-up tables. Due to the number of degrees of freedom in the telescope (over 10,000), the ability of the primary mirror to correct aberrations on other optics, the tight optical performance requirements and the multiple instrument locations, it is challenging to develop, test and validate these alignment procedures. In this paper, we consider several GMS and APS operational scenarios. We apply the alignment procedures to the model-generated TMT, which consists of various quasi-static errors such as polishing errors, passive supports errors, thermal and gravity deformations and installation position errors. Using an integrated optical model and Monte-Carlo framework, we evaluate the TMT’s aligned states using optical performance metrics at multiple instrument and field of view locations. The optical performance metrics include the Normalized Point Source Sensitivity (PSSN), RMS wavefront error before and after Adaptive Optics (AO) correction, pupil position change, and plate scale distortion.

Rogers, John↗
NASA NTRS2020

Recent Developments in Tolerancing Methods for Imaging Spectrometers

We discuss detailed tolerancing methods developed for imaging spectrometers at NASA Jet Propulsion Laboratory, California Institute of Technology using the Earth Surface Mineral Dust Source Investigation (EMIT) imaging spectrometer as an illustrative example. We tolerance five metrics simultaneously: along-track response function, cross-track response function, spectral response function, spectral centroid uniformity, and spatial centroid uniformity. A method to calculate tolerancing sensitivities for each metric directly, a method to statistically combine Monte Carlo files from multiple tolerancing runs, and an example summary error budgets that communicate the key and driving tolerances for each metric are discussed. These methods facilitate rapid and semi-automated assessment of the predicted performance of imaging spectrometer systems from design through to assembly and launch life cycle, using metrics that are directly relevant to the extraction of accurate spectroscopic data from these instruments.

Mouroulis, Pantazis↗
NASA NTRS2018

Performance Metrics for the Assessment of Satellite Data Products: An Ocean Color Case Study

Performance assessment of ocean color satellite data has generally relied on statistical metrics chosen for their common usage and the rationale for selecting certain metrics is infrequently explained. Commonly reported statistics based on mean squared errors, such as the coefficient of determination (r2), root mean square error, and regression slopes, are most appropriate for Gaussian distributions without outliers and, therefore, are often not ideal for ocean color algorithm performance assessment, which is often limited by sample availability. In contrast, metrics based on simple deviations, such as bias and mean absolute error, as well as pair-wise comparisons, often provide more robust and straightforward quantities for evaluating ocean color algorithms with non-Gaussian distributions and outliers. This study uses a SeaWiFS chlorophyll-a validation data set to demonstrate a framework for satellite data product assessment and recommends a multimetric and user-dependent approach that can be applied within science, modeling, and resource management communities.

remote sensing↗