Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “testing statistics”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 487 records · Page 27
All sourcesNASA NTRSDOE OSTI
NASA NTRS2004

A Geomagnetic Estimate of Mean Paleointensity

To test a statistical hypothesis about Earth's magnetic field against paleomagnetism, the present field is used to estimate time averaged paleointensity. The estimate used the modern magnetic multipole spectrum R(n), which gives the mean square induction represented by spherical harmonics of degree n averaged over the sphere of radius a = 6371.2 km. The hypothesis asserts that low degree multi-pole powers of the coresource field are distributed as chi-squared with 2n+1 degrees of freedom and expectation values, where c is the 3480 km radius of the Earth's core. (This is compatible with a usually mainly geocentric axial dipolar field). Amplitude K is estimated by fitting theoretical to observational spectra through degree 12. The resulting calibrated expectation spectrum is summed through degree 12 to estimate expected square intensity F(exp 2). The sum also estimates F(exp 2) averaged over geologic time, in so far as the present magnetic spectrum is a fair sample of that generated in the past by core geodynamic processes. Additional information is included in the original extended abstract.

Voorhies, Coerte V.↗
NASA NTRS2007

Measuring Image Navigation and Registration Performance at the 3-Sigma Level Using Platinum Quality Landmarks

Geostationary Operational Environmental Satellite (GOES) Image Navigation and Registration (INR) performance is specified at the 3- level, meaning that 99.7% of a collection of individual measurements must comply with specification thresholds. Landmarks are measured by the Replacement Product Monitor (RPM), part of the operational GOES ground system, to assess INR performance and to close the INR loop. The RPM automatically discriminates between valid and invalid measurements enabling it to run without human supervision. In general, this screening is reliable, but a small population of invalid measurements will be falsely identified as valid. Even a small population of invalid measurements can create problems when assessing performance at the 3-sigma level. This paper describes an additional layer of quality control whereby landmarks of the highest quality ("platinum") are identified by their self-consistency. The platinum screening criteria are not simple statistical outlier tests against sigma values in populations of INR errors. In-orbit INR performance metrics for GOES-12 and GOES-13 are presented using the platinum landmark methodology.

Carr, James L.↗
NASA NTRS2015

Characterization of Machine Variability and Progressive Heat Treatment in Selective Laser Melting of Inconel 718

The absence of an economy of scale in spaceflight hardware makes additive manufacturing an immensely attractive option for propulsion components. As additive manufacturing techniques are increasingly adopted by government and industry to produce propulsion hardware in human-rated systems, significant development efforts are needed to establish these methods as reliable alternatives to conventional subtractive manufacturing. One of the critical challenges facing powder bed fusion techniques in this application is variability between machines used to perform builds. Even with implementation of robust process controls, it is possible for two machines operating at identical parameters with equivalent base materials to produce specimens with slightly different material properties. The machine variability study presented here evaluates 60 specimens of identical geometry built using the same parameters. 30 samples were produced on machine 1 (M1) and the other 30 samples were built on machine 2 (M2). Each of the 30-sample sets were further subdivided into three subsets (with 10 specimens in each subset) to assess the effect of progressive heat treatment on machine variability. The three categories for post-processing were: stress relief, stress relief followed by hot isostatic press (HIP), and stress relief followed by HIP followed by heat treatment per AMS 5664. Each specimen (a round, smooth tensile) was mechanically tested per ASTM E8. Two formal statistical techniques, hypothesis testing for equivalency of means and one-way analysis of variance (ANOVA), were applied to characterize the impact of machine variability and heat treatment on six material properties: tensile stress, yield stress, modulus of elasticity, fracture elongation, and reduction of area. This work represents the type of development effort that is critical as NASA, academia, and the industrial base work collaboratively to establish a path to certification for additively manufactured parts. For future flight programs, NASA and its commercial partners will procure parts from vendors who will use a diverse range of machines to produce parts and, as such, it is essential that the AM community develop a sound understanding of the degree to which machine variability impacts material properties.

Prater, T.↗
NASA NTRS2015

Characterization of Machine Variability and Progressive Heat Treatment in Selective Laser Melting of Inconel 718

The absence of an economy of scale in spaceflight hardware makes additive manufacturing an immensely attractive option for propulsion components. As additive manufacturing techniques are increasingly adopted by government and industry to produce propulsion hardware in human‐rated systems, significant development efforts are needed to establish these methods as reliable alternatives to conventional subtractive manufacturing. One of the critical challenges facing powder bed fusion techniques in this application is variability between machines used to perform builds. Even with implementation of robust process controls, it is possible for two machines operating at identical parameters with equivalent base materials to produce specimens with slightly different material properties. The machine variability study presented here evaluates 60 specimens of identical geometry built using the same parameters. 30 samples were produced on machine 1 (M1) and the other 30 samples were built on machine 2 (M2). Each of the 30‐sample sets were further subdivided into three subsets (with 10 specimens in each subset) to assess the effect of progressive heat treatment on machine variability. The three categories for post‐processing were: stress relief, stress relief followed by hot isostatic press (HIP), and stress relief followed by HIP followed by heat treatment per AMS 5664. Each specimen (a round, smooth tensile) was mechanically tested per ASTM E8. Two formal statistical techniques, hypothesis testing for equivalency of means and one‐way analysis of variance (ANOVA), were applied to characterize the impact of machine variability and heat treatment on six material properties: tensile stress, yield stress, modulus of elasticity, fracture elongation, and reduction of area. This work represents the type of development effort that is critical as NASA, academia, and the industrial base work collaboratively to establish a path to certification for additively manufactured parts. For future flight programs, NASA and its commercial partners will procure parts from vendors who will use a diverse range of machines to produce parts and, as such, it is essential that the AM community develop a sound understanding of the degree to which machine variability impacts material properties.

Prater, Tracie↗
NASA NTRSDate not supplied

QuantifyML: How Good is My Machine Learning Model?

We present QuantifyML, which applies model counting to assess the learn ability, safety, and robustness of machine learning models. Typically, the efficacy of machine learning models is determined by computing their accuracy statistically on test datasets. However, this may be misleading, if the test data is not representative of the problem that is being studied. With QuantifyML we aim to precisely quantify the extent to which machine learning models have learned and generalized from the given data. In QuantifyML, a trained model is translated into aC program, which is fed to the CBMC model checking tool to produce a formula in Conjunctive Normal Form (CNF), which in turn is analyzed with state-of-the-art model counters to efficiently obtain precise countsw.r.t different outputs. QuantifyML enables i) evaluating the learnability of models by comparing the counts for the outputs to ground truth, ex-pressed as logical predicates (if available), ii) comparing the performance of different models that may be built with different machine learning algorithms (e.g., decision-trees vs. neural networks), and iii) quantifying the safety and robustness of trained models.

Deep Neural Networks↗
NASA NTRS2025

A Minority Report Submitted as an Addendum to the Report of the Mars 2020 Organic Contamination Panel

This Minority Report (MR) presents seven findings in addition to or contrary to the main OCP report: 1. Contamination control for the Mars 2020 cache must be strict. Mars’ surface is known to be organics-poor from laboratory studies, Mars meteorite analyses, and from four previous NASA missions. It is imperative that contamination control measures are enacted that enable reliable and robust detection of potential biomarker compounds at the ppb level. 2. Since Mars 2020 is a sample return mission and analyses of samples in the returnable cache are expected to occur after return to Earth, positive controls are not recommended for flight on Mars 2020 unless a compelling case can be made for their use. 3. The findings of previous panels dedicated to organic compound analysis in Martian samples (OCSSG, ND-SAG, SDT) recommend TOC limits between 10-40 ppb. The MR finds that the lower limit of 10 ppb is recommended and that insufficient justification is given by the OCP Panel Report (PR) to raise the TOC limit to 40 ppb. 4. Analytical capability is sufficiently advanced that analytical capability is an irrelevant consideration with respect to differentiating between 10 and 40 ppb TOC. 5. Perceived contamination control challenges are an irrelevant consideration for raising the TOC contamination limit from 10 to 40 ppb since those challenges, and the procedures to ameliorate them, will exist regardless of whether the limit is 10 or 40 ppb. 6. The “dilution cleaning” method has not been adequately proven for utilization on the Mars 2020 mission. Shortcomings have been identified in terms of peer review, method verification, analytical and testing approach, application to space flight hardware, and performance under Martian conditions. The method should be revisited and independently tested using statistically and analytically robust methods, and scrutinized in a rigorous peer review process. 7. The Mars 2020 mission claims considerable heritage from the Mars Science Laboratory (MSL) mission, but MSL contamination control efforts contain significant errors and implementation discrepancies that would imperil the Mars 2020 caching mission were they repeated. A standing contamination control panel should be formed to provide independent oversight for the Mars 2020 mission.

Mars Sample Return↗
DOE OSTI2020

Machine Learning Approaches for Nuclear Material Accounting Data from Irradiation and Reprocessing

We are currently exploring data analysis methods for their ability to strengthen the synthesis and evaluation of information generated within domestic and international safeguard regimes. Safeguard data typically includes rich heterogenous datasets amenable to advanced data analytics methods, such as machine learning. We have converted transactional data containing both numerical and categorical attributes from a domestic nuclear material control and accountability (NMC&A) system into a low-dimensional numerical structure via linear principal component analysis. This data representation allows for global structure discovery via cluster analysis, which can characterize the typical behavior of each of the primary types of transaction events. Furthermore, the structure of the top principal components captures the “typical behavior” of the data and is thus amenable to anomaly detection through statistical hypothesis testing. We explored this capability by generating erroneous permutations of the data and computing the Q-residual quantity increase associated with the information loss when these data are projected into the low-dimensional principal component analysis space representative of typical transactions.Future work will focus on identifying data transformations (e.g., graph networks) that more closely align with the inherent structure of these transactions to explain more salient information and disambiguate the underlying nuclear process—in this case, irradiation and reprocessing—from artifacts of NMC&A system transactional record keeping data.

Drescher, Adam↗
DOE OSTI2019

Investigation of Thermolytic Production of Hydrogen Applicable to Savannah River Site High Level Waste - 19443

Several experiments have been performed to measure the generation rates of hydrogen gas due to reaction (thermolysis) of organic compounds in both simulated and radioactive waste. Initial measurements of hydrogen generation rates at 100 deg. C were made to downselect molecules and functional groups that produce significant amounts of hydrogen for subsequent kinetic determination. A statistically-designed test matrix was employed to quantitatively determine chemical conditions of highest reactivity at 100 deg. C. Additional experiments were performed at 100 deg. C to determine the impact of specific chemical components on hydrogen generation rates for relevant organic compounds. A semi-empirical model for thermolytic hydrogen generation rate from glycolate at 100 deg. C was developed and compared to results from radioactive waste and literature reports. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
NASA NTRS2016

Improved Test Planning and Analysis Through the Use of Advanced Statistical Methods

The goal of this work is, through computational simulations, to provide statistically-based evidence to convince the testing community that a distributed testing approach is superior to a clustered testing approach for most situations. For clustered testing, numerous, repeated test points are acquired at a limited number of test conditions. For distributed testing, only one or a few test points are requested at many different conditions. The statistical techniques of Analysis of Variance (ANOVA), Design of Experiments (DOE) and Response Surface Methods (RSM) are applied to enable distributed test planning, data analysis and test augmentation. The D-Optimal class of DOE is used to plan an optimally efficient single- and multi-factor test. The resulting simulated test data are analyzed via ANOVA and a parametric model is constructed using RSM. Finally, ANOVA can be used to plan a second round of testing to augment the existing data set with new data points. The use of these techniques is demonstrated through several illustrative examples. To date, many thousands of comparisons have been performed and the results strongly support the conclusion that the distributed testing approach outperforms the clustered testing approach.

Green, Lawrence L.↗
NASA NTRS1985

Probability of detection of internal voids in structural ceramics using microfocus radiography

The reliability of microfocus x-radiography for detecting subsurface voids in structural ceramic test specimens was statistically evaluated. The microfocus system was operated in the projection mode using low X-ray photon energies (20 keV) and a 10 micro m focal spot. The statistics were developed for implanted subsurface voids in green and sintered silicon carbide and silicon nitride test specimens. These statistics were compared with previously-obtained statistics for implanted surface voids in similar specimens. Problems associated with void implantation are discussed. Statistical results are given as probability-of-detection curves at a 95 percent confidence level for voids ranging in size from 20 to 528 micro m in diameter.

Baaklini, G. Y.↗
NASA NTRS1986

Probability of detection of internal voids in structural ceramics using microfocus radiography

The reliability of microfocous X-radiography for detecting subsurface voids in structural ceramic test specimens was statistically evaluated. The microfocus system was operated in the projection mode using low X-ray photon energies (20 keV) and a 10 micro m focal spot. The statistics were developed for implanted subsurface voids in green and sintered silicon carbide and silicon nitride test specimens. These statistics were compared with previously-obtained statistics for implanted surface voids in similar specimens. Problems associated with void implantation are discussed. Statistical results are given as probability-of-detection curves at a 95 precent confidence level for voids ranging in size from 20 to 528 micro m in diameter.

Baaklini, G. Y.↗
NASA NTRS1984

Applications of model structure determination to flight test data

Several statistical and information criteria need to be considered when selecting an adequate model. Incorrect stability and control derivates result from inadequate aerodynamic model structure. Stepwise regression is used to determine the structure for an adequate model. Flight data which covers a nonlinear aerodynamic model range may be analyzed as a single data set or partitioned into several distinct sets. Stepwise regression for model structure detemination and parameter estimation was successfully applied to three aircraft types (single engine general aviation, unaugmented modern jet fighter, jet transport).

Batterson, J. G.↗
NASA NTRS2023

Fitting Leak Test Report: Ground-Based Cryogenic Leak Test of Fittings for Cryogenic Fluid Management

EXECUTIVE SUMMARY Mechanically connected joints used in cryogenic fluid lines as part of space flight elements need to survive launch vibrations and remain leak-free to minimize the loss of on-board commodity and hazardous gas accumulation. In 2020, a cryogenic test apparatus was developed which can evaluate the leak performance of pressurized threaded fluid fittings. The fittings were mounted in the TVAC and cooled to cryogenic test temperature and pressurized with helium while the leak rate was measured using a calibrated GHe leak detector. The test articles for the initial proof of concept testing were ¼ and 1 inch Swagelok VCR fittings with three different types of seal rings copper, nickel, and Ni. Each fitting configuration (size/seal material) was subjected to two consecutive cryogenic thermal cycles, followed by exposure to a launch vibration profile at ambient temperature, after which two additional TVAC cycle tests were performed. The testing reported here is a continuation of the 2020 tests with a statistically significant large number of samples and test runs. Three Swagelok VCR fitting sizes were tested ¼, ½ and 1 inch, and five (5) samples of each fitting size, each sample was tested with SST and Ni seal rings (Total of 30 unique test articles). Each test article was subjected to four (4) thermal cycles. Half of these cycles were performed before vibration testing and half were performed after vibration testing. The vibration testing was performed to evaluate the ability of the fittings to survive launch-type vibration profiles and remain leak-free. Leak checking of each fitting was completed at temperatures between 20K – 30K. The test procedure in Section 8.0 was designed to facilitate a qualification test program by allowing a higher test throughput rate coupled with repeatable test profiles. Results were very positive and show that out of the 30 samples they all passed with leak rates a factor of 2-3 lower than the established 10-6sccs GHe leak threshold. The result showed the Ni seals had lower leak rate, but the SST was more rugged. There were two deviations where damage to the Ni seal ring during assembly resulted in a leaky fitting, this is discussed in Section 10.7 Test Deviations. These fittings show great promise for space flight use and further testing is recommended to fully qualify the fittings per the ASTM F1387-19 and/or other relevant NASA specifications. The test equipment hardware and software capability developed for this testing is generic and not restricted to VCR fittings. It can be employed to evaluate/qualify the leak performance of other types of fittings and a wide range of other cryogenic fluid components such as valves, gages, connectors, etc.

Cryogenic↗
DOE OSTI2025

Testing convolutional neural network based deep learning systems: a statistical metamorphic approach

Machine learning technology spans many areas and today plays a significant role in addressing a wide range of problems in critical domains,i.e., healthcare, autonomous driving, finance, manufacturing, cybersecurity,etc. Metamorphic testing (MT) is considered a simple but very powerful approach in testing such computationally complex systems for which either an oracle is not available or is available but difficult to apply. Conventional metamorphic testing techniques have certain limitations in verifying deep learning-based models (i.e., convolutional neural networks (CNNs)) that have a stochastic nature (because of randomly initializing the network weights) in their training. In this article, we attempt to address this problem by using a statistical metamorphic testing (SMT) technique that does not require software testers to worry about fixing the random seeds (to get deterministic results) to verify the metamorphic relations (MRs). We propose seven MRs combined with different statistical methods to statistically verify whether the program under test adheres to the relation(s) specified in the MR(s). We further use mutation testing techniques to show the usefulness of the proposed approach in the healthcare space and test two CNN-based deep learning models (used for pneumonia detection among patients). The empirical results show that our proposed approach uncovers 85.71% of the implementation faults in the classifiers under test (CUT). Furthermore, we also propose an MRs minimization algorithm for the CUT, thus saving computational costs and organizational testing resources.

Computer Science↗
NASA NTRS2011

Statistical Issues for Uncontrolled Reentry Hazards Empirical Tests of the Predicted Footprint for Uncontrolled Satellite Reentry Hazards

A number of statistical tools have been developed over the years for assessing the risk of reentering objects to human populations. These tools make use of the characteristics (e.g., mass, material, shape, size) of debris that are predicted by aerothermal models to survive reentry. The statistical tools use this information to compute the probability that one or more of the surviving debris might hit a person on the ground and cause one or more casualties. The statistical portion of the analysis relies on a number of assumptions about how the debris footprint and the human population are distributed in latitude and longitude, and how to use that information to arrive at realistic risk numbers. Because this information is used in making policy and engineering decisions, it is important that these assumptions be tested using empirical data. This study uses the latest database of known uncontrolled reentry locations measured by the United States Department of Defense. The predicted ground footprint distributions of these objects are based on the theory that their orbits behave basically like simple Kepler orbits. However, there are a number of factors in the final stages of reentry - including the effects of gravitational harmonics, the effects of the Earth s equatorial bulge on the atmosphere, and the rotation of the Earth and atmosphere - that could cause them to diverge from simple Kepler orbit behavior and possibly change the probability of reentering over a given location. In this paper, the measured latitude and longitude distributions of these objects are directly compared with the predicted distributions, providing a fundamental empirical test of the model assumptions.

Matney, Mark↗