Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Statistical Testing Approach”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 271 records · Page 15
All sourcesNASA NTRSDOE OSTI
DOE OSTI2014

Scaling Relationships Based on Scaled Tank Mixing and Transfer Test Results

This report documents the statistical analyses performed (by Pacific Northwest National Laboratory for Washington River Protection Solutions) on data from 26 tests conducted using two scaled tanks (43 and 120 inches) in the Small Scale Mixing Demonstration platform. The 26 tests varied several test parameters, including mixer-jet nozzle velocity, base simulant, supernatant viscosity, and capture velocity. For each test, samples were taken pre-transfer and during five batch transfers. The samples were analyzed for the concentrations (lbs/gal slurry) of four primary components in the base simulants (gibbsite, stainless steel, sand, and ZrO2). The statistical analyses including modeling the component concentrations as functions of test parameters using stepwise regression with two different model forms. The resulting models were used in an equivalent performance approach to calculate values of scaling exponents (for a simple geometric scaling relationship) as functions of the parameters in the component concentration models. The resulting models and scaling exponents are displayed in tables and graphically. The sensitivities of component concentrations and scaling exponents to the test parameters are presented graphically. These results will serve as inputs to subsequent work by other researchers to develop scaling relationships that are applicable to full-scale tanks.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2022

A Qualitative Strategy for Fusion of Physics into Empirical Models for Process Anomaly Detection

To facilitate the automated online monitoring of power plants, a systematic and qualitative strategy for anomaly detection is presented. This strategy is essential to provide credible reasoning on why and when an empirical versus hybrid (i.e., physics-supported) approach should be used and to determine the ideal mix of these two approaches for a defined anomaly detection scope. Empirical methods are usually based on pattern, statistical, and causal inference. Hybrid methods include the use of physics models to train and test data methods, reduce data dimensionality, reduce data-model complexity, augment data, and reduce empirical uncertainty; hybrid methods also include the use of data to tune physics models. The presented strategy is driven by key decision points related to data relevance, simple modeling feasibility, data inference, physics-modeling value, data dimensionality, physics knowledge, method of validation, performance, data availability, and suitability for training and testing, cause-effect, entropy inference, and model fitting. The strategy is demonstrated through a pilot use case for the application of anomaly detection to capture a valve packing leak at the high-pressure coolant injection system of a nuclear power plant.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
NASA NTRS1987

A computer program for model verification of dynamic systems

Dynamic model verification is the process whereby an analytical model of a dynamic system is compared with experimental data, and then qualified for future use in predicting system response in a different dynamic environment. There are various ways to conduct model verification. The approach adopted in MOVER II employs Bayesian statistical parameter estimation. Unlike curve fitting whose objective is to minimize the difference between some analytical function and a given quantity of test data (or curve), Bayesian estimation attempts also to minimize the difference between the parameter values of that function (the model) and their initial estimates, in a least squares sense. The objectives of dynamic model verification, therefore, are to produce a model which: (1) is in agreement with test data, (2) will assist in the interpretation of test data, (3) can be used to help verify a design, (4) will reliably predict performance, and (5) in the case of space structures, facilitate dynamic control.

Chrostowski, J. D.↗
NASA NTRS1985

Development of thermomechanical life prediction models for thermal barrier coatings

Thermal barrier coatings (TBCs) for turbine airfoils in high-performance engines represent an advanced materials technology with both performance and durability benefits. The foremost TBC benefit is the reduction of heat transferred into air-cooled components. To achieve these benefits, however, the TBC system must be reliable. Mechanistic thermomechanical and thermochemical life models and statistically significant design data are therefore required for the reliable exploitation of TBC benefits on gas turbine airfoils. Garrett's NASA-HOST Program (NAS3-23945) is designed to fulfill these requirements. This program focuses on predicting the lives of two types of strain-tolerant and oxidation-resistant TBC systems that are produced by commercial coating suppliers to the gas turbine industry. The plasma-sprayed TBC system, composed of a low-pressure plasma-spray (LPPS) applied oxidation resistant NiCrAlY bond coating, and an air-plasma-sprayed yttria partially stabilized zirconia insulated layer is applied by both Chromalloy and Klock. The second type of TBC is applied by the electron beam-physical vapor deposition process by Temescal. Thermomechanical life models are being tailored to predict TBC strain tolerance in terms of materials, engine, and mission parameters. Continuum and fracture mechanics approaches and statistical methods are being evaluated to develop tensile and compressive strain functions required to drive a mission analysis capable thermomechanical life model for TBCs. Results of initial testing to calibrate these life models will be presented.

Strangman, T. E.↗
DOE OSTI2018

Bayesian approach to model-based extrapolation of nuclear observables

Here, we considered 10 global models based on nuclear density functional theory with realistic energy density functionals as well as two more phenomenological mass models. The emulators of two-neutron separation energy residuals and Bayesian confidence intervals defining theoretical error bars were constructed using Bayesian Gaussian processes and Bayesian neural networks. By establishing statistical methodology and parameters, we carried out extrapolations toward the two-neutron dripline. While both Gaussian processes (GP) and Bayesian neural networks reduce the root-mean-square (rms) deviation from experiment significantly, GP offers a better and much more stable performance. The increase in the predictive power of microscopic models aided by the statistical treatment is quite astonishing: The resulting rms deviations from experiment on the testing dataset are similar to those of more phenomenological models. The estimated credibility intervals on predictions make it possible to evaluate predictive power of individual models and also make quantified predictions using groups of models. The proposed robust statistical approach to extrapolation of nuclear model results can be useful for assessing the impact of current and future experiments in the context of model developments. The new Bayesian capability to evaluate residuals is also expected to impact research in the domains where experiments are currently impossible, for instance, in simulations of the astrophysical r process.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2015

Statistical tests on clustered global earthquake synthetic data sets

Abstract We study the ability of statistical tests to identify nonrandom features of earthquake catalogs, with a focus on the global earthquake record since 1900. We construct four types of synthetic data sets containing varying strengths of clustering, with each data set containing on average 10,000 events over 100 years with magnitudes above M = 6. We apply a suite of statistical tests to each synthetic realization in order to evaluate the ability of each test to identify the sequences of events as nonrandom. Our results show that detection ability is dependent on the quantity of data, the nature of the type of clustering, and the specific signal used in the statistical test. Data sets that exhibit a stronger variation in the seismicity rate are generally easier to identify as nonrandom for a given background rate. We also show that we can address this problem in a Bayesian framework, with the clustered data sets as prior distributions. Using this new Bayesian approach, we can place quantitative bounds on the range of possible clustering strengths that are consistent with the global earthquake data. At M = 7, we can estimate 99th percentile confidence bounds on the number of triggered events, with an upper bound of 20% of the catalog for global aftershock sequences, with a stronger upper bound on the fraction of triggered events of 10% for long‐term event clusters. At M = 8, the bounds are less strict due to the reduced number of events. However, our analysis shows that other types of clustering could be present in the data that we are unable to detect. Our results aid in the interpretation of the results of statistical tests on earthquake catalogs, both worldwide and regionally.

Daub, Eric G.↗
DOE OSTI2017

Simultaneous 3D MR elastography of the in vivo mouse brain

The feasibility of sample interval modulation (SLIM) magnetic resonance elastography (MRE) for the in vivo mouse brain is assessed, and an alternative SLIM-MRE encoding method is introduced. In SLIMMRE, the phase accumulation for each motion direction is encoded simultaneously by varying either the start time of the motion encoding gradient (MEG), SLIM-phase constant (SLIM-PC), or the initial phase of the MEG, SLIM-phase varying (SLIM-PV). SLIM-PC provides gradient moment nulling, but the mutual gradient shift necessitates increased echo time (TE). SLIM-PV requires no increased TE, but exhibits nonuniform flow compensation. Comparison was to conventional MRE using six C57BL/6 mice. For SLIMPC, the Spearman’s rank correlation to conventional MRE for the shear storage and loss modulus images were 80% and 76%, respectively, and likewise for SLIM-PV, 73% and 69%, respectively. The results of the Wilcoxon rank sum test showed that there were no statistically significant differences between the spatially averaged shear moduli derived from conventional-MRE, SLIM-PC, and SLIM-PV acquisitions. Both SLIM approaches were comparable to conventional MRE scans with Spearman’s rank correlation of 69%-80% and with 3 times reduction in scan time. As a result, the SLIM-PC method had the best correlation, and SLIM-PV may be a useful tool in experimental conditions, where both measurement time and T2 relaxation is critical.

60 APPLIED LIFE SCIENCES↗
NASA NTRS1991

Effect of short-term exposure to stereoscopic three-dimensional flight displays on real-world depth perception

High-fidelity color pictorial displays that incorporate depth cues in the display elements are currently available. Depth cuing applied to advanced head-down flight display concepts potentially enhances the pilot's situational awareness and improves task performance. Depth cues provided by stereopsis exhibit constraints that must be fully understood so depth cuing enhancements can be adequately realized and exploited. A fundamental issue (the goal of this investigation) is whether the use of head-down stereoscopic displays in flight applications degrade the real-world depth perception of pilots using such displays. Stereoacuity tests are used in this study as the measure of interest. Eight pilots flew repeated simulated landing approaches using both nonstereo and stereo 3-D head-down pathway-in-the-sky displays. At this decision height of each approach (where the pilot changes to an out-the-window view to obtain real-world visual references) the pilots changed to a stereoacuity test that used real objects. Statistical analysis of stereoacuity measures (data for a control condition of no exposure to any electronic flight display compared with data for changes from nonstereo and from stereo displays) reveals no significant differences for any of the conditions. Therefore, changing from short-term exposure to a head-down stereo display has no more effect on real-world relative depth perception than does changing from a nonstereo display. However, depth perception effects based on sized and distance judgements and on long-term exposure remain issues to be investigated.

Busquets, Anthony M.↗
NASA NTRS2014

A Multi-Parameter Approach for Calculating Crack Instability

An accurate fracture control analysis of spacecraft pressure systems, boosters, rocket hardware and other critical low-cycle fatigue cases where the fracture toughness highly impacts cycles to failure requires accurate knowledge of the material fracture toughness. However, applicability of the measured fracture toughness values using standard specimens and transferability of the values to crack instability analysis of the realistically complex structures is refutable. The commonly used single parameter Linear Elastic Fracture Mechanics (LEFM) approach which relies on the key assumption that the fracture toughness is a material property would result in inaccurate crack instability predictions. In the past years extensive studies have been conducted to improve the single parameter (K-controlled) LEFM by introducing parameters accounting for the geometry or in-plane constraint effects]. Despite the importance of the thickness (out-of-plane constraint) effects in fracture control problems, the literature is mainly limited to some empirical equations for scaling the fracture toughness data] and only few theoretically based developments can be found. In aerospace hardware where the structure might have only one life cycle and weight reduction is crucial, reducing the design margin of safety by decreasing the uncertainty involved in fracture toughness evaluations would result in lighter hardware. In such conditions LEFM would not suffice and an elastic-plastic analysis would be vital. Multi-parameter elastic plastic crack tip field quantifying developments combined with statistical methods] have been shown to have the potential to be used as a powerful tool for tackling such problems. However, these approaches have not been comprehensively scrutinized using experimental tests. Therefore, in this paper a multi-parameter elastic-plastic approach has been used to study the crack instability problem and the transferability issue by considering the effects of geometrical constraints as well as the thickness. The feasibility of the approach has been examined using a wide range of specimen geometries and thicknesses manufactured from 7075-T7351 aluminum alloy.

Zanganeh, M.↗
DOE OSTI2020

Improving air leakage prediction of buildings using the fan pressurization method with the Weighted Line of Organic Correlation

In many countries, the fan pressurization method is the most frequently chosen approach for measuring the air leakage of houses. The measurements are usually performed at pressures that far exceed pressures to which buildings are exposed to under normal conditions. A fit of these tests to the power-law formulation allows an extrapolation to data points outside the measured pressure range. With the Ordinary Least Square (OLS) fitting method, the pressure exponent and flow coefficient can be determined. However, the measurement results are highly sensitive to uncertainties induced by external factors like changing wind conditions during the tests, which is neglected by OLS. This may lead to errors in the prediction of flows at lower pressures. The Weighted Line of Organic Correlation (WLOC) is an alternative approach and takes measurement uncertainty into account. In this paper, a statistical analysis of an extensive data set of pressurization measurements has been performed. Both regression techniques have been compared for almost 7500 fan pressurization measurements of six houses in 109 different house leak configurations. The variability in predicting pressure exponent and flow coefficient for both WLOC and OLS regression was analyzed using probability density functions. It was found that the Weighted Line of Organic Correlation significantly decreases the uncertainty in predicting pressure exponent, flow coefficient, and other low-pressure air leakage metrics compared to the Ordinary Least Square fitting. Here, the authors highly recommend the implementation of WLOC in current measurement standards and test equipment.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2015

Assessment of Dimensional Integrity and Spatial Defect Localization in Additive Manufacturing Using Spectral Graph Theory

The ability of additive manufacturing (AM) to produce components with virtually any ge-ometry presents a unique challenge in terms of quantifying the quality of the part. In this paper, a novel spectral graph theory (SGT) approach is proposed for resolving two critical quality assurance concerns in AM: (1) how to quantify the dimensional integrity of complex AM components; and (2) how to isolate specific spatial locations on a component that deviate significantly from their normal condition. Here, the SGT approach is demonstrated for classifying the dimensional integrity of standardized test components produced from three different AM processes. The SGT-based topological invariant Fiedler number ( λ 2 ) was calculated from 3D point cloud coordinate measurements and used to quantify the dimensional integrity of test components. The Fiedler number was found to differ significantly for parts originating from different AM processes (statistical significance p-val. < 1%). By comparison, traditional statistical quantifiers (such as mean and standard deviation) and examination of specific facets/landmarks failed to capture these variations and proved incapable of quantifying the quality of the component in a consistent manner. The SGT approach was verified to consistently rank the quality of the AM components with a high degree of statistical confidence independent of sampling technique used (pooled standard deviation ~ 3% 10% of mean). Finally, from a practical standpoint, the SGT approach can be a powerful tool for quantifying and monitoring the dimensional integrity and overall quality of AM components, and thus encourage wider adoption of AM capabilities.

42 ENGINEERING↗
NASA NTRS2011

Signal Processing Methods for Liquid Rocket Engine Combustion Stability Assessments

The J2X Gas Generator engine design specifications include dynamic, spontaneous, and broadband combustion stability requirements. These requirements are verified empirically based high frequency chamber pressure measurements and analyses. Dynamic stability is determined with the dynamic pressure response due to an artificial perturbation of the combustion chamber pressure (bomb testing), and spontaneous and broadband stability are determined from the dynamic pressure responses during steady operation starting at specified power levels. J2X Workhorse Gas Generator testing included bomb tests with multiple hardware configurations and operating conditions, including a configuration used explicitly for engine verification test series. This work covers signal processing techniques developed at Marshall Space Flight Center (MSFC) to help assess engine design stability requirements. Dynamic stability assessments were performed following both the CPIA 655 guidelines and a MSFC in-house developed statistical-based approach. The statistical approach was developed to better verify when the dynamic pressure amplitudes corresponding to a particular frequency returned back to pre-bomb characteristics. This was accomplished by first determining the statistical characteristics of the pre-bomb dynamic levels. The pre-bomb statistical characterization provided 95% coverage bounds; these bounds were used as a quantitative measure to determine when the post-bomb signal returned to pre-bomb conditions. The time for post-bomb levels to acceptably return to pre-bomb levels was compared to the dominant frequency-dependent time recommended by CPIA 655. Results for multiple test configurations, including stable and unstable configurations, were reviewed. Spontaneous stability was assessed using two processes: 1) characterization of the ratio of the peak response amplitudes to the excited chamber acoustic mode amplitudes and 2) characterization of the variability of the peak response's frequency over the test duration. This characterization process assists in evaluating the discreteness of a signal as well as the stability of the chamber response. Broadband stability was assessed using a running root-mean-square evaluation. These techniques were also employed, in a comparative analysis, on available Fastrac data, and these results are presented here.

Kenny, R. Jeremy↗
DOE OSTI2022

Understanding and improving anode performance in an alkaline membrane electrolyzer using statistical design of experiments

The behavior of the oxygen-evolving positive electrode (i.e. anode) in the anion exchange membrane water electrolyzer (AEMEL) is complex and influenced by several factors. Very few studies have been performed to understand oxygen evolution reaction (OER) behavior by optimizing the individual factors that influence performance. Here, this study highlights the effects of catalyst loading, catalyst selection, porous transport layer (PTL) type and conductive additive content. The influence of each factor is elucidated through a design of experiments (DoE) approach with a full statistical analysis. Electrochemical data, alongside Pareto charts, parametric trends and their mutual interactions are discussed. This DoE approach is also helpful in making useful predictions and discovering new combinations to be tested. The end result was a high-performance AEMEL able to operate at a current density of 1.0 A/cm 2 at 1.80 V with IrOx OER and PtNi hydrogen evolution reaction (HER) catalysts using 0.3 M KOH fed to the anode. Even lower operating voltage was observed with PbRuOx catalyst at the anode, 1.64 V @ 1.0 A/cm 2 , though the cell decay rate was higher. Lastly, a IrOx/PtNi cell was stably operated continuously for 30 days (720 h) at 1.0 A/cm 2 . This study can serve as a guide for optimal electrode design with insights into component-performance compromises, which can aid in making design choices and performing techno-economic analyses.

08 HYDROGEN↗
DOE OSTI2015

Towards intelligent diagnostic system employing integration of mathematical and engineering model

The development of medical diagnostic system has been one of the main research fields during years. The goal of the medical diagnostic system is to place a nosological system that could ease the diagnostic evaluation normally performed by scientists and doctors. Efficient diagnostic evaluation is essentials and requires broad knowledge in order to improve conventional diagnostic system. Several approaches on developing the medical diagnostic system have been designed and tested since the earliest 60s. Attempts on improving their performance have been made which utilizes the fields of artificial intelligence, statistical analyses, mathematical model and engineering theories. With the availability of the microcomputer and software development as well as the promising aforementioned fields, medical diagnostic prototypes could be developed. In general, the medical diagnostic system consists of several stages, namely the 1) data acquisition, 2) feature extraction, 3) feature selection, and 4) classifications stages. Data acquisition stage plays an important role in converting the inputs measured from the real world physical conditions to the digital numeric values that can be manipulated by the computer system. One of the common medical inputs could be medical microscopic images, radiographic images, magnetic resonance image (MRI) as well as medical signals such as electrocardiogram (ECG) and electroencephalogram (EEG). Normally, the scientist or doctors have to deal with myriad of data and redundant to be processed. In order to reduce the complexity of the diagnosis process, only the significant features of the raw data such as peak value of the ECG signal or size of lesion in the mammogram images will be extracted and considered in the subsequent stages. Mathematical models and statistical analyses will be performed to select the most significant features to be classified. The statistical analyses such as principal component analysis and discriminant analysis as well as mathematical model of clustering technique have been widely used in developing the medical diagnostic systems. The selected features will be classified using mathematical models that embedded engineering theory such as artificial intelligence, support vector machine, neural network and fuzzy-neuro system. These classifiers will provide the diagnostic results without human intervention. Among many publishable researches, several prototypes have been developed namely NeuralPap, Neural Mammo, and Cervix Kit. The former system (NeuralPap) is an automatic intelligent diagnostic system for classifying and distinguishing between the normal and cervical cancerous cells. Meanwhile, the Cervix Kit is a portable Field-programmable gate array (FPGA)-based cervical diagnostic kit that could automatically diagnose the cancerous cell based on the images obtained during sampling test. Besides the cervical diagnostic system, the Neural Mammo system is developed to specifically aid the diagnosis of breast cancer using a fine needle aspiration image.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2023

Advancing the mechanical integrity and fragmentation behavior of reactive projectiles

A multivariant statistical approach was used to identify treatment conditions that improve the survivability of structural reactive material (SRM) projectiles upon launch and enhance energy release upon impact. The study included both mechanical testing of projectiles as well as their reactive characterization. The projectiles were launched in a high-velocity impact-ignition testing system and impacted an anvil for vented chamber calorimetry. This study examined a link between ultimate compressive stress and combustion performance. Two treatments were applied to consolidated aluminum projectiles including annealing and addition of silica (SiO 2 ) inclusions. Results showed annealing at moderate temperatures resulted in intact SRM projectiles upon launch. Adding small concentrations (1–2 wt. %) of SiO 2 to the SRM promoted fragmentation and combustion performance upon impact. Compared to the untreated projectiles, annealing with SiO 2 inclusion processing treatments improved the energy conversion efficiency from 37–84% (for untreated projectiles) up to 54–98%. In conclusion, increasing interparticle dislocation recovery by annealing while balancing inclusions promoting fragmentation upon impact was the key to optimizing combustion performance for SRM ballistic impact applications.

42 ENGINEERING↗
DOE OSTI2024

Impacts of floating-point non-associativity on reproducibility for HPC and deep learning applications

Run to run variability in parallel programs caused by floating-point non-associativity has been known to significantly affect reproducibility in iterative algorithms, due to accumulating errors. Non-reproducibility can critically affect the efficiency and effectiveness of correctness testing for stochastic programs. Recently, the sensitivity of deep learning training and inference pipelines to floating-point non-associativity has been found to sometimes be extreme. It can prevent certification for commercial applications, accurate assessment of robustness and sensitivity, and bug detection. New approaches in scientific computing applications have coupled deep learning models with high-performance computing, leading to an aggravation of debugging and testing challenges. Here we perform an investigation of the statistical properties of floating-point non-associativity within modern parallel programming models, and analyze performance and productivity impacts of replacing atomic operations with deterministic alternatives on GPUs. We examine the recently-added deterministic options in PyTorch within the context of GPU deployment for deep learning, uncovering and quantifying the impacts of input parameters triggering run to run variability and reporting on the reliability and completeness of the documentation. Finally, we evaluate the strategy of exploiting automatic determinism that could be provided by deterministic hardware, using the Groq LPUTM accelerator for inference portions of the deep learning pipeline. We demonstrate the benefits that a hardware-based strategy can provide within reproducibility and correctness efforts.

Shanmugavelu, Sanjif↗
DOE OSTI2021

Sub-pixel high-resolution imaging of high-energy x-rays inspired by sub-wavelength optical imaging

We have developed and demonstrated an image super-resolution method—XR-UNLOC: X-Ray UNsupervised particle LOCalization—for hard x-rays measured with fast-frame-rate detectors that is an adaptation of the principle of photo-activated localization microscopy (PALM) and stochastic optical reconstruction microscopy (STORM), which enabled biological fluorescence imaging at sub-optical-wavelength scales. We demonstrate the approach on experimental coherent Bragg diffraction data measured with 52 keV x-rays from a nanocrystalline sample. From this sample, we resolve the fine fringe detail of a high-energy x-ray Bragg coherent diffraction pattern to an upsampling factor of 16 of the native pixel pitch of 30 μ m of a charge-integrating fastCCD detector. This was accomplished by analysis of individual photon locations in a series of “nearly-dark” instances of the diffraction pattern that each contain only a handful of photons. Central to our approach was the adaptation of the UNLOC photon fitting routine for PALM/STORM to the hard x-ray regime to handle much smaller point spread functions, which required a different statistical test for photon detection and for sub-pixel localization. A comparison to a photon-localization strategy used in the x-ray community (“droplet analysis”) showed that XR-UNLOC provides significant improvement in super-resolution. We also developed a metric by which to estimate the limit of reliable upsampling with XR-UNLOC under a given set of experimental conditions in terms of the signal-to-noise ratio of a photon detection event and the size of the point spread function for guiding future x-ray experiments in many disciplines where detector pixelation limits must be overcome.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
NASA NTRSDate not supplied

QuantifyML: How good is my machine learning model?

This paper presents an approach, QuantifyML, which employs model counting to assess the learnability and robustness of machine learning models. Typically the efficacy of machine learning models is determined by computing their accuracy statistically on test data sets. However, this may be misleading, if the test data is not representative of the problem that is being studied. Further, two different models may have the same accuracy on a given data set, measured statistically, but may be very different in their behavior on unseen data. Also, models with high accuracy could have poor adversarial robustness. In QuantifyML, our goal is to precisely quantify the extent to which machine learning models have learned and generalized from the given data. In QuantifyML, a trained model is translated into a C program, which is fed to the CBMC model checking tool to produce a formula in Conjunctive Normal Form (CNF), which in turn is analyzed with state-of-the-art model counters to efficiently obtain precise counts w.r.t different outputs. QuantifyML enables i) evaluating the learnability of models by comparing the counts for the outputs to ground truth, expressed as logical predicates (if available), ii) comparing the performance of different models that may be built with different machine learning algorithms (e.g., decision-trees vs. neural networks), and iii) quantifying the robustness of trained models around given inputs. Our evaluation demonstrates these applications of QuantifyML on decision trees and neural networks trained to learn relational properties of graphs, for which we know the ground truth, and to perform image classification, for which we do not have the ground truth, but we can quantify local robustness.

Deep Neural Networks↗