SEARCH · Engineering Papers

Results for “Statistical Testing Approach”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

All sources NASA NTRS DOE OSTI

DOE OSTI2016

A Statistical Testing Approach for Quantifying Software Reliability; Application to an Example System

The U.S. Nuclear Regulatory Commission (NRC) encourages the use of probabilistic risk assessment (PRA) technology in all regulatory matters, to the extent supported by the state-of-the-art in PRA methods and data. Although much has been accomplished in the area of risk-informed regulation, risk assessment for digital systems has not been fully developed. The NRC established a plan for research on digital systems to identify and develop methods, analytical tools, and regulatory guidance for (1) including models of digital systems in the PRA's of nuclear power plants (NPPs), and, (2) incorporating digital systems in the NRC's risk-informed licensing and oversight activities. Under NRC's sponsorship, Brookhaven National Laboratory (BNL) explored approaches for addressing the failures of digital instrumentation and control (I and C) systems in the current NPP PRA framework. Specific areas investigated included PRA modeling digital hardware , development of a philosophical basis for defining software failure , and identification of desirable attributes of quantitative software reliability methods . Based on the earlier research, statistical testing is considered a promising method for quantifying software reliability. It is widely recognized that software failures are due to the triggering of pre-existing defects by the software's operational environment. Software defects can arise from errors made in user requirements or coding errors introduced during the developmental process. The software's operational environment includes factors such as the time history of digital system inputs, communication interfaces, the internal state of the digital system, and external conditions. Thus, software reliability is a function of both the number of pre-existing defects and the presence of a triggering condition caused by the manner in which the software is used. In this paper, we describe a statistical software-testing approach for quantifying software reliability and applied it to the loop-operating control system (LOCS) of an experimental loop of the Advanced Test Reactor (ATR) at Idaho National Laboratory (INL). The work involved collaboration between BNL and INL. The objectives of the study include: (1) Development of a statistical testing approach for estimating software failure probability on demand, the results of which are suitable for including in a probabilistic risk assessment (PRA); and, (2) Application of the approach to the LOCS to estimate its failure probability, and obtain insights into the feasibility, practicality, and usefulness of the estimation in models of digital systems for inclusion in nuclear power plants' PRAs. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

DOE OSTI2016

A Statistical Testing Approach for Quantifying Software Reliability; Application to an Example System

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

DOE OSTI2016

A Statistical Testing Approach for Quantifying Software Reliability; Application to an Example System

N/A

42 ENGINEERING↗

DOE OSTI2020

Statistical Modeling and Analysis of k -Layer Coverage of Two-Dimensional Materials in Inkjet Printing Processes

Two-dimensional layered materials/flakes, also known as crystalline atom-thick layer nanosheets, have recently been receiving great attention in electronics fabrication due to their unique and intriguing properties. The k-layer coverage area (i.e., the area covered by k number of overlapping layers) of the printed flake pattern significantly impacts on the properties of the printed electronics. In this work, we constructed a statistical model to describe the k-layer coverage of randomly distributed two-dimensional materials. A series of results are obtained to provide not only the expectation but also the variance of the coverage area. The boundary effects on the random flakes coverage are also studied. In addition, an approximated statistical testing approach is also developed in this work to detect abnormal coverage patterns. Furthermore, the case studies based on simulated data and real flakes images obtained from the inkjet printing process demonstrate the accuracy and effectiveness of the proposed model and analysis methods.

36 MATERIALS SCIENCE↗

DOE OSTI2020

Causal Interactions between Southern Ocean Polynyas and High-Latitude Atmosphere–Ocean Variability

Abstract Weddell Sea open-ocean polynyas have been observed to occasionally release heat from the deep ocean to the atmosphere, indicating that their sporadic appearances may be an important feature of high-latitude atmosphere–ocean variability. Yet, observations of the phenomenon are sparse and many standard-resolution models represent these features poorly, if at all. We use a fully coupled, synoptic-scale preindustrial control simulation of the Energy Exascale Earth System Model (E3SMv0-HR) to effectively simulate open-ocean polynyas and investigate their role in the climate system. Our approach employs statistical tests of Granger causality to diagnose local and remote drivers of, and responses to, polynya heat loss on interannual to decadal time scales. First, we find that polynya heat loss Granger causes a persistent increase in surface air temperature over the Weddell Sea, strengthening the local cyclonic wind circulation. Along with responding to polynyas, atmospheric conditions also facilitate their development. When the Southern Ocean experiences a rapid poleward shift in the circumpolar westerlies following a prolonged negative phase of the southern annular mode (SAM), Weddell Sea salinity increases, promoting density destratification and convection in the water column. Finally, we find that the reduction of surface heat fluxes during periods of full ice cover is not fully compensated by ocean heat transport into the high latitudes. This imbalance leads to a buildup of ocean heat content that supplies polynya heat loss. These results disentangle the complex, coupled climate processes that both enable the polynya’s existence and respond to it, providing insights to improve the representation of these highly episodic sea ice features in climate models.

Kaufman, Zachary S.↗

DOE OSTI2025

Testing convolutional neural network based deep learning systems: a statistical metamorphic approach

Machine learning technology spans many areas and today plays a significant role in addressing a wide range of problems in critical domains,i.e., healthcare, autonomous driving, finance, manufacturing, cybersecurity,etc. Metamorphic testing (MT) is considered a simple but very powerful approach in testing such computationally complex systems for which either an oracle is not available or is available but difficult to apply. Conventional metamorphic testing techniques have certain limitations in verifying deep learning-based models (i.e., convolutional neural networks (CNNs)) that have a stochastic nature (because of randomly initializing the network weights) in their training. In this article, we attempt to address this problem by using a statistical metamorphic testing (SMT) technique that does not require software testers to worry about fixing the random seeds (to get deterministic results) to verify the metamorphic relations (MRs). We propose seven MRs combined with different statistical methods to statistically verify whether the program under test adheres to the relation(s) specified in the MR(s). We further use mutation testing techniques to show the usefulness of the proposed approach in the healthcare space and test two CNN-based deep learning models (used for pneumonia detection among patients). The empirical results show that our proposed approach uncovers 85.71% of the implementation faults in the classifiers under test (CUT). Furthermore, we also propose an MRs minimization algorithm for the CUT, thus saving computational costs and organizational testing resources.

Computer Science↗

DOE OSTI2026

Enhanced climate reproducibility testing with false discovery rate correction

Simulating the Earth's climate is an important and complex problem, thus climate models are similarly complex, comprised of millions of lines of code. In order to appropriately utilize the latest computational and software infrastructure advancements in Earth system models running on modern hybrid computing architectures to improve their performance, precision, accuracy, or all three; it is important to ensure that model simulations are repeatable and robust. This introduces the need for establishing statistical or non-bit-for-bit reproducibility, since bit-for-bit reproducibility may not always be achievable. Here, we propose a short-simulation ensemble-based test for an atmosphere model to evaluate the null hypothesis that modified model results are statistically equivalent to that of the original model. We implement this test in version 2 of the US Department of Energy's Energy Exascale Earth System Model (E3SM). The test evaluates a standard set of output variables across the two simulation ensembles and uses a false discovery rate correction to account for multiple testing. The false positive rates of the test are examined using re-sampling techniques on large simulation ensembles and are found to be lower than the currently implemented bootstrapping-based testing approach in E3SM. We also evaluate the statistical power of the test using perturbed simulation ensemble suites, each with a progressively larger magnitude of change to a tuning parameter. The new test is generally found to exhibit more statistical power than the current approach, being able to detect smaller changes in parameter values with higher confidence.

Kelleher, Michael E. [Oak Ridge National Laborator↗

DOE OSTI2024

Predictive Modeling of NOx Emissions from Lean Direct Injection of Hydrogen and Hydrogen/Natural Gas Blends Using Flame Imaging and Machine Learning

This research paper explores the use of machine learning to relate images of flame structure and luminosity to measured NOx emissions. Images of reactions produced by 16 aero-engine derived injectors for a ground-based turbine operated on a range of fuel compositions, air pressure drops, preheat temperatures and adiabatic flame temperatures were captured and postprocessed. The experimental investigations were conducted under atmospheric conditions, capturing CO, NO and NOx emissions data and OH* chemiluminescence images from 27 test conditions. The injector geometry and test conditions were based on a statistically designed test plan. These results were first analyzed using the traditional analysis approach of analysis of variance (ANOVA). The statistically based test plan yielded 432 data points, leading to a correlation for NOx emissions as a function of injector geometry, test conditions and imaging responses, with 70.2% accuracy. As an alternative approach to predicting emissions using imaging diagnostics as well as injector geometry and test conditions, a random forest machine learning algorithm was also applied to the data and was able to achieve an accuracy of 82.6%. This study offers insights into the factors influencing emissions in ground-based turbines while emphasizing the potential of machine learning algorithms in constructing predictive models for complex systems.

08 HYDROGEN↗

DOE OSTI2018

Spatially Dependent Multiple Testing Under Model Misspecification, With Application to Detection of Anthropogenic Influence on Extreme Climate Events

The Weather Risk Attribution Forecast (WRAF) is a forecasting tool that uses output from global climate models to make simultaneous attribution statements about whether and how greenhouse gas emissions have contributed to extreme weather across the globe. However, in conducting a large number of simultaneous hypothesis tests, the WRAF is prone to identifying false “discoveries.” A common technique for addressing this multiple testing problem is to adjust the procedure in a way that controls the proportion of true null hypotheses that are incorrectly rejected, or the false discovery rate (FDR). Unfortunately, generic FDR procedures suffer from low power when the hypotheses are dependent, and techniques designed to account for dependence are sensitive to misspecification of the underlying statistical model. In this paper, we develop a Bayesian decision-theoretical approach for dependent multiple testing and a nonparametric hierarchical statistical model that flexibly controls false discovery and is robust to model misspecification. We illustrate the robustness of our procedure to model error with a simulation study, using a framework that accounts for generic spatial dependence and allows the practitioner to flexibly specify the decision criteria. Finally, we apply our procedure to several seasonal forecasts and discuss implementation for the WRAF workflow. Lastly, supplementary materials for this article, including a standardized description of the materials available for reproducing the work, are available as an online supplement.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2021

Thermodynamics of order and randomness in dopant distributions inferred from atomically resolved imaging

Abstract Exploration of structure-property relationships as a function of dopant concentration is commonly based on mean field theories for solid solutions. However, such theories that work well for semiconductors tend to fail in materials with strong correlations, either in electronic behavior or chemical segregation. In these cases, the details of atomic arrangements are generally not explored and analyzed. The knowledge of the generative physics and chemistry of the material can obviate this problem, since defect configuration libraries as stochastic representation of atomic level structures can be generated, or parameters of mesoscopic thermodynamic models can be derived. To obtain such information for improved predictions, we use data from atomically resolved microscopic images that visualize complex structural correlations within the system and translate them into statistical mechanical models of structure formation. Given the significant uncertainties about the microscopic aspects of the material’s processing history along with the limited number of available images, we combine model optimization techniques with the principles of statistical hypothesis testing. We demonstrate the approach on data from a series of atomically-resolved scanning transmission electron microscopy images of Mo x Re 1- x S 2 at varying ratios of Mo/Re stoichiometries, for which we propose an effective interaction model that is then used to generate atomic configurations and make testable predictions at a range of concentrations and formation temperatures.

25 ENERGY STORAGE↗

DOE OSTI2020

XRD-PUAT (X-Ray Diffraction - Parameter Uncertainty Analysis Toolkit)

This XRD uncertainty toolkit was built to investigate uncertainty and local least squares topology over a specified parameter space on a single histogram in a gpx GSAS-II file. The parameter uncertainty can be investigated using either a frequentist F-test approach or a Bayesian Inference statistical inversion method on a weighted least squares or peak fit refinements.SAND Number: SAND2020-12227 M Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Moore, Alexander↗

DOE OSTI2016

Integrating Symbolic and Statistical Methods for Testing Intelligent Systems Applications to Machine Learning and Computer Vision

Embedded intelligent systems ranging from tiny im- plantable biomedical devices to large swarms of autonomous un- manned aerial systems are becoming pervasive in our daily lives. While we depend on the flawless functioning of such intelligent systems, and often take their behavioral correctness and safety for granted, it is notoriously difficult to generate test cases that expose subtle errors in the implementations of machine learning algorithms. Hence, the validation of intelligent systems is usually achieved by studying their behavior on representative data sets, using methods such as cross-validation and bootstrapping.In this paper, we present a new testing methodology for studying the correctness of intelligent systems. Our approach uses symbolic decision procedures coupled with statistical hypothesis testing to. We also use our algorithm to analyze the robustness of a human detection algorithm built using the OpenCV open-source computer vision library. We show that the human detection implementation can fail to detect humans in perturbed video frames even when the perturbations are so small that the corresponding frames look identical to the naked eye.

Jha, Sumit Kumar↗

DOE OSTI2018

Bootstrap inversion technique for atmospheric trace gas source detection and quantification using long open-path laser measurements

Advances in natural gas extraction technology have led to increased activity in the production and transport sectors in the United States and, as a consequence, an increased need for reliable monitoring of methane leaks to the atmosphere. We present a statistical methodology in combination with an observing system for the detection and attribution of fugitive emissions of methane from distributed potential source location landscapes such as natural gas production sites. We measure long (> 500 m), integrated open-path concentrations of atmospheric methane using a dual frequency comb spectrometer and combine measurements with an atmospheric transport model to infer leak locations and strengths using a novel statistical method, the non-zero minimum bootstrap (NZMB). The new statistical method allows us to determine whether the empirical distribution of possible source strengths for a given location excludes zero. Using this information, we identify leaking source locations (i.e., natural gas wells) through rejection of the null hypothesis that the source is not leaking. The method is tested with a series of synthetic data inversions with varying measurement density and varying levels of model–data mismatch. It is also tested with field observations of (1) a non-leaking source location and (2) a source location where a controlled emission of 3.1 × 10 -5 kg s -1 of methane gas is released over a period of several hours. This series of synthetic data tests and outdoor field observations using a controlled methane release demonstrates the viability of the approach for the detection and sizing of very small leaks of methane across large distances (4 + km 2 in synthetic tests). The field tests demonstrate the ability to attribute small atmospheric enhancements of 17 ppb to the emitting source location against a background of combined atmospheric (e.g., background methane variability) and measurement uncertainty of 5 ppb (1 σ ), when measurements are averaged over 2 min. The results of the synthetic and field data testing show that the new observing system and statistical approach greatly decreases the incidence of false alarms (that is, wrongly identifying a well site to be leaking) compared with the same tests that do not use the NZMB approach and therefore offers increased leak detection and sizing capabilities.

03 NATURAL GAS↗

DOE OSTI2017

A statistical approach to combining multisource information in one-class classifiers

A new method is introduced in this paper for combining information from multiple sources to support one-class classification. The contributing sources may represent measurements taken by different sensors of the same physical entity, repeated measurements by a single sensor, or numerous features computed from a single measured image or signal. The approach utilizes the theory of statistical hypothesis testing, and applies Fisher's technique for combining p-values, modified to handle nonindependent sources. Classifier outputs take the form of fused p-values, which may be used to gauge the consistency of unknown entities with one or more class hypotheses. The approach enables rigorous assessment of classification uncertainties, and allows for traceability of classifier decisions back to the constituent sources, both of which are important for high-consequence decision support. Application of the technique is illustrated in two challenge problems, one for skin segmentation and the other for terrain labeling. Finally, the method is seen to be particularly effective for relatively small training samples.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Shared micromobility as a first- and last-mile transit solution? Spatiotemporal insights from a novel dataset

The first- and last-mile (FM/LM) problem is a major deterrent to public transit use. With the rise of shared micromobility options such as shared e-scooters in recent years, there is a growing interest in understanding their potential to serve as a last-mile transit solution. However, empirical data regarding the integrated use of shared micromobility and public transit have been limited so far. As a result, much is unknown regarding the spatiotemporal patterns and characteristics of shared micromobility trips serving as an FM/LM connection to transit. Here, this paper addresses these knowledge gaps by leveraging a novel dataset (i.e., the Spin post-ride survey dataset) that records thousands of transit-connecting shared e-scooter trips in Washington DC. Specifically, we used the dataset to reveal the spatiotemporal patterns of transit-connecting shared e-scooter trips in Washington DC, resulting in some major policy insights regarding the integral use of shared e-scooters and public transit. We further leveraged the dataset to validate if and to what extent a commonly applied buffer-zone approach can infer FM/LM micromobility trips accurately. Statistical tests showed that the actual FM/LM Spin e-scooter trips differ from inferred FM/LM Spin e-scooter trips in both spatial and temporal dimensions. This indicates that the common practice of inferring FM/LM micromobility trips with a buffer-zone approach can lead to inaccurate estimates of transit-connecting micromobility trips.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

DOE OSTI2022

Testing the impact of electromagnetic fields on the directed flow of constituent quarks in heavy-ion collisions

It has been proposed that strong electromagnetic fields produced in the early stages of heavy-ion collisions can lead to splitting of the rapidity-odd directed flow of positive and negative hadrons. For light hadrons, the interpretation of such measurements is complicated by the low magnitude of directed flow as well as by ambiguities arising from transported quarks. To overcome these complications, we propose measurements using only hadrons carrying produced quarks (u¯, d¯, s s¯ ). Here, we discuss how to identify the kinematics where such hadrons are produced via the coalescence mechanism and therefore their flow is the sum of the flow of their constituent quarks. With this sum rule verified for certain combinations of hadrons, the expected systematic violation of this rule with increasing electric charge can be measured, which could be a consequence of the electromagnetic fields produced in the collisions. Our approach can be tested with the high statistics data from Phase II of the Beam Energy Scan program at the Relativistic Heavy Ion Collider.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2015

An Overview of High-performance Parallel Big Data transfers over multiple network channels with Transport Layer Security (TLS) and TLS plus Perfect Forward Secrecy (PFS)

This Technical Note provides an overview of high-performance parallel Big Data transfers with and without encryption for data in-transit over multiple network channels. It shows that with the parallel approach, it is feasible to carry out high-performance parallel "encrypted" Big Data transfers without serious impact to throughput. But other impacts, e.g. the energy-consumption part should be investigated. It also explains our rationales of using a statistics-based approach for gaining understanding from test results and for improving the system. The presentation is of high-level nature. Nevertheless, at the end we will pose some questions and identify potentially fruitful directions for future work.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2018

Challenges of Using Indistinguishable from Background as a Release Criterion at FUSRAP Projects - 18157

The US Army Corps of Engineers (USACE) utilizes the Multi-Agency Radiation Survey and Site Investigation Manual (MARSSIM) approach to site release on Formerly Utilized Sites Remedial Action Program (FUSRAP) projects. Use of Indistinguishable from Background (IFB) as a release criterion is discussed in MARSSIM as Scenario B; however, the discussion is arguably limited and refers the user to other documents. USACE has utilized the IFB approach at select FUSRAP projects. USACE experience is that use of IFB as a release criterion has been difficult to implement at best. There are many challenges in using the IFB approach. These include: release criterion is required per MARSSIM; multiple reference areas need to be established; and data presentation. Potential methods to overcome these challenges are discussed. USACE has utilized alternative statistical approaches as well as the IFB approach discussed in MARSSIM for specific purposes at FUSRAP sites. Alternatives to the IFB approach discussed in this paper include other statistical tests such as the Tukey Kramer test. These alternatives may be better suited to release an area of a property prior to the development of release criteria. USACE has found using the IFB criterion useful in certain specific circumstances. These include post MARSSIM Scenario A final status survey data evaluations and for review of historical data. The challenges, alternatives, and circumstances to use the IFB approach are discussed. Other environmental remediation projects attempting to utilize IFB as a criterion may benefit from the USACE experiences as discussed herein. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗