Probabilistic recalibration of forecasts
Explore the source record for details and available documents.
SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Explore the source record for details and available documents.
Accurate energy forecasting is becoming critical due to many reasons: i ) optimal distributed energy resources operations and dispatch, ii) fault detection and diagnostics, and iii) meeting operational energy efficiency targets. Wattile uses deep learning (DL) for the building's short-term load forecasting application. Two specific types of neural networks called, Long Short Term Memory (LSTM) and Sequence-to-Sequence (S2S) models are used to make predictions. Forecasting models are trained using online historical weather and occupancy indicator data streams from the Intelligent Campus Program's data acquisition systems at the National Renewable Energy Laboratory (NREL) for main meters and sub-meters of multiple building types. These models use probabilistic methods to provide quantile-based forecasts in addition to nominal conditional median predictions of electricity consumption.
Probabilistic safety requirements currently formulated or proposed for space systems, nuclear reactor systems, nuclear weapon systems, and other types of systems that have a low-probability potential for high-consequence accidents depend on showing that the probability of such accidents is below a specified safety threshold or goal. Verification of compliance depends heavily upon synthetic modeling techniques such as PRA. To determine whether or not a system meets its probabilistic requirements, it is necessary to consider whether there are significant risks that are not fully considered in the PRA either because they are not known at the time or because their importance is not fully understood. The ultimate objective is to establish a reasonable margin to account for the difference between known risks and actual risks in attempting to validate compliance with a probabilistic safety threshold or goal. In this paper, we examine data accumulated over the past 60 years from the space program, from nuclear reactor experience, from aircraft systems, and from human reliability experience to formulate guidelines for estimating probabilistic margins to account for risks that are initially unknown or underappreciated. The formulation includes a review of the safety literature to identify the principal causes of such risks.
Human reliability analysis (HRA) is a method for evaluating human errors in a variety of complex systems such as nuclear power plants, military systems, aircraft, and chemical plants. Most HRA methods currently used by regulatory institutes or utilities are called static HRA and are carried out by simple worksheets or simple calculators. To date, there are many unsolved or intrinsic challenges in static HRA. For example, existing static HRA does not realistically model and evaluate human actions as they would be performed at actual systems. There is no method with HRA to objectively estimate the time required for human actions despite being essential to HRA processes. In addition, many HRA methods still rely on a dataset generated prior to the 1980s, from unrelated industry experience or simply from expert judgment. Accordingly, this study attempted to research how to overcome the challenges of existing HRA via dynamic risk assessment (a.k.a., simulation-based or computation-based risk assessment) techniques. First, this study developed a dynamic HRA method, named as PRocedure-based Investigation Method of EMRALD Risk Assessment – HRA (PRIMERA-HRA). The PRIMERA-HRA mainly concentrates on providing HRA analysts with specific guidelines on how to reasonably model human actions, assign human reliability data and evaluate output of simulation within a dynamic probabilistic risk assessment tool, called as Event Modeling Risk Assessment using Linked Diagrams (EMRALD). Second, this study also developed a module for performance shaping factors (i.e., the key concept in HRA quantification) applicable to dynamic HRA, then implemented it based on PRIMERA-HRA within the EMRALD tool. Third, this study developed an HRA data collection framework to support dynamic HRA, called as Simplified Human Error Experimental Program (SHEEP). Originally, the SHEEP study aimed to support static HRA and its data collection, but recently extended the scope to the new technologies such as dynamic HRA or HRA for advanced reactors. SHEEP focuses on the use of data collected from simplified simulators to complement—but not replace—data collection studies using full-scope simulators and actual operators. To date, many experiments were conducted under the SHEEP framework. Multiple analyses, such as human performance analysis, human error analysis, task complexity analysis, learning effect analysis and time distribution analysis, were also carried out using the collected data. Then, based on the major insights, an approach to inferring full-scope data based on simplified simulator data was proposed. The PRIMERA-HRA and SHEEP research are expected to evaluate human actions more realistically than existing static HRA, provide an opportunity to collect more HRA data with reasonable cost and labor, then contribute to enhance the quality of HRA.
Human reliability analysis (HRA) is a method for evaluating human errors in a variety of complex systems such as nuclear power plants, military systems, aircraft, and chemical plants. Most HRA methods currently used by regulatory institutes or utilities are called static HRA and are carried out by simple worksheets or simple calculators. To date, there are many unsolved or intrinsic challenges in static HRA. For example, existing static HRA does not realistically model and evaluate human actions as they would be performed at actual systems. There is no method with HRA to objectively estimate the time required for human actions despite being essential to HRA processes. In addition, many HRA methods still rely on a dataset generated prior to the 1980s, from unrelated industry experience or simply from expert judgment. Accordingly, this study attempted to research how to overcome the challenges of existing HRA via dynamic risk assessment (a.k.a., simulation-based or computation-based risk assessment) techniques. First, this study developed a dynamic HRA method, named as PRocedure-based Investigation Method of EMRALD Risk Assessment – HRA (PRIMERA-HRA). The PRIMERA-HRA mainly concentrates on providing HRA analysts with specific guidelines on how to reasonably model human actions, assign human reliability data and evaluate output of simulation within a dynamic probabilistic risk assessment tool, called as Event Modeling Risk Assessment using Linked Diagrams (EMRALD). Second, this study also developed a module for performance shaping factors (i.e., the key concept in HRA quantification) applicable to dynamic HRA, then implemented it based on PRIMERA-HRA within the EMRALD tool. Third, this study developed an HRA data collection framework to support dynamic HRA, called as Simplified Human Error Experimental Program (SHEEP). Originally, the SHEEP study aimed to support static HRA and its data collection, but recently extended the scope to the new technologies such as dynamic HRA or HRA for advanced reactors. SHEEP focuses on the use of data collected from simplified simulators to complement—but not replace—data collection studies using full-scope simulators and actual operators. To date, many experiments were conducted under the SHEEP framework. Multiple analyses, such as human performance analysis, human error analysis, task complexity analysis, learning effect analysis and time distribution analysis, were also carried out using the collected data. Then, based on the major insights, an approach to inferring full-scope data based on simplified simulator data was proposed. The PRIMERA-HRA and SHEEP research are expected to evaluate human actions more realistically than existing static HRA, provide an opportunity to collect more HRA data with reasonable cost and labor, then contribute to enhance the quality of HRA.
Was the emergence of life a predictable outcome of chemical evolution on earth? Could evolution produce life very different from ours? These are one of the oldest questions in the field of the origin of life that not only have broad philosophical implications but also impact how we approach the problem from the methodological standpoint. Framing the issue in terms of the dichotomy between contingency and determinism is not a fortunate because these two terms in their conventional meaning are neither mutually exclusive nor jointly exhaustive. Determinism, represented in natural sciences by Newtonian physics, relies on the assumption that every event is causally determined by a chain of previous events. In the context of the origin of life it means that once the initial conditions on the early earth have been specified further evolution follows inevitably. Considering uncertainties about conditions on the prebiotic earth, many plausible sets of initial conditions can be defined, each followed by a separate deterministic trajectory. This conventional understanding of determinism does not admit contingency. Further, it has no implications for evaluating how many sets of initial conditions lead to the emergence of life. It appears that a better framing of the problem is as follows: given plausible sets of initial conditions on the early earth how probable and broadly spread are evolutionary trajectories that lead to life? Instead of undertaking an impossible task of specifying microscopic initial conditions for all components of the system one uses a reduced representation of this system and specify only a small set of essential macroscopic parameters, values (or ranges of values) of which can be identified, inferred or estimated from experiment, theory or historical record. The following evolutionary trajectories are still governed by laws of physics and chemistry but become probabilistic and "contingency" is admitted as variations in other variables in the system. A similar reasoning is common in other fields of science, for example in statistical mechanics. Some trajectories lead to life, perhaps in different forms, whereas others do not. Of our true interest is the ratio of these two outcomes. The issue of determinism does not directly enter the picture. The debate about the likelihood of the emergence of life is quite old. One view holds that the origin of life is an event governed by chance, and the result of so many random events (contingencies) is unpredictable. This view was eloquently expressed by Monod. In his book "Chance or Necessity" he argued that life was a product of "nature's roulette." In an alternative view, expressed in particular by deDuve and Morowitz, the origin of life is considered a highly probable or even inevitable event (although its details need not be determined in every respect). Only in this sense the origin of life can be considered a "deterministic event".
This NASA educational grant has the goal of promoting probabilistic analysis methods to undergraduate and graduate UTSA engineering students. Two undergraduate-level and one graduate-level course were offered at UTSA providing a large number of students exposure to and experience in probabilistic techniques. The grant provided two research engineers from Southwest Research Institute the opportunity to teach these courses at UTSA, thereby exposing a large number of students to practical applications of probabilistic methods and state-of-the-art computational methods. In classroom activities, students were introduced to the NESSUS computer program, which embodies many algorithms in probabilistic simulation and reliability analysis. Because the NESSUS program is used at UTSA in both student research projects and selected courses, a student version of a NESSUS manual has been revised and improved, with additional example problems being added to expand the scope of the example application problems. This report documents two research accomplishments in the integration of a new sampling algorithm into NESSUS and in the testing of the new algorithm. The new Latin Hypercube Sampling (LHS) subroutines use the latest NESSUS input file format and specific files for writing output. The LHS subroutines are called out early in the program so that no unnecessary calculations are performed. Proper correlation between sets of multidimensional coordinates can be obtained by using NESSUS' LHS capabilities. Finally, two types of correlation are written to the appropriate output file. The program enhancement was tested by repeatedly estimating the mean, standard deviation, and 99th percentile of four different responses using Monte Carlo (MC) and LHS. These test cases, put forth by the Society of Automotive Engineers, are used to compare probabilistic methods. For all test cases, it is shown that LHS has a lower estimation error than MC when used to estimate the mean, standard deviation, and 99th percentile of the four responses at the 50 percent confidence level and using the same number of response evaluations for each method. In addition, LHS requires fewer calculations than MC in order to be 99.7 percent confident that a single mean, standard deviation, or 99th percentile estimate will be within at most 3 percent of the true value of the each parameter. Again, this is shown for all of the test cases studied. For that reason it can be said that NESSUS is an important reliability tool that has a variety of sound probabilistic methods a user can employ; furthermore, the newest LHS module is a valuable new enhancement of the program.
Engineering Design is one of the most important areas in engineering education. Deterministic Design Methodology (DDM) is the only design method that is taught in most engineering schools. This method does not give a direct account of uncertainties in design parameters. Hence, it is impossible to quantify the uncertainties in the response and the actual safety margin remains unknown. The desire for a design methodology tha can identify the primitive (random) variables that affect the structural behavior has led to a growing interest on Probabilistic Design Methodology (PDM). This method is gaining more recognition in industries than in educational institutions. Some of the reasons for the limited use of the PDM at the moment are that many are unaware of its potentials, and most of the software developed for PDM are very recent. The central goal of the PDM project at Tennessee State University is to introduce engineering students to the method. The students participating in the project learn about PDM and the computer codes that are available to the design engineer. The software being used of this project is NESSUS (Numerical Evaluation of Stochastic Structures Under Stress) developed under NASA probabilistic structural analysis program. NESSUS has three different modules which make it a very comprehensive computer code for PDM. A research in technology transfer through course offering in PDM is in effect a Tennessee State University. The aim is to familiarize students with the problem of uncertainties in engineering design. Included in the paper are some projects on PDM carried out by some students and faculty. The areas this method is being applied at the moment include, Design of Gears (spur and worm); Design of Shafts; Design of Statistically Indeterminate Frame Structures; Design of Helical Springs; and Design of Shock Absorbers. Some of the current results of these projects are presented.
The emergence of variational quantum applications has led to the development of automatic differentiation techniques in quantum computing. Existing work has formulated differentiable quantum programming with bounded loops, providing a framework for scalable gradient calculation by quantum means for training quantum variational applications. However, promising parameterized quantum applications, e.g., quantum walk and unitary implementation, cannot be trained in the existing framework due to the natural involvement of unbounded loops. To fill in the gap, we provide the first differentiable quantum programming framework with unbounded loops, including a newly designed differentiation rule, code transformation, and their correctness proof. Technically, we introduce a randomized estimator for derivatives to deal with the infinite sum in the differentiation of unbounded loops, whose applicability in classical and probabilistic programming is also discussed. We implement our framework with Python and Q# and demonstrate a reasonable sample efficiency. Through extensive case studies, we showcase an exciting application of our framework in automatically identifying close-to-optimal parameters for several parameterized quantum applications.
Systems of the utmost consequence need a means to establish authenticity of software and data. Cryptosystems implement authentication, but can be vulnerable to cryptographic and implementation attacks. With the threat of quantum cryptographic attacks, “post-quantum” cryptosystems (PQCs) must be henceforth used in these systems. However, the new cryptography needs new ways to, rigorously and machine-checkably, prove systems free of vulnerabilities. We propose a retargetable capability to rapidly instantiate proven correct postquantum cryptosystems through novel proof-carrying synthesis and proof-automation technique, extending those proven successful on existing systems. This capability is crucial to meeting the cryptographic requirements for future high-consequence systems. Since specifications for high consequence cryptography are presently captured in a domain specific language known as Cryptol. While this can enable convenient fully automated reasoning about Cryptol specificaitons and implementations via the Software Analysis Workbench (SAW), Cryptol has expressivity gaps, so that cryptosystems with probabilistic programming features like Falcon cannot be fully expressed in the language. Moreover, SAW’s automation fails for programs and specificaitons with inductive and recursive structure, as in the Sphincs+ PQC. Finally, Cryptol and SAW together represent some 200,000 lines of unverified Haskell, so that the any guarantees about high consequence cryptography are presently contingent on a large, unverified, yet trusted computing base. The first step of the larger project of agile, assured crpytography is therefore to provide a formal, mechanized semantics for Cryptol, so that the specifications expressed by cryptographers in Cryptol can be reasoned about and compiled into performant implementations with a foundational, machine checkable certificate of correctness. This report describes our work on this first step, culminating in the design of a certified denotational interpreter, in Coq, for core Cryptol.
Fractures provide flow paths that can potentially lead to fast migration of fluids or contaminants. A number of energy-related applications involve fluid injections that significantly perturb both the pressures and chemical composition of subsurface fluids. These perturbations can cause both mechanical deformation and chemical alteration of host rocks with potential for significant changes in permeability. In fractured rock subjected to coupled chemical and mechanical stresses, it can be difficult to predict the sign of permeability changes, let alone the magnitude. This project integrated experimental and computational studies to improve mechanistic understanding of these coupled processes and develop and test predictive models and monitoring techniques. The project involved three major components: (1) study of two-phase flow processes involving mass transfer between phases and dissolution of minerals along fracture surfaces (Detwiler et al., 2009; Detwiler, 2010); (2) study of fracture dissolution in fractures subjected to normal stresses using experimental techniques (Ameli, et al., 2013; Elkhoury et al., 2013; Elkhoury et al., 2014) and newly developed computational models (Ameli, et al., 2014); (3) evaluation of electrical resistivity tomography (ERT) as a method to detect and quantify gas leakage through a fractured caprock (Breen et al., 2012; Lochbuhler et al., 2014). The project provided support for one PhD student (Dr. Pasha Ameli; 2009-2013) and partially supported a post-doctoral scholar (Dr. Jean Elkhoury; 2010-2013). In addition, the project provided supplemental funding to support collaboration with Dr. Charles Carrigan at Lawrence Livermore National Laboratory in connection with (3) and supported one MS student (Stephen Breen; 2011-2013). Major results from each component of the project include the following: (1) Mineral dissolution in fractures occupied by two fluid phases (e.g., oil-water or water-CO 2 ) causes changes in local capillary forces and redistribution of fluids. These coupled processes enhance channel formation and the potential for development of fast flow paths through fractures. (2) Dissolution in fractures subjected to normal stress can result in behaviors ranging from development of dissolution channels and rapid permeability increases to fracture healing and significant permeability decreases. The timescales associated with advective transport of dissolved ions in the fracture, mineral dissolution rates, and diffusion within the adjacent porous matrix dictate the sign and magnitude of the resulting permeability changes. Furthermore, a high--resolution mechanistic model that couples elastic deformation of contacts and aperture-dependent dissolution rates predicts the range of observed behaviors reasonably well. (3) ERT has potential as a tool for monitoring gas leakage in deep formations. Using probabilistic inversion methods further enhances the results by providing uncertainty estimates of inverted parameters.
Recent work is Causal Reasoning has provided practical techniques for multiple fault diagnosis. These techniques provide a hypothesis/measurement diagnosis cycle. Using probabilistic methods, they choose the best measurements to make, then update fault hypotheses in response. For many applications such as computers and spacecraft, few measurement points may be accessible, or values may change quickly as the system under diagnosis operates. In these cases, a hypothesis/measurement cycle is insufficient. A technique is presented for a hypothesis/test-input/measurement diagnosis cycle. In contrast to generating tests a priori for determining device functionality, it dynamically generates tests in response to current knowledge about fault probabilities. It is shown how the mathematics previously used for measurement specification can be applied to the test input generation process. An example from an efficient implementation called Multi-Purpose Causal (MPC) is presented.
NASA (National Aeronautics and Space Administration) Johnson Space Center (JSC) Safety and Mission Assurance (S&MA) uses two human reliability analysis (HRA) methodologies. The first is a simplified method which is based on how much time is available to complete the action, with consideration included for environmental and personal factors that could influence the human's reliability. This method is expected to provide a conservative value or placeholder as a preliminary estimate. This preliminary estimate or screening value is used to determine which placeholder needs a more detailed assessment. The second methodology is used to develop a more detailed human reliability assessment on the performance of critical human actions. This assessment needs to consider more than the time available, this would include factors such as: the importance of the action, the context, environmental factors, potential human stresses, previous experience, training, physical design interfaces, available procedures/checklists and internal human stresses. The more detailed assessment is expected to be more realistic than that based primarily on time available. When performing an HRA on a system or process that has an operational history, we have information specific to the task based on this history and experience. In the case of a Probabilistic Risk Assessment (PRA) that is based on a new design and has no operational history, providing a "reasonable" assessment of potential crew actions becomes more challenging. In order to determine what is expected of future operational parameters, the experience from individuals who had relevant experience and were familiar with the system and process previously implemented by NASA was used to provide the "best" available data. Personnel from Flight Operations, Flight Directors, Launch Test Directors, Control Room Console Operators and Astronauts were all interviewed to provide a comprehensive picture of previous NASA operations. Verification of the assumptions and expectations expressed in the assessments will be needed when the procedures, flight rules and operational requirements are developed and then finalized.
NASA (National Aeronautics and Space Administration) Johnson Space Center (JSC) Safety and Mission Assurance (S&MA) uses two human reliability analysis (HRA) methodologies. The first is a simplified method which is based on how much time is available to complete the action, with consideration included for environmental and personal factors that could influence the human's reliability. This method is expected to provide a conservative value or placeholder as a preliminary estimate. This preliminary estimate or screening value is used to determine which placeholder needs a more detailed assessment. The second methodology is used to develop a more detailed human reliability assessment on the performance of critical human actions. This assessment needs to consider more than the time available, this would include factors such as: the importance of the action, the context, environmental factors, potential human stresses, previous experience, training, physical design interfaces, available procedures/checklists and internal human stresses. The more detailed assessment is expected to be more realistic than that based primarily on time available. When performing an HRA on a system or process that has an operational history, we have information specific to the task based on this history and experience. In the case of a Probabilistic Risk Assessment (PRA) that is based on a new design and has no operational history, providing a "reasonable" assessment of potential crew actions becomes more challenging. To determine what is expected of future operational parameters, the experience from individuals who had relevant experience and were familiar with the system and process previously implemented by NASA was used to provide the "best" available data. Personnel from Flight Operations, Flight Directors, Launch Test Directors, Control Room Console Operators, and Astronauts were all interviewed to provide a comprehensive picture of previous NASA operations. Verification of the assumptions and expectations expressed in the assessments will be needed when the procedures, flight rules, and operational requirements are developed and then finalized.
The purpose of this note is to draw attention to Pál’s derivation of the Pál Bell equation which is used by LLNL and LANL to model Probability of Initiation (POI) problems. Both laboratories credit the equation to Bell and Lee. Although the equation formulated by Bell and Lee and the equation derived by Pál appear to be the same, the derivation by Bell and Lee is flawed. Scattering, which was absent in Bell’s original formulation of a POI problem, was later introduced by Bell and Lee by simply replacing a fission term with a scattering term. Such a replacement, however, leads to an inaccurate average number v̅ of neutrons that is produced by a fission event. By approaching a POI problem from probabilistic point of view, Pál formulates the problem as a branching process that determines v̅ correctly. There are six reasons for writing this report. The first is to clarify the failing in Bell and Lee’s derivation. The second is to establish the origin of the equation used by the laboratories. The third is to elucidate the fixed point technique formulated by Mokhtar-Kharroubi for solving the Pál-Bell equation. The fourth is to derive the α eigenvalue that is associated with a time dependent Pál-Bell equation. The fifth is to show in Appendix A that the non-linear fission operator of the Pál-Bell equation is descended from the operator of the Galton-Watson process of stochastic theory. The sixth is to describe in Appendix B a second order positivity preserving technique that accelerates the convergence of the first order fixed point method developed by Mokhtar-Kharroubi.
In this work, we present a scheme by which a probabilistic forecasting system whose predictions have a poor probabilistic calibration may be recalibrated through the incorporation of past performance information in order to produce a new forecasting system that is demonstrably superior to the original, inasmuch as one may use it to win wagers consistently against someone who is using the original system. The scheme utilizes Gaussian process (GP) modeling to estimate a probability distribution over the probability integral transform (PIT) of a scalar predictand. The GP density estimate gives closed-form access to information entropy measures that are associated with the estimated distribution, which allows the prediction of winnings in wagers against the base forecasting system. A separate consequence of the procedure is that the recalibrated forecast has a uniform expected PIT distribution. One distinguishing feature of the procedure is that it is appropriate even if the PIT values are not i.i.d. The recalibration scheme is formulated in a framework that exploits the deep connections among information theory, forecasting, and betting. We demonstrate the effectiveness of the scheme in two case studies: a laboratory experiment with a nonlinear circuit and seasonal forecasts of the intensity of the El Nino-Southern Oscillation phenomenon.
The fact that Deterministic Design Method no longer satisfies most design needs calls for methods that will cope with the high trend in technology. The advance in computer technology has reduced the rigors that normally accompany many design analysis methods that account for uncertainties in design parameters. Probabilistic Design Methodology (PDM) is beginning to make impact in engineering design. This method is gaining more recognition in industries than in educational institutions. Some of the reasons for the limited use of the PDM at the moment are that many are unaware of its potentials, and most of the software developed for PDM are very recent. The central goal of the PDM project at Tennessee State University is to introduce engineering students to this method. The students participating in the project learn about PDM and the computer codes that are available to the design engineer. The software being used for this project is NESSUS (Numerical Evaluation of Stochastic Structures Under Stress) developed under NASA probabilistic structural analysis program. NESSUS has three different modules which make it a very comprehensive computer code for PDM. Since this method is new to the students, its introduction into the engineering curriculum is to be in stages. These range from the introduction of PDM and its software to the applications. While this program is being developed for its eventual inclusion into the engineering curriculum, some graduate and undergraduate students are already carrying out some projects using this method. As the students are increasing their understanding on PDM, they are at the same time applying it to some common design problems. The areas this method is being applied at the moment include, Design of Gears (spur and worm); Design of Brakes; Design of Heat Exchangers Design of Helical Springs; and Design of Shock Absorbers. Some of the current results of these projects are presented.
This paper considers response surface approximations for discontinuous quantities of interest. Our objective is not to adaptively characterize the manifold defining the discontinuity. Instead, we utilize an epistemic description of the uncertainty in the location of a discontinuity to produce robust bounds on sample-based estimates of probabilistic quantities of interest. We demonstrate that two common machine learning strategies for classification, one based on nearest neighbors (Voronoi cells) and one based on support vector machines, provide reasonable descriptions of the region where the discontinuity may reside. In higher dimensional spaces, we demonstrate that support vector machines are more accurate for discontinuities defined by smooth manifolds. We also show how gradient information, often available via adjoint-based approaches, can be used to define indicators to effectively detect a discontinuity and to decompose the samples into clusters using an unsupervised learning technique. Numerical results demonstrate the epistemic bounds on probabilistic quantities of interest for simplistic models and for a compressible fluid model with a shock-induced discontinuity.