Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “false negative analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Random forest models accurately classify synthetic opioids using high-dimensionality mass spectrometry datasets

Detection of novel threat agents presents several challenges, a principle one being the development of untargeted methods to screen an increasing number of threat chemicals whose exact structures are unknown. With the use of Machine Learning (ML) tools, we can guide the development of analytical methods for broad-spectrum detection of unbounded threat chemical families in complex mixtures. Toward this goal, we used nominal mass and high-resolution mass spectrometry data for hundreds of synthetic opioids and non-opioid compounds. We tested two ML techniques, logistic regression and random forest, to develop models towards a practical, implementable method for opioid detection. We found that of these tested ML methods, random forest models resulted in the highest validation accuracy (95+%) for both nominal mass and high-resolution classification of opioids versus non-opioids, with low false positive and false negative rates. The RF models were then used to successfully predict the classification of 10 compounds—five opioids and five non-opioids not part of the training and validation analysis. This application of ML is a critical step towards the development of field-deployable nominal mass spectrometers with ML-driven analyses for classification of emergent threats.

Chemistry↗

Random forest models accurately classify synthetic opioids using high-dimensionality mass spectrometry datasets

Detection of novel threat agents presents several challenges, a principle one being the development of untargeted methods to screen an increasing number of threat chemicals whose exact structures are unknown. With the use of Machine Learning (ML) tools, we can guide the development of analytical methods for broad-spectrum detection of unbounded threat chemical families in complex mixtures. Toward this goal, we used nominal mass and high-resolution mass spectrometry data for hundreds of synthetic opioids and non-opioid compounds. We tested two ML techniques, logistic regression and random forest, to develop models towards a practical, implementable method for opioid detection. We found that of these tested ML methods, random forest models resulted in the highest validation accuracy (95+%) for both nominal mass and high-resolution classification of opioids versus non-opioids, with low false positive and false negative rates. The RF models were then used to successfully predict the classification of 10 compounds—five opioids and five non-opioids not part of the training and validation analysis. This application of ML is a critical step towards the development of field-deployable nominal mass spectrometers with ML-driven analyses for classification of emergent threats.

Arasteh, Kourosh [Lawrence Livermore National Labo↗

The effect of abnormal cell proportion on specimen classifier performance

An analysis is presented of the results obtained from a cell classifier which is confronted with an abnormal/normal cell ratio which is different from the ratio assumed in the calibration of the classifier. False negative and false positive error rates are determined in advance for classifier operation, along with the necessary sample size in order to validate the predicted distributions. Changes are demonstrated to happen only regarding the false negative rate, where reductions in the abnormal cell rate below the expected rates would cause totally unreliable data. Substantial overproduction of abnormal cells would be quickly noticeable, while production rates beyond, but close to, the expected rates would only require more extensive sampling. Classifier systems for 10% proportions of abnormal cells are concluded to be possible, but difficulties are present with much lower rates

Castleman, K. R.↗

Application of Fault Management Theory to the Quantitative Selection of a Launch Vehicle Abort Trigger Suite

The theory of System Health Management (SHM) and of its operational subset Fault Management (FM) states that FM is implemented as a "meta" control loop, known as an FM Control Loop (FMCL). The FMCL detects that all or part of a system is now failed, or in the future will fail (that is, cannot be controlled within acceptable limits to achieve its objectives), and takes a control action (a response) to return the system to a controllable state. In terms of control theory, the effectiveness of each FMCL is estimated based on its ability to correctly estimate the system state, and on the speed of its response to the current or impending failure effects. This paper describes how this theory has been successfully applied on the National Aeronautics and Space Administration's (NASA) Space Launch System (SLS) Program to quantitatively estimate the effectiveness of proposed abort triggers so as to select the most effective suite to protect the astronauts from catastrophic failure of the SLS. The premise behind this process is to be able to quantitatively provide the value versus risk trade‐off for any given abort trigger, allowing decision makers to make more informed decisions. All current and planned crewed launch vehicles have some form of vehicle health management system integrated with an emergency launch abort system to ensure crew safety. While the design can vary, the underlying principle is the same: detect imminent catastrophic vehicle failure, initiate launch abort, and extract the crew to safety. Abort triggers are the detection mechanisms that identify that a catastrophic launch vehicle failure is occurring or is imminent and cause the initiation of a notification to the crew vehicle that the escape system must be activated. While ensuring that the abort triggers provide this function, designers must also ensure that the abort triggers do not signal that a catastrophic failure is imminent when in fact the launch vehicle can successfully achieve orbit. That is, the abort triggers must have low false negative rates to be sure that real crew‐threatening failures are detected, and also low false positive rates to ensure that the crew does not abort from non‐crew‐threatening launch vehicle behaviors. The analysis process described in this paper is a compilation of over six years of lessons learned and refinements from experiences developing abort triggers for NASA's Constellation Program (Ares I Project) and the SLS Program, as well as the simultaneous development of SHM/FM theory. The paper will describe the abort analysis concepts and process, developed in conjunction with SLS Safety and Mission Assurance (S&MA) to define a common set of mission phase, failure scenario, and Loss of Mission Environment (LOME) combinations upon which the SLS Loss of Mission (LOM) Probabilistic Risk Assessment (PRA) models are built. This abort analysis also requires strong coordination with the Multi‐Purpose Crew Vehicle (MPCV) and SLS Structures and Environments (STE) to formulate a series of abortability tables that encapsulate explosion dynamics over the ascent mission phase. The design and assessment of abort conditions and triggers to estimate their Loss of Crew (LOC) Benefits also requires in‐depth integration with other groups, including Avionics, Guidance, Navigation and Control(GN&C), the Crew Office, Mission Operations, and Ground Systems. The outputs of this analysis are a critical input to SLS S&MA's LOC PRA models. The process described here may well be the first full quantitative application of SHM/FM theory to the selection of a sensor suite for any aerospace system.

Lo, Yunnhon↗

Evaluation of Minimum Detectable Activities for Stack Sample Analyses

The Rad-NESHAP program takes samples at major point sources to track how much of select nuclides are being emitted out of the stack. Samples are taken at the site where the radiological operations are being performed. The samples are then sent to analytical laboratories for analysis to determine what nuclides exist in the sample. However, due to the sensitivity of the instruments, false positives are often reported. To reduce the number of false positives, the results reported from the analytical labs are compared to set detection limits. Anything below the set detection limit is considered part of the background while anything above the limit is considered a positive hit. Therefore, the detection limit must be set low enough that any false negatives do not result in a significant dose missed at the end of the year but high enough that there are not enough false positives to skew the annual dose being reported. All detection limits listed in this section for the analyses below were designed to meet the stated equivalent emissions rate and subsequent off-site doses. These minimum detectable activities (MDAs) were developed in the 1990s and documented in the memo cited for each analysis below. These limits and memos are all currently referenced in the Quality Assurance Project Plan (QAPP) for the Rad-NESHAP Compliance Task, which is being updated to a Project Implementation Plan (PIP) concurrently with this document. As part of this update process, the original MDAs defined in this section needed to be verified that they were still conservative and meet the off-site dose levels stated.

2019 MDA↗

Element Load Data Processor (ELDAP) Users Manual

Often, the shear and tensile forces and moments are extracted from finite element analyses to be used in off-line calculations for evaluating the integrity of structural connections involving bolts, rivets, and welds. Usually the maximum forces and moments are desired for use in the calculations. In situations where there are numerous structural connections of interest for numerous load cases, the effort in finding the true maximum force and/or moment combinations among all fasteners and welds and load cases becomes difficult. The Element Load Data Processor (ELDAP) software described herein makes this effort manageable. This software eliminates the possibility of overlooking the worst-case forces and moments that could result in erroneous positive margins of safety and/or selecting inconsistent combinations of forces and moments resulting in false negative margins of safety. In addition to forces and moments, any scalar quantity output in a PATRAN report file may be evaluated with this software. This software was originally written to fill an urgent need during the structural analysis of the Ares I-X Interstage segment. As such, this software was coded in a straightforward manner with no effort made to optimize or minimize code or to develop a graphical user interface.

Ramsey, John K., Jr.↗

Planning, design and logistics of a decision analysis study: The FBI/Ames study involving forensic firearms examiners

This paper describes design and logistical aspects of a decision analysis study to assess the performance of qualified firearms examiners working in accredited laboratories in the United States in terms of accuracy (error rate), repeatability, and reproducibility of decisions involving comparisons of fired bullets and cartridge cases. The purpose of the study was to validate current practice of the forensic discipline of firearms/toolmarks (F/T) examination. It elicited error rate data by counting the number of false positive and false negative conclusions. Preceded by the experimental design, decisions, and logistics described herein, testing was ultimately administered 173 qualified, practicing F/T examiners in public and private crime laboratories. The first round of testing evaluated accuracy, while two subsequent rounds evaluated repeatability and reproducibility of examiner conclusions. This project expands on previous studies by involving many F/T examiners in challenging comparisons and by executing the study in the recommended double-blind format.

99 GENERAL AND MISCELLANEOUS↗

A robust benchmark for detection of germline large deletions and insertions

New technologies and analysis methods are enabling genomic structural variants (SVs) to be detected with ever-increasing accuracy, resolution and comprehensiveness. To help translate these methods to routine research and clinical practice, we developed a sequence-resolved benchmark set for identification of both false-negative and false-positive germline large insertions and deletions. To create this benchmark for a broadly consented son in a Personal Genome Project trio with broadly available cells and DNA, the Genome in a Bottle Consortium integrated 19 sequence-resolved variant calling methods from diverse technologies. The final benchmark set contains 12,745 isolated, sequence-resolved insertion (7,281) and deletion (5,464) calls ≥50 base pairs (bp). The Tier 1 benchmark regions, for which any extra calls are putative false positives, cover 2.51 Gbp and 5,262 insertions and 4,095 deletions supported by ≥1 diploid assembly. We demonstrate that the benchmark set reliably identifies false negatives and false positives in high-quality SV callsets from short-, linked- and long-read sequencing and optical mapping.

59 BASIC BIOLOGICAL SCIENCES↗

Reducing False Positives in Runtime Analysis of Deadlocks

This paper presents an improvement of a standard algorithm for detecting dead-lock potentials in multi-threaded programs, in that it reduces the number of false positives. The standard algorithm works as follows. The multi-threaded program under observation is executed, while lock and unlock events are observed. A graph of locks is built, with edges between locks symbolizing locking orders. Any cycle in the graph signifies a potential for a deadlock. The typical standard example is the group of dining philosophers sharing forks. The algorithm is interesting because it can catch deadlock potentials even though no deadlocks occur in the examined trace, and at the same time it scales very well in contrast t o more formal approaches to deadlock detection. The algorithm, however, can yield false positives (as well as false negatives). The extension of the algorithm described in this paper reduces the amount of false positives for three particular cases: when a gate lock protects a cycle, when a single thread introduces a cycle, and when the code segments in different threads that cause the cycle can actually not execute in parallel. The paper formalizes a theory for dynamic deadlock detection and compares it to model checking and static analysis techniques. It furthermore describes an implementation for analyzing Java programs and its application to two case studies: a planetary rover and a space craft altitude control system.

Bensalem, Saddek↗

A Metabolomics Assay to Diagnose Citrus Huanglongbing Disease and to Aid in Assessment of Treatments to Prevent or Cure Infection

Citrus greening disease, or Huanglongbing (HLB), has devastated citrus crops globally in recent years. The causal bacterium, ‘ Candidatus Liberibacter asiaticus’, presents a sampling issue for qPCR diagnostics and results in a high false negative rate. In this work, we compared six metabolomics assays to identify HLB-infected citrus trees from leaf tissue extracted from 30 control and 30 HLB-infected trees. A liquid chromatography-mass spectrometry-based assay was most accurate. A final partial least squares-discriminant analysis (PLS-DA) model was trained and validated on 690 leaf samples with corresponding qPCR measures from three citrus varieties (Rio Red grapefruit, Hamlin sweet orange, and Valencia sweet orange) from orchards in Florida and Texas. Trees were naturally infected with HLB transmitted by the insect vector Diaphorina citri. In a randomized validation set, the assay was 99.9% accurate to classify diseased from nondiseased samples. This model was applied to samples from trees receiving plant defense-inducer compounds or biological treatments to prevent or cure HLB infection. From two trials, HLB-related metabolite abundances and PLS-DA scores were tracked longitudinally and compared with those of control trees. We demonstrate how our assay can assess tree health and the efficacy of HLB treatments and conclude that no trialed treatment was efficacious.

Plant Sciences↗

Real-Time Detection of Dust Devils from Pressure Readings

A method for real-time detection of dust devils at a given location is based on identifying the abrupt, temporary decreases in atmospheric pressure that are characteristic of dust devils as they travel through that location. The method was conceived for use in a study of dust devils on the Martian surface, where bandwidth limitations encourage the transmission of only those blocks of data that are most likely to contain information about features of interest, such as dust devils. The method, which is a form of intelligent data compression, could readily be adapted to use for the same purpose in scientific investigation of dust devils on Earth. In this method, the readings of an atmospheric- pressure sensor are repeatedly digitized, recorded, and processed by an algorithm that looks for extreme deviations from a continually updated model of the current pressure environment. The question in formulating the algorithm is how to model current normal observations and what minimum magnitude deviation can be considered sufficiently anomalous as to indicate the presence of a dust devil. There is no single, simple answer to this question: any answer necessarily entails a compromise between false detections and misses. For the original Mars application, the answer was sought through analysis of sliding time windows of digitized pressure readings. Windows of 5-, 10-, and 15-minute durations were considered. The windows were advanced in increments of 30 seconds. Increments of other sizes can also be used, but computational cost increases as the increment decreases and analysis is performed more frequently. Pressure models were defined using a polynomial fit to the data within the windows. For example, the figure depicts pressure readings from a 10-minute window wherein the model was defined by a third-degree polynomial fit to the readings and dust devils were identified as negative deviations larger than both 3 standard deviations (from the mean) and 0.05 mbar in magnitude. An algorithm embodying the detection scheme of this example was found to yield a miss rate of just 8 percent and a false-detection rate of 57 percent when evaluated on historical pressure-sensor data collected by the Mars Pathfinder lander. Since dust devils occur infrequently over the course of a mission, prioritizing observations that contain successful detections could greatly conserve bandwidth allocated to a given mission. This technique can be used on future Mars landers and rovers, such as Mars Phoenix and the Mars Science Laboratory.

Wagstaff, Kiri↗

Multi-Sensor Observations of Earthquake Related Atmospheric Signals over Major Geohazard Validation Sites

We are conducting a scientific validation study involving multi-sensor observations in our investigation of phenomena preceding major earthquakes. Our approach is based on a systematic analysis of several atmospheric and environmental parameters, which we found, are associated with the earthquakes, namely: thermal infrared radiation, outgoing long-wavelength radiation, ionospheric electron density, and atmospheric temperature and humidity. For first time we applied this approach to selected GEOSS sites prone to earthquakes or volcanoes. This provides a new opportunity to cross validate our results with the dense networks of in-situ and space measurements. We investigated two different seismic aspects, first the sites with recent large earthquakes, viz.- Tohoku-oki (M9, 2011, Japan) and Emilia region (M5.9, 2012,N. Italy). Our retrospective analysis of satellite data has shown the presence of anomalies in the atmosphere. Second, we did a retrospective analysis to check the re-occurrence of similar anomalous behavior in atmosphere/ionosphere over three regions with distinct geological settings and high seismicity: Taiwan, Japan and Kamchatka, which include 40 major earthquakes (M>5.9) for the period of 2005-2009. We found anomalous behavior before all of these events with no false negatives; false positives were less then 10%. Our initial results suggest that multi-instrument space-borne and ground observations show a systematic appearance of atmospheric anomalies near the epicentral area that could be explained by a coupling between the observed physical parameters and earthquake preparation processes.

Ouzounov, D.↗

A novel experimental system for the KDK measurement of the 40 K decay scheme relevant for rare event searches

Potassium-40 ( 40 K) is a long-lived, naturally occurring radioactive isotope. The decay products are prominent backgrounds for many rare event searches, including those involving NaI-based scintillators. 40 K also plays a role in geochronological dating techniques. The branching ratio of the electron capture directly to the ground state of argon-40 has never been measured, which can cause difficulty in interpretingcertain results or can lead to lack of precision depending on the field and analysis technique. Additionally, the KDK (Potassium (K) Decay (DK)) collaboration is measuring this decay. A composite method has a silicon drift detector with an enriched, thermally deposited 40 K source inside the Modular Total Absorption Spectrometer. This setup has been characterized in terms of energy calibration, gamma tagging efficiency, live time and false negatives and positives. A complementary, homogeneous, method is also discussed; it employs a KSr 2 I 5 :Eu scintillator as source and detector.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Study of LANDSAT-D thematic mapper performance as applied to hydrocarbon exploration

Analysis of the tapes of the Detroit, Michigan scene, which were received in fully processed format with geometric and radiometric correction, shows evidence of an along line data slip every sixteenth line in TM channel 2. Very large scale products were therefore generated in false color using channels 1, 3, and 4. Subjective evaluation of these enhanced scenes indicates that they are acceptable for interpretation at scales up to 1:50,000 and should be useful for change mapping probably up to 1:24,000 scale. The significant striping visible in water bodies for both the natural color and false color products indicates that the detector calibration is probably performing below the preflight specification. Variance-covariance matrices were computed and principal component analysis were performed for a set of 512 x 512 windows within the Arkansas scene. Initial analysis shows the shortwave infrared channels (TM 5 and 6) are a highly significant data source. The thermal channel (TM 7) shows negative correlation with TM 1 through 4.

Source record↗

Data-driven modeling of dynamic occupant thermostat override behavior for demand response applications

Buildings consume nearly 40% of global energy and produce similar emissions. Whiletechnological advances address efficiency, occupant behavior causes energy use variations up to 300% between identical buildings. This gap between predicted and actual building performance impacts building design, operations, and grid demand management programs. Through analyses of smart thermostat data from 1,400 single-occupant homes, the researchdemonstrates that occupants respond to 8°F thermostat setpoint changes within a median of 15 minutes, while 2°F changes trigger responses within a median of 30 minutes. This highlights an understudied temporal relationship between thermostat setbacks and response time of occupant behaviors. Models of such behavior dynamics are required to incorporate occupant impacts into building performance simulation. A key contribution of this dissertation is the Thermal Frustration Theory (TFT), which positsthat thermal discomfort driven behaviors are caused by the time-accumulation of discomfort, not simply a temperature deviation threshold or a delay from an initiating event. Using a dataset of 634 thermostats, each with 25+ manual setpoint changes, a comparative analysis of TFT and comfort zone and a delayed response theories demonstrated that personalized TFT models better predict when manual setpoint change occur. This was measured by the area under the curve statistical measure (AUC); all three models perform similarly by a Matthews Correlation Coefficient measure. Higher AUC performance is especially important for modeling occupant behavior in demand response programs where false negatives of rare occupant interactions could adversely affect grid stability. EnergyPlus based simulations were conducted with TFT-derived occupant models, demonstrating the ability to identify parameters of known TFT models from only data observable with smart thermostats, even under the presence of noise from routine overrides. Overall, the dissertation highlights that thermostat interactions are neither static,instantaneous, nor driven solely by the environment. Instead, temporal accumulation of discomfort and routine-based behavior play important roles. The methodology and results offer a pathway towards more accurate modeling of human-building interactions for policy assessment, building design, and demand response programs.

Sharma, Kunind [Northeastern University] (ORCID:00↗

Deep Learning for Automated Identification of Eels in Sonar Data

Freshwater eels, such as the American eel (Anguilla rostrata) present numerous challenges related to safe downstream fish passage at hydroelectric facilities. One of those challenges is effective monitoring of their abundance, movements, and behavior to facilitate design and operation of eel protection and passage facilities. A previous EPRI study documented the ability of human analysts to reliably identify American eels in data obtained with a 1100/1800 kHz, multibeam sonar. This report describes a project to develop deep learning (a subset of artificial intelligence) tools to automate the time-consuming, subjective process of eel identification in multibeam sonar data. The project exploited new data collected in the laboratory and the existing data from the prior EPRI field study to develop and test deep learning and other data analytic tools, including wavelet filtering, differencing for static object removal, and convolutional neural network analysis. The analysis of the laboratory data demonstrated feasibility of the approach, revealed object characteristics observed with the sonar that distinguish eels from similarly sized and shaped acoustic targets, and provided additional data for algorithm selection and training. Deep learning algorithms trained and tested on the laboratory data alone achieved accuracy rates of greater than 98% when classifying acoustic images of eels and similar-sized neutrally buoyant sticks. The algorithm trained and tested on the pre-existing field data alone, and yielded classification accuracy of 9.3% false positives and 13.3% false negatives when distinguishing between eels and sticks/PVC pipes based on video clips (i.e., multiple, consecutive images). This performance is comparable to the classification accuracy achieved by human analysts in the prior study. The deep learning algorithm trained on a combination of video clips obtained in the laboratory and the field and tested on video clips from the field, was able to distinguish eels from sticks and PVC pipes (a river debris analog) of similar size with 100% accuracy. Outreach to the hardware, software, and end-user communities early in the project helped to identify needs and specify the application space. Outreach to those communities at the end of the project communicated project results and opportunities for further development. The project achieved proof of concept for automated identification of eel in multibeam sonar data. Future work should focus on acquisition of additional data for more robust algorithm training and testing; modification of the software tools to accommodate multiple acoustic targets in the acoustic field at a given time; identification of additional object classes; incorporation of motion in the object identification and classification algorithms; operationalizing the software tools, including integration with other existing sonar data analysis tools; and partnering with hardware and software providers for distribution of the software tools with their commercial products.

13 HYDRO ENERGY↗

Jensen–Shannon divergence based novel loss functions for Bayesian neural networks

Bayesian neural networks (BNNs) are state-of-the-art machine learning methods that can naturally regularize and systematically quantify uncertainties using their stochastic parameters. Kullback–Leibler (KL) divergence-based variational inference used in BNNs suffer from unstable optimization and challenges in approximating light-tailed posteriors due to the unbounded nature of the KL divergence. To resolve these issues, we formulate a novel loss function for BNNs based on a new modification to the generalized Jensen–Shannon (JS) divergence, which is bounded. In addition, we propose a Geometric JS divergence-based loss, which is computationally efficient since it can be evaluated analytically. We found that the JS divergence-based variational inference is intractable, and hence employed a constrained optimization framework to formulate these losses. Our theoretical analysis and empirical experiments on multiple regression and classification data sets suggest that the proposed losses perform better than the KL divergence-based loss, especially when the data sets are noisy or biased. Specifically, there are approximately 5% and 8% improvements in accuracy for a noise-added CIFAR-10 dataset and a regression dataset, respectively. There is about 13% reduction in false negative predictions of a biased histopathology dataset. Additionally, we quantify and compare the uncertainty metrics for the regression and classification tasks.

97 MATHEMATICS AND COMPUTING↗

Harnessing Collaborative Learning Automata to Guide Multi-objective Optimization based Inverse Analysis for Structural Damage Identification

Structural damage identification based on physical models is often transformed into an optimization problem that minimizes the difference between measurement information of structure being monitored and the model prediction in the parametric space. However, the objective function in this context often exhibits multimodality, involving high-dimensional variables due to the reliance on finite element models for damage identification. These features pose challenges to optimization algorithms, where entrapment in local solutions can lead to false positives and false negatives in damage identification. In this research, we propose a reinforcement learning based multi-swarm optimizer to tackle such challenges in pursuit of a small yet diverse solution set that can capture the true damage scenario as one of the solutions. The proposed method leverages the flexibility of the particle swarm optimizer and incorporates novel strategies of metaheuristics to realize targeted improvement. To enable the particle swarm to adaptively select the appropriate search strategy based on the current environment, we adopt the learning automata technique, which sidesteps the need for reward strategy selection that is usually ad hoc at each step of the search. The integration harnesses the automatic learning and self-adaptation capabilities of learning automata, enabling the particles to navigate based on environmental signals. This leads to accumulated probabilities tied to advantageous movements, fostering an adaptive exploration of particles in the search space. The proposed approach is first validated through implementing into benchmark test cases with comparisons. It is then applied to structural damage identification with piezoelectric admittance experimental signals. `The results highlight the capability of the algorithm to identify a small solution set with high accuracy to match the actual damage scenario.

Yang Zhang↗