Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “adversarial evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7
All sourcesNASA NTRSDOE OSTI
NASA NTRS1995

Integrated helmet mounted display concepts for air combat

A piloted simulation study was conducted in a dome simulator to evaluate several Helmet Mounted Display (HMD) formats developed as part of the NASA High Alpha Technology Program (HATP). The display formats conveyed energy management, spatial orientation, and weapons management information. The HMD format was compared to a generic Heads Up Display (HUD) typical of current operational fighter aircraft. Pilots were tasked to spend as much time in a weapon solution as possible, to have the correct weapon selected for the envelope they were in, and to avoid the adversary's weapon envelope as much as possible. Several different displays were tested individually and simultaneously to see how separate display concepts coexisted. Objective results showed that the ability for the pilot to select the correct weapon for the envelope he was in increased by 50% in a moderate workload condition and 90% in a high workload condition with the HMD format. In the post-test comments pilots generally favored the helmet display formats over the HUD formats with a few instances where pilots preferred a simple numeric readout of the parameter. Short term exposure effects of the HMD on visual acuity were also measured and showed no advers results.

Clark, Joseph W.↗
NASA NTRSDate not supplied

QuantifyML: How good is my machine learning model?

This paper presents an approach, QuantifyML, which employs model counting to assess the learnability and robustness of machine learning models. Typically the efficacy of machine learning models is determined by computing their accuracy statistically on test data sets. However, this may be misleading, if the test data is not representative of the problem that is being studied. Further, two different models may have the same accuracy on a given data set, measured statistically, but may be very different in their behavior on unseen data. Also, models with high accuracy could have poor adversarial robustness. In QuantifyML, our goal is to precisely quantify the extent to which machine learning models have learned and generalized from the given data. In QuantifyML, a trained model is translated into a C program, which is fed to the CBMC model checking tool to produce a formula in Conjunctive Normal Form (CNF), which in turn is analyzed with state-of-the-art model counters to efficiently obtain precise counts w.r.t different outputs. QuantifyML enables i) evaluating the learnability of models by comparing the counts for the outputs to ground truth, expressed as logical predicates (if available), ii) comparing the performance of different models that may be built with different machine learning algorithms (e.g., decision-trees vs. neural networks), and iii) quantifying the robustness of trained models around given inputs. Our evaluation demonstrates these applications of QuantifyML on decision trees and neural networks trained to learn relational properties of graphs, for which we know the ground truth, and to perform image classification, for which we do not have the ground truth, but we can quantify local robustness.

Deep Neural Networks↗
DOE OSTI2020

Image-driven discriminative and generative machine learning algorithms for establishing microstructure–processing relationships

We investigate methods of microstructure representation for the purpose of predicting processing condition from microstructure image data. A binary alloy that is currently under development as a nuclear fuel was studied for the purpose of developing an improved machine learning approach to image recognition, characterization, and building predictive capabilities linking microstructure to processing conditions. Here, we test different microstructure representations and evaluate model performance based on classification accuracy. A classification accuracy of 95.8% was achieved fordistinguishing between micrographs corresponding to ten different thermo-mechanical material processing conditions.We find that our newly developed microstructure representation describes image data well, and the traditional approachof utilizing area fractions of different phases is insufficient for distinguishing between multiple classes using a relativelysmall, imbalanced original data set of 272 images. To explore the applicability of generative methods for supplementing such limited data sets, generative adversarial networks were trained to generate artificial microstructure images. Two different generative networks were trained and tested to assess performance. Challenges and best practices associated with applying machine learning to limited microstructure image data sets is also discussed. Our work has implications for quantitative microstructure analysis, and development of microstructure-processing relationships in limited data sets typical of metallurgical process design studies.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Inferring adversarial behaviour in cyber‐physical power systems using a Bayesian attack graph approach

Abstract Highly connected smart power systems are subject to increasing vulnerabilities and adversarial threats. Defenders need to proactively identify and defend new high‐risk access paths of cyber intruders that target grid resilience. However, cyber‐physical risk analysis and defense in power systems often requires making assumptions on adversary behaviour, and these assumptions can be wrong. Thus, this work examines the problem of inferring adversary behaviour in power systems to improve risk‐based defense and detection. To achieve this, a Bayesian approach for inference of the Cyber‐Adversarial Power System (Bayes‐CAPS) is proposed that uses Bayesian networks (BNs) to define and solve the inference problem of adversarial movement in the grid infrastructure towards targets of physical impact. Specifically, BNs are used to compute conditional probabilities to queries, such as the probability of observing an event given a set of alerts. Bayes‐CAPS builds initial Bayesian attack graphs for realistic power system cyber‐physical models. These models are adaptable using collected data from the system under study. Then, Bayes‐CAPS computes the posterior probabilities of the occurrence of a security breach event in power systems. Experiments are conducted that evaluate algorithms based on time complexity, accuracy and impact of evidence for different scales and densities of network. The performance is evaluated and compared for five realistic cyber‐physical power system models of increasing size and complexities ranging from 8 to 300 substations based on computation and accuracy impacts.

Sahu, Abhijeet↗
DOE OSTI2020

Semisupervised Learning for Seismic Monitoring Applications

The impressive performance that deep neural networks demonstrate on a range of seismic monitoring tasks depends largely on the availability of event catalogs that have been manually curated over many years or decades. However, the quality, duration, and availability of seismic event catalogs vary significantly across the range of monitoring operations, regions, and objectives. Semisupervised learning (SSL) enables learning from both labeled and unlabeled data and provides a framework to leverage the abundance of unreviewed seismic data for training deep neural networks on a variety of target tasks. We apply two SSL algorithms (mean-teacher and virtual adversarial training) as well as a novel hybrid technique (exponential average adversarial training) to seismic event classification to examine how unlabeled data with SSL can enhance model performance. In general, we find that SSL can perform as well as supervised learning with fewer labels. We also observe in some scenarios that almost half of the benefits of SSL are the result of the meaningful regularization enforced through SSL techniques and may not be attributable to unlabeled data directly. Lastly, the benefits from unlabeled data scale with the difficulty of the predictive task when we evaluate the use of unlabeled data to characterize sources in new geographic regions. Finally, in geographic areas where supervised model performance is low, SSL significantly increases the accuracy of source-type classification using unlabeled data.

58 GEOSCIENCES↗
DOE OSTI2024

The Digital Assurance for High Consequence Systems (DAHCS) Mission Campaign Whitepaper

The DAHCS (pronounced “Dax”) MC is a 7-year, $\$$45 million research portfolio within Sandia’s Laboratory Directed Research and Development program. The DAHCS MC arose in response to a great need: to ensure that the use of digital technologies does not weaken our nation’s high consequence systems. Digital technologies offer many benefits in speed, cost, and flexibility, and we seek to reap those benefits without introducing new system failures. However, digital technologies cannot be evaluated the same way as analog technologies. Initiatives across the nation highlight the capability gap that prevents efficient, effective digital assurance. The Challenge Today’s digital assurance tools, techniques, and methods are inadequate to confidently characterize, assess, and manage digital risk; they are ad hoc, slow, costly, and rarely scalable to increasingly complex digital technologies. The rapidly evolving cyber threat landscape exacerbates this problem because digital assurance now must secure against digital risks now and in the future, including those introduced by rapidly evolving technologies, adversaries, and systems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Application of Cyber-Informed Engineering for Protecting BESS

This white paper synthesizes an array of crucial grid services provided by BESS technology, assesses its architecture and communications, and presents a case study for analysis against the principles introduced by Cyber-Informed Engineering (CIE). Furthermore, in walking through the analysis, this paper presents a framework to evaluate risks and solutions when considering BESS components. Asset owners and buyers could perform this analysis to assess their BESS product implementations, alternative inverter-based resources (IBR), and energy management systems (EMS). Battery systems fulfill various roles contingent on the unique market demands and the specific challenges presented by regional grid infrastructures. These roles also vary due to the differing utility models for ownership and operation, which are adapted to meet regional and local capabilities and requirements. Concerns have been raised regarding the potential for adversaries to exploit knowledge of battery operational patterns to orchestrate decisive attacks. However, the security of operational data for these systems may not be the primary vulnerability, as much of this information is already well-understood within the community. Applying a modest degree of subject matter expertise can often yield valuable predictions regarding how a battery will respond under certain conditions, such as grid emergencies, high or low-temperature days, Public Safety Power Shutoff (PSPS) events, and outages. The operational characteristics of batteries are well-documented, and their capabilities, including the risks associated with misoperation and the resulting consequences, are published and understood within the industry. CIE practices represent the next step in gaining functional assurance and providing an acceptable level of risk, regardless of whether a battery vendor can support a trusted and validated supply chain. While this issue has exacerbated supply chain challenges, it is not an isolated condition. This foreign supply route is the primary source of BESS for the U.S. market. Significant efforts are underway through the Bipartisan Infrastructure Law (BIL) to change that. Still, strategic short-term operational mitigations are needed to ensure the security of our operational technology (OT) systems, which are enhanced by instilling trust and are separate from vendors implementing CIE principles.

25 ENERGY STORAGE↗
DOE OSTI2022

Revisiting Current Paradigms: Subject Matter Expert Views on High Consequence Facility Security Assessments

Security assessments support decision-makers' ability to evaluate current capabilities of high consequence facilities (HCF) to respond to possible attacks. However, increasing complexity of today's operational environment requires a critical review of traditional approaches to ensure that implemented assessments are providing relevant and timely insights into security of HCFs. Using interviews and focus groups with diverse subject matter experts (SMEs), this study evaluated the current state of security assessments and identified opportunities to achieve a more "ideal" state. The SME-based data underscored the value of a systems approach for understanding the impacts of changing operational designs and contexts (as well as cultural influences) on security to address methodological shortcomings of traditional assessment processes. These findings can be used to inform the development of new approaches to HCF security assessments that are able to more accurately reflect changing operational environments and effectively mitigate concerns arising from new adversary capabilities.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Advanced Reactor Safeguards & Security Program: Cybersecurity Scenarios

The use of digital control systems and automation in advanced nuclear power systems introduces different types of vulnerabilities compared to legacy (i.e. analog) control systems that cyber adversaries can exploit. These vulnerabilities pose a challenge to reactor operators and cyber operations staff due to the dynamic nature of the event in which a human response or a lack of response can potentially evolve into a worsening plant condition. Using the Department of Homeland Security Cyber and Infrastructure Security Agency’s (CISA) critical infrastructure exercise framework, this document presents several cyber security scenarios typical of digital control systems that could be used in advanced reactor designs. These scenarios can be used in tabletop exercises to evaluate cyber security posture or conduct training on different aspects of cyber security, including detection, threat hunting using indicators of compromise, evaluating incident response, risk mitigation, incident reporting, information sharing and recovery.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2020

Assessing DER Network Cybersecurity Defences in a Power-Communication Co-Simulation Environment

Increasing penetrations of interoperable distributed energy resources (DER) in the electric power system are expanding the power system attack surface. Maloperation or malicious control of DER equipment can now cause substantial disturbances to grid operations. Fortunately, many options exist to defend and limit adversary impact on these newly-created DER communication networks, which typically traverse the public internet. However, implementing these security features will increase communication latency, thereby adversely impacting real-time DER grid support service effectiveness. In this work, a collection of software tools called SCEPTRE were used to create a co-simulation environment where SunSpec-compliant PV inverters were deployed as virtual machines and interconnected to simulated communication network equipment. Network segmentation, encryption, and moving target defence security features were deployed on the control network to evaluate their influence on cybersecurity metrics and power system performance. The results indicated that adding these security features did not impact DER-based grid control systems but improved the cybersecurity posture of the network when implemented appropriately.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1991

Real-time simulation of helicopter air-to-air combat

The AUTOMAN computer program develops automated maneuvering decisions for helicopters during air-to-air combat over hilly terrain. Recently, the capabilities of this program have been extended and enhanced significantly. The revised program was installed at the NASA Ames manned flight-simulation facility to drive a computer-generated image of an enemy helicopter, thereby providing an adversary for the human pilot. Maneuvers are selected by employing game theory. Enhancements include a guidance law for target acquisition when a firing opportunity arises; fire-control sequence logic; improved low-flying capabilities; line-of-sight computations for the cockpit field-of-view, terrain obstructions, and visual range limits; use of terrain for masking; air-to-air collision-avoidance maneuvers; decision on dispensing flares and chaff; and adjustable levels of pilot experience. The program was found to be extremely useful for both rotorcraft handling-quality evaluations and air-to-air combat training.

Austin, Fred↗
DOE OSTI2021

Comprehension of Spatial Constraints by Neural Logic Learning from a Single RGB-D Scan

Autonomous industrial assembly relies on the precise measurement of spatial constraints as designed by computer-aided design (CAD) software such as SolidWorks. This paper proposes a framework for an intelligent industrial robot to understand the spatial constraints for model assembly. An extended generative adversary network (GAN) with a 3D long short-term memory (LSTM) network was designed to composite 3D point clouds from a single RGB-D scan. The spatial constraints of the segmented point clouds are identified by a neural-logic network that incorporates general knowledge of spatial constraints in terms of first-order logic. The model was designed to comprehend a complete set of spatial constraints that are consistent with industrial CAD software, including left, right, above, below, front, behind, parallel, perpendicular, concentric, and coincident relations. The accuracy of 3D model composition and spatial constraint identification was evaluated by the RGB-D scans and 3D models in the ABC dataset. The proposed model achieved 57.23% intersection over union (IoU) in 3D model composition, and over 99% in comprehending all spatial constraints.

Wang, Dali↗
DOE OSTI2022

Evaluation of Joint Cyber/Safety Risk in Nuclear Power Systems

This report presents an analysis of the Emergency Core Cooling System (ECCS) for a generic Boiling Water Reactor (BWR)-4 NPP. The Electric Power Research Institute (EPRI) developed Hazards and Consequences Analysis for Digital Systems (HAZCADS) process is applied to the ECCS and its subsystems to identify unsafe control actions (UCAs) which act as possible cyber events of concern. The analysis is performed for two design basis events: Small-break Loss of Coolant Accident (SLOCA) and general transients (TRANS), such as unintended reactor trip. In previous work, HAZCADS UCAs were combined with other cyber-attack analysis to develop a risk-informed approach; however, this was for a single system. This report explores advanced systems engineering modeling approaches to model the interactions between digital assets across multiple systems which may be targeted by cyber adversaries. The complex and interdependent design of digital systems has the potential to introduce emergent cyber properties that are generally not covered by hazard analyses nor formal nuclear Probabilistic Risk Assessment (PRA). The R&D and supporting analysis presented here explores approaches to predict and manage how interdependent system properties effect risk. To show the potential impact of a successful cyber-attack to formal PRA event tree probabilities, HAZCADS analysis was also used. HAZCADS was also used to model the automatic depressurization system (ADS) automatic actuation. This analysis extended to an integrated system analysis for common-cause failure (CCF). In this aspect, the HAZCADS analysis continued by analyzing plant design details for system connectivity in support of critical plant functions. A dependency matrix was developed to depict the integrated functionality of the interconnected systems. Areas of potential CCF are indicated. Future work could include adversary attack development to show how CCF could be caused, resulting in PRA events. Across the multiple systems that comprise the ECCS, the analysis shows that the change in such probabilities was very different between systems. This indicates that some systems have a larger potential risk impact from successful cyber-attack or digital failure, which indicates a need for these systems to have a higher priority for design and defensive measures. Furthermore, we were able to establish that a risk analysis using any arbitrary threat model establishes an ordering of components with regard to cyber-risk. This ordering can be used to influence the overall system design with an eye to lowering risk, or as a way to understand real-time risk to operational systems based on a current threat landscape. Expert knowledge of both the analysis process and the system being analyzed is required to perform a HAZCADS analysis. The need for a tiered risk analysis is demonstrated by the results of this report.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2021

ForSE: A GAN-based Algorithm for Extending CMB Foreground Models to Subdegree Angular Scales

We present ForSE (Foreground Scale Extender), a novel Python package that aims to overcome the current limitations in the simulation of diffuse Galactic radiation, in the context of cosmic microwave background (CMB) experiments. ForSE exploits the ability of generative adversarial neural networks (GANs) to learn and reproduce complex features present in a set of images, with the goal of simulating realistic and non-Gaussian foreground radiation at subdegree angular scales. This is of great importance in order to estimate the foreground contamination to lensing reconstruction, delensing, and primordial B-modes for future CMB experiments. We applied this algorithm to Galactic thermal dust emission in both total intensity and polarization. Our results show how ForSE is able to generate small-scale features (at 12') having as input the large-scale ones (80'). The injected structures have statistical properties, evaluated by means of the Minkowski functionals, in good agreement with those of the real sky and which show the correct amplitude scaling as a function of the angular dimension. Furthermore, the obtained thermal dust Stokes Q and U full-sky maps as well as the ForSE package are publicly available for download.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2021

Adoption of image-driven machine learning for microstructure characterization and materials design: A Perspective

Microstructure characterization enables the development of structure-processing-property relationships critical to several research areas within the broad field of materials science, from alloy design to the assessment of corrosion resistance, and failure analysis. Conventional approaches to material characterization have relied on either qualitative inference by the human ex-pert or software applications that can extract high-level features from images, such as boundary segmentation, average grain diameter, etc. Such approaches rely heavily on subject matter expert user intervention and knowledge of what phases or more generally, what microstructural features, are of interest. The recent surge in the adoption of machine learning techniques to address problems in materials engineering has brought with it an increased interest and application of Image Driven Machine Learning (IDML) approaches. In this work, we review the applications of IDML to the field of materials characterization. A canonical hierarchy of stages is defined, which when put sequentially together completes an IDML study: problem definition, dataset building, model selection and training, model evaluation, and integration with existing instrumentation or simulation workflow. The studies reviewed in this work are analyzed from the perspective of each of these stages. Such a review permits agranular assessment of the field, for example the impact of IDML on materials characterization at the nanoscale, the size of a typical dataset required to train a semantic segmentation model on electron microscopy images, ubiquitousness of transfer learning in the domain, etc. Finally, we discuss the importance of interpretability and explainability in the field of IDML for materials characterization, and provide an overview of two emerging techniques in the field: semantic segmentation and generative adversarial networks.

Baskaran, Arun↗
DOE OSTI2024

The Delicate Balance Redux: The Role of Nuclear Forces, Damage Limitation and Uncertainty in Future U.S.-China Crises

What is the impact of damage limitation capabilities like counterforce and missile defenses on deterrence, when their efficacy in stopping an adversary nuclear attack is uncertain? This is a key unanswered question to understand “how much is enough” for the United States to deter China and Russia in future nuclear crises. In this paper we extend an established, single move game theory model to capture the dynamics of two players in a nuclear crisis having varying damage limitation capabilities with uncertain effectiveness. Our model formalizes the logic of the “delicate balance” school of deterrence, which states that leverage in a crisis is driven by the risk each player can take with their combined strategic forces, and that those risks carry uncertainty as nuclear forces are hard to deliver against technologically advanced adversaries. Our model shows that damage limitation capabilities—even those with significant uncertainty around them like cyber or electronic warfare—can drive bargaining outcomes in an array of nuclear crises. We then apply these bargaining outcomes to the expected U.S.-China strategic balance as China builds out its nuclear force through 2035. We apply published force exchange models to determine the expected damage each side will be able to deliver, and we use these values to determine the likelihood that the U.S. can prevail in crises of varying stakes. Last, we show that U.S. policymakers have an array of options to improve future bargaining outcomes, evaluating how additional nuclear forces trade against improvements in damage limitation.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗
NASA NTRSDate not supplied

Analyzing Natural Language Context in Human-Machine Teaming using Supervised Machine Learning

Building a foundation for trustworthiness and trust verification in multi-asset teaming is the research challenge of Autonomy Teaming and TRAjectories for Complex Trusted Operational Reliability (ATTRACTOR). The Design Reference Mission (DRM) for ATTRACTOR is a search and rescue mission objective governed by a multi-member team consisting of human and machine operators. A crucial component to the effort is the communication between humans and autonomous agents throughout both planning and execution stages of the mission. Intuitive communication methods and modalities are posited as critical enablers for certifying trust and trustworthiness. This paper reports on the data collection and analysis conducted in support of the Human Informed Natural-language GANs Evaluation (HINGE)project to attain explainable and trusted communication between human-machine assets. Two identically curated image description datasets were acquired for HINGE, both consisting of two unique input modalities (typed vs. verbal) and retrieved in two distinct contexts (general vs. specific). The gathered datasets were assessed and compared using Parts-of-Speech (POS)features, sentence similarity metrics, and linguistic analysis. Then, the datasets were modeled and tested separately and in combination with one another using machine learning algorithms. The comparison and testing results reveal a superior dataset, by which a preferred context and input is understood, for generating image representations of missing persons using a Generative Adversarial Network (GAN).

Bryan A Barrows↗
DOE OSTI2025

CaloChallenge 2022: a community challenge for fast calorimeter simulation

Here, we present the results of the ‘Fast Calorimeter Simulation Challenge 2022’—the CaloChallenge. We study state-of-the-art generative models on four calorimeter shower datasets of increasing dimensionality, ranging from a few hundred voxels to a few tens of thousand voxels. The 31 individual submissions span a wide range of current popular generative architectures, including variational autoencoders (VAEs), generative adversarial networks (GANs), normalizing flows, diffusion models, and models based on conditional flow matching. We compare all submissions in terms of quality of generated calorimeter showers, as well as shower generation time and model size. To assess the quality we use a broad range of different metrics including differences in one-dimensional histograms of observables, KPD/FPD scores, AUCs of binary classifiers, and the log-posterior of a multiclass classifier. The results of the CaloChallenge provide the most complete and comprehensive survey of cutting-edge approaches to calorimeter fast simulation to date. In addition, our work provides a uniquely detailed perspective on the important problem of how to evaluate generative models. As such, the results presented here should be applicable for other domains that use generative AI and require fast and faithful generation of samples in a large phase space.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗