Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “adversarial evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Autonomous Tools for Attack Surface Reduction (Final Report)

The electric power grid is a complex critical infrastructure that forms the lifeline of modern society, and its secure and reliable operation is of paramount importance to national security and economic wellbeing. However, recent findings documented in authoritative sources indicate the threat of cyber-based attacks growing in numbers and sophistication. However, securing the grid against stealthy cyberattacks is a challenging task due to legacy nature of the infrastructure coupled with dynamic nature of threat landscape and ever-growing sophistication of the adversaries. Additionally, the grid’s attack surface continues to grow with the increased dependence on digital communications and control that now extends to each consumer through smart meters and distributed energy resources. Unfortunately, this expansive surface increases the grid’s vulnerability and further exposes critical control systems in both substations and control centers. To respond to this emerging need, we had successfully assembled an interdisciplinary team with academic- industry partnership to successfully conduct research, development, evaluation, demonstration, and commercialization of attack surface reduction tools, whose goal was to significantly reduce the cyber attack surface in the North American power grid. Our proposed project was a synergistic collaborative effort leveraging the synergistic expertise of the team members across power systems, cyber security and CPS security, testbeds, field deployments and demonstration, and successful commercialization. The following are the specific tasks that have been successfully completed two phases (2016-2020). Phase I: Task 1: Developed and implemented a robust Project Management and Data Management Plan, coupled with a well thought out Risk Mitigation Plan. Task 2.1: Developed a comprehensive framework that continually assesses and autonomously reduces the attack surface for the power grid control environment spanning across substations, control center and the SCADA network to significantly reduce the risks of cyber attacks. Task 2.2: Developed attack surface analysis techniques, metrics, and tools that assess the attack surface at multiple levels including the control center, substations, and the SCADA network. Task 2.3: Developed attack surface reduction techniques and tools that dynamically reduce attack surface and hence increase attacker’s cost without interfering in the critical functions of the system. Task 2.4: Prototyped, implemented, and quantitatively evaluated/validated the techniques and tools on a realistic industrial CPS security testbed environment by leveraging the unique resources of the team. Task 3: Developed Commercialization plan to transition the developed tools into power system industry stakeholders for a broader adoption by leveraging the expertise of our industrial members. Phase II: Task 4: Successfully completed field demonstration, verification, and evaluation of the effectiveness of the attack surface analysis and reduction techniques on a realistic utility testbed environment. This also involved the development of realistic scenarios, sound metrics, data sets, evaluation criteria, and documentation. Technology integration & Field demonstration: The project had significantly advanced the state-of-the-art research and practice in improving the cybersecurity of our nation’s power grid infrastructure against cyber threats. In particular, the proposed, designed, and deployed attack surface analysis and reduction algorithms and tools have contributed to significantly reducing the exposure and risk of the devices, substations, and the integrated SCADA/EMS/ DMS grid environment to cyber threat. Strong demonstration and evaluation techniques have verified the feasibility of the developed techniques on realistic cyber-physical testbeds and utility partner's real grid environment, and collaborative research and evaluation of attack surface reduction techniques (for wide-are monitoring and control) within a vendor (GE) EMS platform. The Attack Host Analyzer (AHA) tool that was developed through this project was made available through GitHub.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2023

Deep generative models for vehicle speed trajectories

Generating realistic vehicle speed trajectories is a crucial component in evaluating vehicle fuel economy and in predictive control of self-driving cars. Traditional generative models rely on Markov chain methods and can produce accurate synthetic trajectories but are subject to the curse of dimensionality. They do not allow to include conditional input variables into the generation process. In this paper, we show how extensions to deep generative models allow accurate and scalable generation. Proposed architectures involve recurrent and feed-forward layers and are trained using adversarial techniques. Our models are shown to perform well on generating vehicle trajectories using a model trained on GPS data from Chicago metropolitan area.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2023

Digital Engineering and Cybersecurity Decision Analysis in Early Phases of SMR-Driven IES Projects

Considerable efforts are underway to ensure cybersecurity is integrated into the systems engineering lifecycle. Cyber-informed engineering and security-by-design frameworks are intended to identify and engineer out cybersecurity risks throughout the lifecycle. While these approaches are valuable for promoting the need to include cybersecurity considerations in early design phases to create more secure systems, they may not consider the entirety of digital risks. Digital risks in a digital instrumentation and control system include adversarial and unintentional risks from internal and external factors, such as human performance errors, design flaws, environmental conditions, and equipment degradation or failure. This report provides a detailed discussion on digital risk prior to describing the background and concept of operations for a small modular reactor-driven integrated energy system connected to industrial applications. The challenges of competing objectives and competing stakeholder requirements are discussed and the impacts on digital engineering, security considerations, and interdependencies are evaluated for mission-level, facility-level, and system-level decisions.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗
DOE OSTI2020

Image-driven discriminative and generative machine learning algorithms for establishing microstructure–processing relationships

We investigate methods of microstructure representation for the purpose of predicting processing condition from microstructure image data. A binary alloy that is currently under development as a nuclear fuel was studied for the purpose of developing an improved machine learning approach to image recognition, characterization, and building predictive capabilities linking microstructure to processing conditions. Here, we test different microstructure representations and evaluate model performance based on classification accuracy. A classification accuracy of 95.8% was achieved fordistinguishing between micrographs corresponding to ten different thermo-mechanical material processing conditions.We find that our newly developed microstructure representation describes image data well, and the traditional approachof utilizing area fractions of different phases is insufficient for distinguishing between multiple classes using a relativelysmall, imbalanced original data set of 272 images. To explore the applicability of generative methods for supplementing such limited data sets, generative adversarial networks were trained to generate artificial microstructure images. Two different generative networks were trained and tested to assess performance. Challenges and best practices associated with applying machine learning to limited microstructure image data sets is also discussed. Our work has implications for quantitative microstructure analysis, and development of microstructure-processing relationships in limited data sets typical of metallurgical process design studies.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Inferring adversarial behaviour in cyber‐physical power systems using a Bayesian attack graph approach

Abstract Highly connected smart power systems are subject to increasing vulnerabilities and adversarial threats. Defenders need to proactively identify and defend new high‐risk access paths of cyber intruders that target grid resilience. However, cyber‐physical risk analysis and defense in power systems often requires making assumptions on adversary behaviour, and these assumptions can be wrong. Thus, this work examines the problem of inferring adversary behaviour in power systems to improve risk‐based defense and detection. To achieve this, a Bayesian approach for inference of the Cyber‐Adversarial Power System (Bayes‐CAPS) is proposed that uses Bayesian networks (BNs) to define and solve the inference problem of adversarial movement in the grid infrastructure towards targets of physical impact. Specifically, BNs are used to compute conditional probabilities to queries, such as the probability of observing an event given a set of alerts. Bayes‐CAPS builds initial Bayesian attack graphs for realistic power system cyber‐physical models. These models are adaptable using collected data from the system under study. Then, Bayes‐CAPS computes the posterior probabilities of the occurrence of a security breach event in power systems. Experiments are conducted that evaluate algorithms based on time complexity, accuracy and impact of evidence for different scales and densities of network. The performance is evaluated and compared for five realistic cyber‐physical power system models of increasing size and complexities ranging from 8 to 300 substations based on computation and accuracy impacts.

Sahu, Abhijeet↗
DOE OSTI2020

Semisupervised Learning for Seismic Monitoring Applications

The impressive performance that deep neural networks demonstrate on a range of seismic monitoring tasks depends largely on the availability of event catalogs that have been manually curated over many years or decades. However, the quality, duration, and availability of seismic event catalogs vary significantly across the range of monitoring operations, regions, and objectives. Semisupervised learning (SSL) enables learning from both labeled and unlabeled data and provides a framework to leverage the abundance of unreviewed seismic data for training deep neural networks on a variety of target tasks. We apply two SSL algorithms (mean-teacher and virtual adversarial training) as well as a novel hybrid technique (exponential average adversarial training) to seismic event classification to examine how unlabeled data with SSL can enhance model performance. In general, we find that SSL can perform as well as supervised learning with fewer labels. We also observe in some scenarios that almost half of the benefits of SSL are the result of the meaningful regularization enforced through SSL techniques and may not be attributable to unlabeled data directly. Lastly, the benefits from unlabeled data scale with the difficulty of the predictive task when we evaluate the use of unlabeled data to characterize sources in new geographic regions. Finally, in geographic areas where supervised model performance is low, SSL significantly increases the accuracy of source-type classification using unlabeled data.

58 GEOSCIENCES↗
DOE OSTI2024

The Digital Assurance for High Consequence Systems (DAHCS) Mission Campaign Whitepaper

The DAHCS (pronounced “Dax”) MC is a 7-year, $\$$45 million research portfolio within Sandia’s Laboratory Directed Research and Development program. The DAHCS MC arose in response to a great need: to ensure that the use of digital technologies does not weaken our nation’s high consequence systems. Digital technologies offer many benefits in speed, cost, and flexibility, and we seek to reap those benefits without introducing new system failures. However, digital technologies cannot be evaluated the same way as analog technologies. Initiatives across the nation highlight the capability gap that prevents efficient, effective digital assurance. The Challenge Today’s digital assurance tools, techniques, and methods are inadequate to confidently characterize, assess, and manage digital risk; they are ad hoc, slow, costly, and rarely scalable to increasingly complex digital technologies. The rapidly evolving cyber threat landscape exacerbates this problem because digital assurance now must secure against digital risks now and in the future, including those introduced by rapidly evolving technologies, adversaries, and systems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Application of Cyber-Informed Engineering for Protecting BESS

This white paper synthesizes an array of crucial grid services provided by BESS technology, assesses its architecture and communications, and presents a case study for analysis against the principles introduced by Cyber-Informed Engineering (CIE). Furthermore, in walking through the analysis, this paper presents a framework to evaluate risks and solutions when considering BESS components. Asset owners and buyers could perform this analysis to assess their BESS product implementations, alternative inverter-based resources (IBR), and energy management systems (EMS). Battery systems fulfill various roles contingent on the unique market demands and the specific challenges presented by regional grid infrastructures. These roles also vary due to the differing utility models for ownership and operation, which are adapted to meet regional and local capabilities and requirements. Concerns have been raised regarding the potential for adversaries to exploit knowledge of battery operational patterns to orchestrate decisive attacks. However, the security of operational data for these systems may not be the primary vulnerability, as much of this information is already well-understood within the community. Applying a modest degree of subject matter expertise can often yield valuable predictions regarding how a battery will respond under certain conditions, such as grid emergencies, high or low-temperature days, Public Safety Power Shutoff (PSPS) events, and outages. The operational characteristics of batteries are well-documented, and their capabilities, including the risks associated with misoperation and the resulting consequences, are published and understood within the industry. CIE practices represent the next step in gaining functional assurance and providing an acceptable level of risk, regardless of whether a battery vendor can support a trusted and validated supply chain. While this issue has exacerbated supply chain challenges, it is not an isolated condition. This foreign supply route is the primary source of BESS for the U.S. market. Significant efforts are underway through the Bipartisan Infrastructure Law (BIL) to change that. Still, strategic short-term operational mitigations are needed to ensure the security of our operational technology (OT) systems, which are enhanced by instilling trust and are separate from vendors implementing CIE principles.

25 ENERGY STORAGE↗
DOE OSTI2022

Revisiting Current Paradigms: Subject Matter Expert Views on High Consequence Facility Security Assessments

Security assessments support decision-makers' ability to evaluate current capabilities of high consequence facilities (HCF) to respond to possible attacks. However, increasing complexity of today's operational environment requires a critical review of traditional approaches to ensure that implemented assessments are providing relevant and timely insights into security of HCFs. Using interviews and focus groups with diverse subject matter experts (SMEs), this study evaluated the current state of security assessments and identified opportunities to achieve a more "ideal" state. The SME-based data underscored the value of a systems approach for understanding the impacts of changing operational designs and contexts (as well as cultural influences) on security to address methodological shortcomings of traditional assessment processes. These findings can be used to inform the development of new approaches to HCF security assessments that are able to more accurately reflect changing operational environments and effectively mitigate concerns arising from new adversary capabilities.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Advanced Reactor Safeguards & Security Program: Cybersecurity Scenarios

The use of digital control systems and automation in advanced nuclear power systems introduces different types of vulnerabilities compared to legacy (i.e. analog) control systems that cyber adversaries can exploit. These vulnerabilities pose a challenge to reactor operators and cyber operations staff due to the dynamic nature of the event in which a human response or a lack of response can potentially evolve into a worsening plant condition. Using the Department of Homeland Security Cyber and Infrastructure Security Agency’s (CISA) critical infrastructure exercise framework, this document presents several cyber security scenarios typical of digital control systems that could be used in advanced reactor designs. These scenarios can be used in tabletop exercises to evaluate cyber security posture or conduct training on different aspects of cyber security, including detection, threat hunting using indicators of compromise, evaluating incident response, risk mitigation, incident reporting, information sharing and recovery.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2020

Assessing DER Network Cybersecurity Defences in a Power-Communication Co-Simulation Environment

Increasing penetrations of interoperable distributed energy resources (DER) in the electric power system are expanding the power system attack surface. Maloperation or malicious control of DER equipment can now cause substantial disturbances to grid operations. Fortunately, many options exist to defend and limit adversary impact on these newly-created DER communication networks, which typically traverse the public internet. However, implementing these security features will increase communication latency, thereby adversely impacting real-time DER grid support service effectiveness. In this work, a collection of software tools called SCEPTRE were used to create a co-simulation environment where SunSpec-compliant PV inverters were deployed as virtual machines and interconnected to simulated communication network equipment. Network segmentation, encryption, and moving target defence security features were deployed on the control network to evaluate their influence on cybersecurity metrics and power system performance. The results indicated that adding these security features did not impact DER-based grid control systems but improved the cybersecurity posture of the network when implemented appropriately.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Comprehension of Spatial Constraints by Neural Logic Learning from a Single RGB-D Scan

Autonomous industrial assembly relies on the precise measurement of spatial constraints as designed by computer-aided design (CAD) software such as SolidWorks. This paper proposes a framework for an intelligent industrial robot to understand the spatial constraints for model assembly. An extended generative adversary network (GAN) with a 3D long short-term memory (LSTM) network was designed to composite 3D point clouds from a single RGB-D scan. The spatial constraints of the segmented point clouds are identified by a neural-logic network that incorporates general knowledge of spatial constraints in terms of first-order logic. The model was designed to comprehend a complete set of spatial constraints that are consistent with industrial CAD software, including left, right, above, below, front, behind, parallel, perpendicular, concentric, and coincident relations. The accuracy of 3D model composition and spatial constraint identification was evaluated by the RGB-D scans and 3D models in the ABC dataset. The proposed model achieved 57.23% intersection over union (IoU) in 3D model composition, and over 99% in comprehending all spatial constraints.

Wang, Dali↗
DOE OSTI2022

Evaluation of Joint Cyber/Safety Risk in Nuclear Power Systems

This report presents an analysis of the Emergency Core Cooling System (ECCS) for a generic Boiling Water Reactor (BWR)-4 NPP. The Electric Power Research Institute (EPRI) developed Hazards and Consequences Analysis for Digital Systems (HAZCADS) process is applied to the ECCS and its subsystems to identify unsafe control actions (UCAs) which act as possible cyber events of concern. The analysis is performed for two design basis events: Small-break Loss of Coolant Accident (SLOCA) and general transients (TRANS), such as unintended reactor trip. In previous work, HAZCADS UCAs were combined with other cyber-attack analysis to develop a risk-informed approach; however, this was for a single system. This report explores advanced systems engineering modeling approaches to model the interactions between digital assets across multiple systems which may be targeted by cyber adversaries. The complex and interdependent design of digital systems has the potential to introduce emergent cyber properties that are generally not covered by hazard analyses nor formal nuclear Probabilistic Risk Assessment (PRA). The R&D and supporting analysis presented here explores approaches to predict and manage how interdependent system properties effect risk. To show the potential impact of a successful cyber-attack to formal PRA event tree probabilities, HAZCADS analysis was also used. HAZCADS was also used to model the automatic depressurization system (ADS) automatic actuation. This analysis extended to an integrated system analysis for common-cause failure (CCF). In this aspect, the HAZCADS analysis continued by analyzing plant design details for system connectivity in support of critical plant functions. A dependency matrix was developed to depict the integrated functionality of the interconnected systems. Areas of potential CCF are indicated. Future work could include adversary attack development to show how CCF could be caused, resulting in PRA events. Across the multiple systems that comprise the ECCS, the analysis shows that the change in such probabilities was very different between systems. This indicates that some systems have a larger potential risk impact from successful cyber-attack or digital failure, which indicates a need for these systems to have a higher priority for design and defensive measures. Furthermore, we were able to establish that a risk analysis using any arbitrary threat model establishes an ordering of components with regard to cyber-risk. This ordering can be used to influence the overall system design with an eye to lowering risk, or as a way to understand real-time risk to operational systems based on a current threat landscape. Expert knowledge of both the analysis process and the system being analyzed is required to perform a HAZCADS analysis. The need for a tiered risk analysis is demonstrated by the results of this report.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2021

ForSE: A GAN-based Algorithm for Extending CMB Foreground Models to Subdegree Angular Scales

We present ForSE (Foreground Scale Extender), a novel Python package that aims to overcome the current limitations in the simulation of diffuse Galactic radiation, in the context of cosmic microwave background (CMB) experiments. ForSE exploits the ability of generative adversarial neural networks (GANs) to learn and reproduce complex features present in a set of images, with the goal of simulating realistic and non-Gaussian foreground radiation at subdegree angular scales. This is of great importance in order to estimate the foreground contamination to lensing reconstruction, delensing, and primordial B-modes for future CMB experiments. We applied this algorithm to Galactic thermal dust emission in both total intensity and polarization. Our results show how ForSE is able to generate small-scale features (at 12') having as input the large-scale ones (80'). The injected structures have statistical properties, evaluated by means of the Minkowski functionals, in good agreement with those of the real sky and which show the correct amplitude scaling as a function of the angular dimension. Furthermore, the obtained thermal dust Stokes Q and U full-sky maps as well as the ForSE package are publicly available for download.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2021

Adoption of image-driven machine learning for microstructure characterization and materials design: A Perspective

Microstructure characterization enables the development of structure-processing-property relationships critical to several research areas within the broad field of materials science, from alloy design to the assessment of corrosion resistance, and failure analysis. Conventional approaches to material characterization have relied on either qualitative inference by the human ex-pert or software applications that can extract high-level features from images, such as boundary segmentation, average grain diameter, etc. Such approaches rely heavily on subject matter expert user intervention and knowledge of what phases or more generally, what microstructural features, are of interest. The recent surge in the adoption of machine learning techniques to address problems in materials engineering has brought with it an increased interest and application of Image Driven Machine Learning (IDML) approaches. In this work, we review the applications of IDML to the field of materials characterization. A canonical hierarchy of stages is defined, which when put sequentially together completes an IDML study: problem definition, dataset building, model selection and training, model evaluation, and integration with existing instrumentation or simulation workflow. The studies reviewed in this work are analyzed from the perspective of each of these stages. Such a review permits agranular assessment of the field, for example the impact of IDML on materials characterization at the nanoscale, the size of a typical dataset required to train a semantic segmentation model on electron microscopy images, ubiquitousness of transfer learning in the domain, etc. Finally, we discuss the importance of interpretability and explainability in the field of IDML for materials characterization, and provide an overview of two emerging techniques in the field: semantic segmentation and generative adversarial networks.

Baskaran, Arun↗
DOE OSTI2024

The Delicate Balance Redux: The Role of Nuclear Forces, Damage Limitation and Uncertainty in Future U.S.-China Crises

What is the impact of damage limitation capabilities like counterforce and missile defenses on deterrence, when their efficacy in stopping an adversary nuclear attack is uncertain? This is a key unanswered question to understand “how much is enough” for the United States to deter China and Russia in future nuclear crises. In this paper we extend an established, single move game theory model to capture the dynamics of two players in a nuclear crisis having varying damage limitation capabilities with uncertain effectiveness. Our model formalizes the logic of the “delicate balance” school of deterrence, which states that leverage in a crisis is driven by the risk each player can take with their combined strategic forces, and that those risks carry uncertainty as nuclear forces are hard to deliver against technologically advanced adversaries. Our model shows that damage limitation capabilities—even those with significant uncertainty around them like cyber or electronic warfare—can drive bargaining outcomes in an array of nuclear crises. We then apply these bargaining outcomes to the expected U.S.-China strategic balance as China builds out its nuclear force through 2035. We apply published force exchange models to determine the expected damage each side will be able to deliver, and we use these values to determine the likelihood that the U.S. can prevail in crises of varying stakes. Last, we show that U.S. policymakers have an array of options to improve future bargaining outcomes, evaluating how additional nuclear forces trade against improvements in damage limitation.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗
DOE OSTI2025

CaloChallenge 2022: a community challenge for fast calorimeter simulation

Here, we present the results of the ‘Fast Calorimeter Simulation Challenge 2022’—the CaloChallenge. We study state-of-the-art generative models on four calorimeter shower datasets of increasing dimensionality, ranging from a few hundred voxels to a few tens of thousand voxels. The 31 individual submissions span a wide range of current popular generative architectures, including variational autoencoders (VAEs), generative adversarial networks (GANs), normalizing flows, diffusion models, and models based on conditional flow matching. We compare all submissions in terms of quality of generated calorimeter showers, as well as shower generation time and model size. To assess the quality we use a broad range of different metrics including differences in one-dimensional histograms of observables, KPD/FPD scores, AUCs of binary classifiers, and the log-posterior of a multiclass classifier. The results of the CaloChallenge provide the most complete and comprehensive survey of cutting-edge approaches to calorimeter fast simulation to date. In addition, our work provides a uniquely detailed perspective on the important problem of how to evaluate generative models. As such, the results presented here should be applicable for other domains that use generative AI and require fast and faithful generation of samples in a large phase space.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2023

Development of a Machine-Learned Cruise Guide Indicator for Rotorcraft

This paper presents a machine-learned virtual cruise guide indicator (vCGI) for Chinook helicopters. Two temporal neural networks were trained and evaluated on measured data from 55 flight tests, one for the fore rotor and another for the aft rotor, to predict a vCGI value, which protects 23 components from fatigue damage during steady-state conditions. Three different classes of machine learning architectures were evaluated for prediction of the vCGI from time sequences: a temporal convolutional neural network with 1D dilated causal convolutions, a long short-term memory recurrent neural network, and an attention-based transformer architecture. The final average model accuracy on unseen flight data is currently greater than 93% for CGI values which could result in fatigue damage and 90% for normal operation CGI values. Model accuracy was improved through a series of advancements in:(1) selection of optimal training data using temporal collective variables and unsupervised learning, (2) dataset augmentation with maximum-entropy temporal collective variables, and (3) implementation of a mixture-of-experts classification- regression approach using an adversarial classification approach to assign maneuver labels. The results are presented for each advancement in model development along with lessons learned in training machine learning models on real- world, time-dependent rotorcraft data.

Boyer, Mathew↗