Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “adversarial evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Domain Adaptation for Measurements of Strong Gravitational Lenses

Upcoming surveys are predicted to discover galaxy-scale strong lenses on the order of 10\textsuperscript{5}, making deep learning methods necessary in lensing data analysis. Currently, there is insufficient real lensing data to train deep learning algorithms, but the alternative of training only on simulated data results in poor performance on real data. Domain Adaptation may be able to bridge the gap between simulated and real datasets. We utilize domain adaptation for the estimation of Einstein radius ($\Theta_E$) in simulated galaxy-scale gravitational lensing images with different levels of observational realism. We evaluate two domain adaptation techniques - Domain Adversarial Neural Networks (DANN) and Maximum Mean Discrepancy (MMD). We train on a source domain of simulated lenses and apply it to a target domain of lenses simulated to emulate noise conditions in the Dark Energy Survey (DES). We show that both domain adaptation techniques can significantly improve the model performance on the more complex target domain dataset. This work is the first application of domain adaptation for a regression task in strong lensing imaging analysis. Our results show the potential of using domain adaptation to perform analysis of future survey data with a deep neural network trained on simulated data.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2023

Domain Adaptation for Measurements of Strong Gravitational Lenses

Upcoming surveys are predicted to discover galaxy-scale strong lenses on the order of $10^5$, making deep learning methods necessary in lensing data analysis. Currently, there is insufficient real lensing data to train deep learning algorithms, but the alternative of training only on simulated data results in poor performance on real data. Domain Adaptation may be able to bridge the gap between simulated and real datasets. We utilize domain adaptation for the estimation of Einstein radius ($\Theta_E$) in simulated galaxy-scale gravitational lensing images with different levels of observational realism. We evaluate two domain adaptation techniques - Domain Adversarial Neural Networks (DANN) and Maximum Mean Discrepancy (MMD). We train on a source domain of simulated lenses and apply it to a target domain of lenses simulated to emulate noise conditions in the Dark Energy Survey (DES). We show that both domain adaptation techniques can significantly improve the model performance on the more complex target domain dataset. This work is the first application of domain adaptation for a regression task in strong lensing imaging analysis. Our results show the potential of using domain adaptation to perform analysis of future survey data with a deep neural network trained on simulated data.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2023

Domain Adaptation for Measurements of Strong Gravitational Lenses

Upcoming surveys are predicted to discover galaxy-scale strong lenses on the magnitude of 105, making deep learning methods necessary in lensing data analysis. Currently, there is insufficient real lensing data to train deep learning algorithms, but training only on simulated data results in poor performance on real data. Domain adaptation can bridge the gap between simulated and real datasets. We adopt domain adaptation on the estimation of Einstein radius in simulated galaxy-scale gravitational lensing images. We evaluate two domain adaptation techniques - domain adversarial neural networks (DANN) and maximum mean discrepancy (MMD). We train on a source domain of simulated lenses and apply it to a target domain with emulation of DES survey conditions. We show that both domain adaptation techniques can significantly improve the model performance on the more complex target domain datasets. Our results show the potential of using domain adaptation to perform analysis on future survey data with a deep neural network trained on simulated data.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2026

Community detection robustness of graph neural networks

Graph neural networks (GNNs) are increasingly widely used for community detection in attributed networks. They combine structural topology with node attributes through message passing and pooling. However, their robustness or lack thereof with respect to different perturbations and targeted attacks in conjunction with community detection tasks is not well understood. To shed light on latent mechanisms behind GNN sensitivity on community detection tasks, we conduct a systematic computational evaluation of six widely adopted GNN architectures graph convolutional network, graph attention network, graph sample and aggregate (GraphSAGE), differentiable pooling (DiffPool), minimum cut pooling (MinCUT), and deep modularity networks (DMoN). The analysis covers three perturbation categories: node attribute manipulations, edge topology distortions, and adversarial attacks. We use element-centric similarity as the evaluation metric on synthetic benchmarks and real-world citation networks. Our findings indicate that supervised GNNs tend to achieve higher baseline accuracy, while unsupervised methods, particularly DMoN, maintain stronger resilience under targeted and adversarial perturbations. Furthermore, robustness appears to be strongly influenced by community strength, with well-defined communities reducing performance loss. Across all models, node attribute perturbations associated with targeted edge deletions and shifts in attribute distributions tend to cause the largest degradation in community recovery. These findings highlight important trade-offs between accuracy and robustness in GNN-based community detection and offer insights into selecting architectures resilient to noise and adversarial attacks.

Goel, Jaidev [Virginia Polytechnic Inst. and State↗
DOE OSTI2022

Dante

SAND2022-12905 O Dante is a physical security simulation package used in evaluating protective staffing and physical security elements against adversarial threats. Dante simulates force-on-force scenarios in a batch Monte Carlo process. Analysts use the included Scenario Editor to define the scenarios for their site and then execute and analyze the results using the Batch Manager tool. Dante provides insight into how well a security system performs against threats and lets analysts mine the results for key indicators of system performance. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Hart, Brian↗
DOE OSTI2020

Cycle-Consistent Adversarial Networks for Realistic Pervasive Change Generation in Remote Sensing Imagery

This paper introduces a new method of generating realistic pervasive changes in the context of evaluating the effectiveness of change detection algorithms in controlled settings. The method - a cycle-consistent adversarial network (CycleGAN) - requires low quantities of training data to generate realistic changes. Here we show an application of CycleGAN in creating realistic snow-covered scenes of multispectral Sentinel-2 imagery, and demonstrate how these images can be used as a test bed for anomalous change detection algorithms.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

GRUMDN: A Multi-Task Model for Predicting Human Patterns-of-Life from Stay Transition Data

Understanding human patterns-of-life (PoL) is essential towards ensuring safe and secure indoor facility environment as well as outdoor urban environment. Prediction of human movement in between places of interest is vital in understanding human PoL. Movement between spaces maybe represented and detected in one of the two forms: 1) trajectories: locations measured at regular time intervals by mobile sensors, bluetooth or GPS sensors; or 2) stay transitions: semantic PoI (points of interest) and stay duration data measurable by eventbased sensors that collect data when a check-in or check-out event is detected. Stay transition data provides a more compressed data format compared to trajectories data, especially in situations with longer stay durations, while preserving the information necessary for PoL analysis. Now as introduced briefly in the paper, our deployed end application (Digital Twin of a facility with non-player characters, besides the interactive user in virtual reality) needed a well-performing and validated AI/ML model for simulating high quality stay transitions behavior. In this study we thus primarily present our findings with developing and validating that model, which is a multi-task neural network for stay transition prediction. The neural network consists of two heads, for corresponding two tasks of stay category prediction and stay duration prediction. We evaluated gated recurrent units and multi-layer perceptrons of varying network sizes for stay category prediction; while mixture density networks, noisy generator-only networks, and generative adversarial networks of varying network sizes for stay duration prediction. We have then evaluated four multi-task models, constructed by combining these specialized models, on their ability to predict stay transition data. We tested our models on datasets from two different cases: 1) a simulation-generated dataset of indoor movement within the HFIR (high flux isotope reactor) nuclear reactor facility at Oak Ridge National Laboratory (ORNL); and 2) the GeoLife human mobility dataset of outdoor urban movement available in literature. Our results indicate that GRUMDN, which combines gated recurrent units (GRU) for stay category prediction task, and mixture density networks (MDN) for stay duration prediction task, did overall outperform other multitask models and the current state-of-the-art.

Gunaratne, Chathika [ORNL] (ORCID:0000000225088745↗
DOE OSTI2025

Security-by-Design: Light Water Small Modular Reactor

The growing demand for nuclear power is increasing pressure to find solutions to cost prohibitive requirements of both construction and security. Offsite response has been proposed as an option to reduce costs associated with training and maintaining an onsite response force. A previous report explored this option and revealed that security could be provided at the required level, but cost savings was not a result of this methodology. An offsite response strategy required costly active and passive delay barriers to provide sufficient time for responders to muster and deploy to a site in time to interrupt a determined and well-equipped adversary. Also, contrary to the hypothesis, the number of responders required for this strategy exceeded that needed for an onsite response force, as the adversaries could avail themselves of advantageous positions within the facility to repel arriving responders. This report builds upon the previous evaluation by using the same hypothetical light water small modular reactor (LWSMR) facility model, but this time an onsite response strategy was assessed. The goal of this analysis was to show that an onsite response strategy could be implemented effectively at a cost point that removes barriers within the industry at this critical time of growth and development. The assessment of the facility design and response strategy was completed through modeling using Scribe3D© and subsequent scenario analysis over the course of a two-day tabletop exercise. Subject matter experts in nuclear security, nuclear facility design, and response strategy and tactics contributed to the effort to ensure accurate representation of hypothetical scenarios. Several adaptations were made to the layout of the LWSMR based on lessons learned during the first day of scenario analysis. The subsequent design evaluated on the second day proved to provide a robust response posture against a large and well-trained adversary force. This report details the process of the analysis and compares the cost of the final facility design with that of the LWSMR model used for evaluation of offsite response. Ultimately, the results of this effort indicate that, when implemented correctly, an onsite response strategy is the best option from a security and cost perspective.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2023

Measuring Equality in Machine Learning Security Defenses: A Case Study in Speech Recognition

Over the past decade, the machine learning security community has developed a myriad of defenses for evasion attacks. An understudied question in that community is: for whom do these defenses defend? This work considers common approaches to defending learned systems and how security defenses result in performance inequities across different sub-populations. We outline appropriate parity metrics for analysis and begin to answer this question through empirical results of the fairness implications of machine learning security methods. We find that many methods that have been proposed can cause direct harm, like false rejection and unequal benefits from robustness training. The framework we propose for measuring defense equality can be applied to robustly trained models, preprocessing-based defenses, and rejection methods. We identify a set of datasets with a user-centered application and a reasonable computational cost suitable for case studies in measuring the equality of defenses. In our case study of speech command recognition, we show how such adversarial training and augmentation have non-equal but complex protections for social subgroups across gender, accent, and age in relation to user coverage. We present a comparison of equality between two rejection-based defenses: randomized smoothing and neural rejection, finding randomized smoothing more equitable due to the sampling mechanism for minority groups. This represents the first work examining the disparity in the adversarial robustness in the speech domain and the fairness evaluation of rejection-based defenses.

• Artificial intelligence (AI) / machine learning ↗
DOE OSTI2022

Deep reinforcement learning with online data augmentation to improve sample efficiency for intelligent HVAC control

Deep Reinforcement Learning (DRL) has started showing success in real-world applications such as building energy optimization. Much of the research in this space utilized simulated environments to train RL-agent in an offline mode. Very few research have used DRL-based control in real-world systems due to two main reasons: 1) sample efficiency challenge---DRL approaches need to perform a lot of interactions with the environment to collect sufficient experiences to learn from, which is difficult in real systems, and 2) comfort or safety related constraints---user's comfort must never or at least rarely be violated. In this work, we propose a novel deep Reinforcement Learning framework with online Data Augmentation (RLDA) to address the sample efficiency challenge of real-world RL. We used a time series Generative Adversarial Network (TimeGAN) architecture as a data generator. We further evaluated the proposed RLDA framework using a case study of an intelligent HVAC control. With a ≈28% improvement in the sample efficiency, RLDA framework lays the way towards increased adoption of DRL-based intelligent control in real-world building energy management systems.

Kurte, Kuldeep↗
DOE OSTI2025

Modularization of Small Modular Reactor Facilities: Physical Protection Recommendations

U.S. nuclear power facilities face increasing challenges in meeting dynamic security requirements caused by evolving and expanding threats while keeping costs reasonable to make nuclear energy competitive. In consideration of the economic viability of small modular reactor (SMR) facilities, many designers and utilities are considering a modular approach to the construction and operation of these facilities. This modular approach considers building and operating a first unit; once the first reactor is in operation, construction will begin on a second reactor. This process would allow the vendor or utility to ensure production of energy and financial income while the second reactor is under construction. This project evaluates the feasibility of modular construction in terms of physical protection and identifies several recommendations for vendors and utilities considering this approach. To conduct this evaluation, a hypothetical three-unit SMR facility was developed, a physical protection system (PPS) was considered for the design, and a hypothetical design basis threat (DBT) was used to evaluate this PPS. Multiple outsider sabotage scenarios were examined, with adversary team sizes ranging from 4-to-8 to determine security system effectiveness. The results of this work will influence PPS designs and facility designs for U.S. domestic SMRs. This work will also demonstrate how a series of experimental and modeling capabilities across the Department of Energy (DOE) complex can impact the design and completion of security-by-design (SeBD) for SMRs considering modular construction. The conclusions and recommendations in this document may be applicable to all SMR designs.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2020

The Grey Zone Test Range Integrated Urban Simulation Environment

Sandia National Laboratories is part of the government test and evaluation team for the Defense Advanced Research Projects Agency Collection and Monitoring via Planning for Active Situational Scenarios program. The program is designed to better understand competition in the area between peace and conventional conflict when adversary actions are subtle and difficult to detect. For the purposes of test and evaluation, Sandia conducted a range of activities for the program: creation of the Grey Zone Test Range; design of the data stream for a user experiment conducted with U.S. Indo-Pacific Command; design, implementation, and execution of the formal evaluation; and analysis and summary of the evaluation results. This report details Sandia's activities and provides additional information on the Grey Zone Test Range urban simulation environment developed to evaluate the performer technologies.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗
DOE OSTI2024

Utilization of Synthetic Near-Infrared Spectra via Generative Adversarial Network to Improve Wood Stiffness Prediction

Near-infrared (NIR) spectroscopy is widely used as a nondestructive evaluation (NDE) tool for predicting wood properties. When deploying NIR models, one faces challenges in ensuring representative training data, which large datasets can mitigate but often at a significant cost. Machine learning and deep learning NIR models are at an even greater disadvantage because they typically require higher sample sizes for training. In this study, NIR spectra were collected to predict the modulus of elasticity (MOE) of southern pine lumber (training set = 573 samples, testing set = 145 samples). To account for the limited size of the training data, this study employed a generative adversarial network (GAN) to generate synthetic NIR spectra. The training dataset was fed into a GAN to generate 313, 573, and 1000 synthetic spectra. The original and enhanced datasets were used to train artificial neural networks (ANNs), convolutional neural networks (CNNs), and light gradient boosting machines (LGBMs) for MOE prediction. Overall, results showed that data augmentation using GAN improved the coefficient of determination (R 2 ) by up to 7.02% and reduced the error of predictions by up to 4.29%. ANNs and CNNs benefited more from synthetic spectra than LGBMs, which only yielded slight improvement. All models showed optimal performance when 313 synthetic spectra were added to the original training data; further additions did not improve model performance because the quality of the datapoints generated by GAN beyond a certain threshold is poor, and one of the main reasons for this can be the size of the initial training data fed into the GAN. LGBMs showed superior performances than ANNs and CNNs on both the original and enhanced training datasets, which highlights the significance of selecting an appropriate machine learning or deep learning model for NIR spectral-data analysis. The results highlighted the positive impact of GAN on the predictive performance of models utilizing NIR spectroscopy as an NDE technique and monitoring tool for wood mechanical-property evaluation. Further studies should investigate the impact of the initial size of training data, the optimal number of generated synthetic spectra, and machine learning or deep learning models that could benefit more from data augmentation using GANs.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

Attack Surface of Wind Energy Technologies in the United States

Low cost, reliable electrical energy production from wind relies upon automation and control systems, arguably more so than traditional thermal generation. These same systems, however, can serve as the target of adversaries’ cyber-attacks. Idaho National Laboratory (INL), at the request of the Department of Energy’s (DOE’s) Office of Cybersecurity, Energy Security, and Emergency Response (CESER) and Energy Efficiency and Renewable Energy’s (EERE’s) Wind Energy Technologies Office (WETO), evaluated a generalized wind plant architecture to understand the classes of potential threat actors and the vectors that could enable a cyber-attack. This evaluation explores the attack surface of a representative wind plant, identifying potential methods and vectors that an adversary could leverage to conduct a cyber-attack. Included in this assessment are some recommended mitigations and approaches. Each recommendation requires a full security evaluation, cost/benefit analysis, and risk analysis by each owner and operator.

17 WIND ENERGY↗
DOE OSTI2026

Evaluating lightweight unsupervised online IDS for masquerade attacks in CAN

Vehicular controller area networks (CANs) are susceptible to masquerade attacks by malicious adversaries. In masquerade attacks, adversaries silence a targeted ID and then send malicious frames with forged content at the expected timing of benign frames. As masquerade attacks could seriously harm vehicle functionality and are the stealthiest attacks to detect in CAN, recent work has devoted attention to compare frameworks for detecting masquerade attacks in CAN. However, most existing works report offline evaluations using CAN logs already collected using simulations that do not comply with the domain’s real-time constraints. Here we contribute to advance the state of the art by presenting a comparative evaluation of four different non-deep learning (DL)-based unsupervised online intrusion detection systems (IDS) for masquerade attacks in CAN. Our approach differs from existing comparative evaluations in that we analyze the effect of controlling streaming data conditions in a sliding window setting. In doing so, we use realistic masquerade attacks being replayed from the ROAD dataset. We show that although evaluated IDS are not effective at detecting every attack type, the method that relies on detecting changes in the hierarchical structure of clusters of time series produces the best results at the expense of higher computational overhead. We discuss limitations, open challenges, and how the evaluated methods can be used for practical unsupervised online CAN IDS for masquerade attacks.

Anomaly detection↗
DOE OSTI2023

Testing and Evaluation of Security Response at High-Consequence Nuclear Facilities

High-consequence nuclear facilities are required to implement a physical protection system (PPS) capable of preventing a design basis threat (DBT) adversary from completing radiological sabotage or theft of nuclear material. Response forces are a core component of the PPS. A rigorous testing and evaluation program is required to ensure that the response forces are appropriately trained and equipped and are capable of meeting the regulatory and mission requirements.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗
DOE OSTI2024

Report on the AAPM grand challenge on deep generative modeling for learning medical image statistics

Abstract Background The findings of the 2023 AAPM Grand Challenge on Deep Generative Modeling for Learning Medical Image Statistics are reported in this Special Report. Purpose The goal of this challenge was to promote the development of deep generative models for medical imaging and to emphasize the need for their domain‐relevant assessments via the analysis of relevant image statistics. Methods As part of this Grand Challenge, a common training dataset and an evaluation procedure was developed for benchmarking deep generative models for medical image synthesis. To create the training dataset, an established 3D virtual breast phantom was adapted. The resulting dataset comprised about 108 000 images of size 512 512. For the evaluation of submissions to the Challenge, an ensemble of 10 000 DGM‐generated images from each submission was employed. The evaluation procedure consisted of two stages. In the first stage, a preliminary check for memorization and image quality (via the Fréchet Inception Distance [FID]) was performed. Submissions that passed the first stage were then evaluated for the reproducibility of image statistics corresponding to several feature families including texture, morphology, image moments, fractal statistics, and skeleton statistics. A summary measure in this feature space was employed to rank the submissions. Additional analyses of submissions was performed to assess DGM performance specific to individual feature families, the four classes in the training data, and also to identify various artifacts. Results Fifty‐eight submissions from 12 unique users were received for this Challenge. Out of these 12 submissions, 9 submissions passed the first stage of evaluation and were eligible for ranking. The top‐ranked submission employed a conditional latent diffusion model, whereas the joint runners‐up employed a generative adversarial network, followed by another network for image superresolution. In general, we observed that the overall ranking of the top 9 submissions according to our evaluation method (i) did not match the FID‐based ranking, and (ii) differed with respect to individual feature families. Another important finding from our additional analyses was that different DGMs demonstrated similar kinds of artifacts. Conclusions This Grand Challenge highlighted the need for domain‐specific evaluation to further DGM design as well as deployment. It also demonstrated that the specification of a DGM may differ depending on its intended use.

Radiology, Nuclear Medicine & Medical Imaging↗
DOE OSTI2021

Side-channel Leakage Assessment Metrics: A Case Study of GIFT Block Ciphers

Determination of an adequate level of security and providing subsequent mechanisms to achieve it, is one of the most pressing problems regarding embedded computing devices. While there are some solutions available for resource-rich computer systems, direct application of these solutions to resource-constrained environments are often unfeasible. The fundamental problem for such resource-constrained systems is the fact that current cryptographic algorithms utilize significant energy consumption and storage overhead. Both the cryptographic algorithm and its physical implementation affect the resilience of a cryptosystem against side-channel attacks. A side-channel attack represents a process that exploits leakages in order to extract sensitive information such as the key. This paper focuses on Correlation Power Analysis (CPA) which is side-channel attack based on the power consumption leakage. In 2016 the U.S. Commerce Department’s National Institute of Standards and Technology (NIST) initiated the call for proposals of new cryptographic algorithms to strengthen the cryptographic defense of networked devices against cyberattacks and to protect the data created by those innumerable device. This work evaluates S-boxes used by NIST candidates PICCOLO, GIFT, and PRESENT, as well as several S-box variants that demonstrated sufficient weaknesses against classical cryptanalysis, for a quantitative comparison in terms of resiliency to CPA attack. Three well-known theoretical metrics are evaluated: transparency order (TO and RTO), nonlinearity, and signal-to-noise (SNR) ratio, aiming to characterize the resistance of these S-boxes against adversaries exploiting physical leakages. Experimental results from attacks on an 8- bit XMEGA were obtained via the ChipWhisperer platform and of all the S-boxes evaluated, GIFT64 with a PICCOLO S-box was found to be the most susceptible to CPA. Results showed that variations in TO and RTO were not sufficient to ensure practical CPA resistance and that among S-boxes with equal non-linearity there were no significant differences in the TO and SNR variants.

97 MATHEMATICS AND COMPUTING↗