Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “system validation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Post-Event Fault Identification with Machine Learning for Protection System Validation

Power system protection devices have transitioned over the past few decades from mechanical to analog devices, then to solid state and finally digital. Relays and their associated critical network of equipment have significantly increased in complexity. Even internally, relays have gained significant intricacy, with relatively simple overcurrent or differential functions now being assisted by a myriad of other functions. This is necessary as the grid becomes more complex, but it brings increased difficulty in monitoring and upkeep. Misoperation caused by accidental improper relay settings or deliberate malicious actions is a constant challenge faced by all utilities. These improper settings can be difficult to identify and may require exhaustive post-mortem analysis, typically after a major outage event has already occurred. A mechanism is needed for monitoring the behavior of protection systems to validate that their performance falls within expectations. Relays that fail to isolate a fault or trip when there is no system disturbance can be flagged for settings review in situations where this behavior may not have been noticed due to manual restoration or backup protection operations. This work presents a concept for a machine learning (ML) system capable of validating the performance of protection systems by identifying fault events and characterizing protection system responses based solely on available current and voltage measurements. As a first step in its development, an experimental dataset is generated, and a random forest model is implemented with high accuracy in distinguishing four power system scenarios.

24 - POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2021

Key Constructs for Reasonable Confidence in System Validation Programs

This paper presents four key constructs developed in preparing IEEE Std P2411TM, Human Factors Engineering Guide for the Validation of System Designs and Integrated Systems Operations at Nuclear Facilities for submission. These constructs reflect judgments that responsible parties make in order to bring validation to closure. Such judgments weigh competing concerns and cost-benefits for stakeholders. Improved processes and methods may support such judgments but cannot alone eliminate uncertainty. Stating and clarifying these constructs aims to reduce uncertainty in the validation process, to better prepare the implementers and reviewers of future validation programs, both in the nuclear industry and beyond. Reasonable Confidence – A proof standard, as used to reach conclusions in a legal case. Legal proof standards are bounded between “preponderance of evidence” and “beyond reasonable doubt”. Reasonable confidence is comparable to an intermediate legal proof standard of “clear and convincing evidence.” Representative Test Set – A collection of scenarios of sufficient variety to represent the anticipated range of operational conditions, events, evolutions, and activities for validating the system(s) under test. Dispositive vs. Diagnostic Criteria – Relevant performance criteria are placed in one of two categories. Dispositive criteria are assessed, without exception, to determine whether a validation test passes or fails overall. Diagnostic criteria are assessed, subject to justified exception, to evaluate the quality or degree of some particular aspect of performance. Repeatability – Consistency in achieving passing results is to be demonstrated for each scenario in the representative test set; thus, the minimum number of formal repetitions for each scenario is two.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2026

Protection System Validation Using Post-Event Anomaly Classification with Machine Learning

Power system protection devices have transitioned over the past few decades from mechanical to analog devices, then to solid state and finally digital. Relays and their associated critical network of equipment have significantly increased in complexity. Even internally, relays have gained significant intricacy, with relatively simple overcurrent or differential functions now being assisted by a myriad of other functions. This is necessary as the grid becomes more complex, but it brings increased difficulty in monitoring and upkeep. Misoperation caused by improper relay settings or malicious actions is a constant challenge faced by all utilities. These improper settings can be difficult to identify and may require exhaustive post-mortem analysis, typically after a major outage event has already occurred. A mechanism is needed for monitoring the behavior of protection systems to validate that they act and perform as expected. This work presents a concept for a machine learning (ML) system capable of validating the performance of protection systems by classifying anomalous events and characterizing protection system responses based solely on available current and voltage measurements. As a first step in its development, an experimental dataset is generated, and a random forest model is implemented with high accuracy in distinguishing four power system scenarios.

24 - POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2021

System Validation Platform Working Directories

A suite of software tools used to interface with power electronic equipment for laboratory testing and device certification. They are typically used as equipment drivers with using the System Validation Platform (SVP) graphical user interface (GUI). Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525. SAND2021-1185 O

Johnson, JayT↗
DOE OSTI2026

Protection System Validation with Machine Learning Anomaly Classification

A poster for the Early Career Poster Session. Power system protection devices have transitioned over the past few decades from mechanical to analog devices, then to solid state and finally digital. Relays and their associated critical network of equipment have significantly increased in complexity. Even internally, relays have gained significant intricacy, with relatively simple overcurrent or differential functions now being assisted by a myriad of other functions. This is necessary as the grid becomes more complex, but it brings increased difficulty in monitoring and upkeep. Misoperation caused by improper relay settings or malicious actions is a constant challenge faced by all utilities. These improper settings can be difficult to identify and may require exhaustive post-mortem analysis, typically after a major outage event has already occurred. A mechanism is needed for monitoring the behavior of protection systems to validate that they act and perform as expected. This work presents a concept for a machine learning (ML) system capable of validating the performance of protection systems by classifying anomalous events and characterizing protection system responses based solely on available current and voltage measurements. As a first step in its development, an experimental dataset is generated, and a random forest model is implemented with high accuracy in distinguishing four power system scenarios.

24 - POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2026

CMS Endcap Timing Layer: System Validation and Assembly

The High-Luminosity Large Hadron Collider (HL-LHC) will operate at unprecedented luminosities, resulting in up to 200 simultaneous interactions per bunch crossing. To mitigate the resulting pileup challenges, the CMS experiment is implementing the MIP Timing Detector (MTD), designed to provide precise timing information with a resolution of 30 – 40 picoseconds per track. The MTD consists of the Barrel Timing Layer (BTL) and the Endcap Timing Layer (ETL), each optimized for different regions of the detector. The ETL, comprising two double-sided disks, utilizes Low-Gain Avalanche Diode (LGAD) sensors coupled with the Endcap Timing Readout Chip (ETROC) to achieve high-precision timing measurements in the forward region. Significant progress has been made toward the realization of the ETL through extensive system-level validation of the ETROC readout chain and the development of scalable module assembly procedures. This presentation will provide a comprehensive overview of the ETL and highlight recent advances in system validation and module assembly, emphasizing their roles in ensuring the detector readiness for HL-LHC operation.

Lee, Dongyub [Kyungpook Natl. U.] (ORCID:000000034↗
DOE OSTI2023

Melt pool instability detection using coaxial photodiode system validated by in-situ X-ray imaging

A co-axial photodiode monitoring system with high temporal resolution has been integrated into a proven test bench enabling synchronized side-view high speed X-ray imaging of melt pool dynamics and top-view spectral emission characterization of the melt pool. Here, this setup enables direct observation of melt pool phenomena and correlation between the two monitoring systems which can be directly scaled to commercial systems. The work demonstrates a 92% detection rate in keyhole collapse phenomena related to defect generation in SLM. Furthermore, the impact of gas flow on monitoring signals is studied to understand the fundamental importance of gas flow in commercial systems.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Systems and methods for enhanced power system model validation

A system for enhanced power system model validation is provided. The system includes a computing device including at least one processor in communication with at least one memory device. The at least one processor is programmed to store a plurality of models for a plurality of devices and a plurality of input files associated with the plurality of models, receive, from a user, a selection of model of the plurality of models to simulate, retrieve one or more input files of the plurality of input files, perform a model validity check on the selected model, if the selected model passed the model validity check, perform a model calibration on the selected model, and if the selected model passed the model calibration, perform a post evaluation on the selected model.

Wang, Honggang↗
DOE OSTI2021

Modeling heat transport processes in enhanced geothermal systems: Validation study from EGS Collab Experiment 1

Heat recovery from enhanced geothermal systems (EGS) is a complex process involving heat transport in both fracture networks and rock formations. A comprehensive understanding of and the ability to model the underlying heat transport mechanisms is important for the success of EGS but remains challenging in practice due to the generally insufficient characterization of EGS reservoirs. In the present study, we analyze an extensively monitored intermediate-scale EGS field experiment performed in a well-characterized testbed. The high-resolution, high-quality measurements from the field experiment enable the development of a high-fidelity model incorporating a well-constrained fracture network. Based on the field experiment, we investigate the complex heat transport processes in an EGS-relevant environment and validate the capability of a numerical approach in simulating these inherently coupled heat transport processes. A series of numerical simulations were performed to study the effects of different heat transport mechanisms, including thermal convection with fracture flow, thermal conduction in rock formations, and the Joule-Thomson effect. The agreement of thermal responses between field measurements and simulation results indicates that our numerical approach can appropriately model the heat transport processes pertaining to heat recovery from EGS reservoirs.

Wu, Hui↗
DOE OSTI2023

Integrated behavior-based infrastructure command validation

A cybersecurity infrastructure command validation system is provided herein for validating asset commands issued within an infrastructure network. The cybersecurity infrastructure command validation system can be integrated into an infrastructure network to monitor and validate infrastructure asset commands in real-time or while the infrastructure network is active. The cybersecurity infrastructure command validation system can receive or intercept commands issued by asset controllers. The cybersecurity infrastructure command validation system can validate the commands based on a command validation model. The command validation model can represent normal operating behavior of the infrastructure network. The cybersecurity infrastructure command validation system can provide valid commands to the intended infrastructure asset, or can reject invalid commands. The cybersecurity infrastructure command validation system can store validation results for use in updating the command validation model. The cybersecurity infrastructure command validation system can flag or otherwise warn the infrastructure network or administrators of invalid commands.

Akyol, Bora A.↗
DOE OSTI2026

Validating Protection System Behavior with Machine Learning in a Master State Overseer

As power system protection devices continue the widespread transition from analog to digital, they become increasingly intricate. The internal functions and communication between critical grid components must now be significantly more complex to keep up with the demands of the modern smart grid. This brings increased difficulty in maintenance and monitoring, making it harder to identify potential misoperation, power anomalies, and cyber threats. Such issues are often only pinpointed after an exhaustive and costly post-mortem analysis, when a major outage or damage has already occurred. A solution is needed for validating protection systems as they operate, independently evaluating grid state and confirming whether the protection system is behaving accordingly. As opposed to incident response, this acts as a constant verification mechanism that raises a flag when subtler issues are noticed, catching them earlier and preventing larger incidents. This work presents the implementation of such a system, expanding on the prototype developed by the authors in a previous paper. This is accomplished with a machine learning (ML) system capable of validating the performance of protection systems by classifying anomalous events and characterizing protection system responses based solely on available current and voltage measurements. Additionally, this system is contextualized within a larger, modular Master State awareness Overseer (MSO) framework, responsible for monitoring, analyzing, and managing an electric grid.

24 - POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2026

Dynamic modeling of heat pipe integrated thermal battery latent heat storage system experiment validation

A heat pipe integrated thermal battery system has been constructed to investigate a high-temperature latent heat thermal energy storage technology that takes advantage of near isothermal operation of latent heat storage and heat pipes to potentially enable high-energy isothermal heat storage. A dynamic model constructed in Modelica has been validated, showing errors between 2.5 °C–39.7 °C across 10-h to 47-h simulations against experiment results, showing good prediction capability of experiment output, especially against phase change time. Model calibrations showing vessel heat-up capability of 3 kW and heat pipes combining to provide 600 W each during experiment operation validate experiment circumstances including reduced material loading and reduced power capability. The experiment configuration uses an Al-Mg-Zn eutectic metal as the storage material, heated via heat tape wrapped around the vessel and guide tubes to bring the system to operation range (>400 °C) and to simulate charging heat exchange, respectively, with heat rejection occurring through the surfaces of the material and facilitated via guide tubes with less insulation wrapping. The model is available in the open-source repository HYBRID on Github.

25 - ENERGY STORAGE↗
DOE OSTI2021

Verification and Validation of Systems in which AI is a Key Element

Many systems are being considered in which artificial intelligence (AI) will be a key element. Failure of an AI element can lead to system failure, hence the need for AI verification and validation (V&V). This article addresses V&V of such a system, focusing on the issues created by characteristics of AI that make V&V challenging. The element(s) containing AI capabilities is treated as a subsystem and V&V is conducted on that subsystem and its interfaces with other elements of the system under study, just as V&V would be conducted on other subsystems. That is, the high-level definitions of verification and of validation do not change for systems containing one or more AI elements. However, AI V&V challenges require approaches and solutions beyond those for conventional or traditional (those without AI elements) systems. This article provides an overview of how machine learning components/subsystems “fit” in the systems engineering framework (Section 1), identifies characteristics of AI subsystems that create challenges in their V&V (Section 2), describes those challenges (Section 3) and provides some potential solutions (Section 4).

Pullum, Laura↗
DOE OSTI2021

Foundational Open Source Solar System Modeling Through Improvement and Validation of the System Advisor Model and PVWatts (FY19-FY21 Final Technical Report)

Accelerating intelligent deployment of solar energy technologies demands accurate system modeling every step of the way. From project development to policy research, grid integration studies to development of novel technologies, industry and researchers alike need validated, transparent, easy-to-use, extensible, cutting-edge, and accurate models of both the performance and financing of solar systems. The System Advisor Model (SAM) and PVWatts tools provide a platform to fill that need. The overarching goal of this set of software tools is to enable accurate PV system modeling across the industry, and our usage metrics indicate that we continue to succeed in that endeavor, with a user starting SAM every 2 minutes globally, and over 17 million PVWatts hits per month. This project leveraged DOEs past investment in the SAM and PVWatts platforms to continue to provide valuable and extensible PV, battery, and financial modeling resources to the larger solar community. We pursued multiple avenues in parallel: software maintenance and technical support that are foundational to the continued usability of the SAM and PVWatts platforms; platform and PV model improvements and stakeholder engagement activities that are core to the continued relevance of the platforms; and open source activities to foster the continued creation of a vibrant open-source community around the SAM and PVWatts tools, which opens up exciting new opportunities for industry interaction.

14 SOLAR ENERGY↗