SEARCH · Engineering Papers

Results for “software quality, testing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9

All sources NASA NTRS DOE OSTI

DOE OSTI2020

Investigating Scientific Data Change with User Research Methods

Scientific datasets are continually expanding and changing due to fluctuations with instruments, quality assessment and quality control processes, and modifications to software pipelines. Datasets include minimal information about these changes or their effects requiring scientists manually assess modifications through a number of labor intensive and ad-hoc steps. The Deduce project is investigating data change to develop metrics, methods, and tools that will help scientists systematically identify and make decisions around data changes. Currently, there is a lack of understanding, and common practices, for identifying and evaluating changes in datasets since systematically measuring and managing data change is under explored in scientific work. We are conducting user research to address this need by exploring scientist's conceptualizations, behaviors, needs, and motivations when dealing with changing datasets. Our user research utilizes multiple methods to produce foundational, generative insights and evaluate research products produced by our team. In this paper, we detail our user research process and outline our findings about data change that emerge from our studies. Our work illustrates how scientific software teams can push beyond just usability testing user interfaces or tools to better probe the underlying ideas they are developing solutions to address.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

DOE OSTI2023

Towards operational atmospheric correction of airborne hyperspectral imaging spectroscopy: Algorithm evaluation, key parameter analysis, and machine learning emulators

Atmospheric correction of airborne hyperspectral imaging spectroscopy (AHIS) to obtain high-quality surface reflectance is the prerequisite for remote sensing applications. Over the last decades, different atmospheric correction methods have been developed based on radiative transfer models (RTMs), however, the relative performances of different algorithms are unclear. Automated operational atmospheric correction methods to process large-volume AHIS data in a high-accurate and high-throughput manner are still lacking. Therefore, this study proposed an operational atmospheric correction pipeline for deriving surface reflectance from AHIS data. To ensure the accuracy and efficiency of the pipeline, we focused on three specific aspects: (1) selecting a suitable RTM for the development of atmospheric lookup tables (LUTs) by comparing the commercial MODerate resolution atmospheric TRANsmission (MODTRAN) and open-sourced Library for Radiative TRANsfer (LibRadTRAN) models, where the widely-used software, Atmospheric/Topographic Correction for Airborne Imagery (ATCOR), was used as benchmarks; (2) identifying key atmospheric correction parameters and determining suitable sources for parameter retrievals including AHIS, Moderate Resolution Imaging Spectroradiometer (MODIS), and AErosol RObotic NETwork (AERONET); and (3) testing the performance of using machine learning emulators to speed up the RTM-based atmospheric correction. Results indicate that (1) atmospheric correction based on MODTRAN LUTs can produce surface reflectance accurately with mean absolute errors < 0.05 and cosine similarities > 0.98 compared to field measurements, which is comparable to the software ATCOR and slightly outperforms the LibRadTRAN LUTs; (2) sobol global sensitivity analysis demonstrates that in the atmospheric correction, visibility and water vapor are two key parameters that can be accurately derived from AHIS in contrast to MODIS or AERONET data; and (3) Random Forest emulators can produce accurate estimations of surface reflectance with mean absolute errors < 0.03 and cosine similarities > 0.98 for higher processing efficiency and determine a suitable set of wavelengths for retrieving atmospheric visibility and water vapor. In conclusion, the proposed atmospheric correction pipeline also improved the four-stream radiative transfer theory for airborne applications by considering adjacent effects from airborne surrounding pixels and can also be applied for atmospheric correction of hyperspectral data from spaceborne missions.

47 OTHER INSTRUMENTATION↗

DOE OSTI2024

guppy i : a code for reducing the storage requirements of cosmological simulations

ABSTRACT As cosmological simulations have grown in size, the permanent storage requirements of their particle data have also grown. Even modest simulations present a major logistical challenge for the groups which run these boxes and researchers without access to high performance computing facilities often need to restrict their analysis to lower quality data. In this paper, we present guppy, a compression algorithm and code base tailored to reduce the sizes of dark matter-only cosmological simulations by approximately an order of magnitude. guppy is a ‘lossy’ algorithm, meaning that it injects a small amount of controlled and uncorrelated noise into particle properties. We perform extensive tests on the impact that this noise has on the internal structure of dark matter haloes, and identify conservative accuracy limits which ensure that compression has no practical impact on single-snapshot halo properties, profiles, and abundances. We also release functional prototype libraries in C, Python, and Go for reading and creating guppy data.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2021

Nondestructive Characterization of Antireflective Coatings on PV Modules

Antireflective coatings (ARCs) are used on the vast majority of solar photovoltaic (PV) modules to increase power production. However, ARC longevity can vary from less than 1 to over 15 years depending on coating quality and deployment conditions. A technique that can quantify ARC degradation nondestructively on commercial modules would be useful both for in-field diagnostics and accelerated aging tests. In this article, we demonstrate that accurate measurements of ARC spectral reflectance can be performed using a modified commercially available integrating-sphere probe. The measurement is fast, accurate, nondestructive, and can be performed outdoors in full-sun conditions. We develop an interferometric model that estimates coating porosity, thickness, and fractional area coverage from the measured reflectance spectrum for a uniform single-layer coating. We demonstrate the measurement outdoors on an active PV installation, identify the presence of an ARC, and estimate the properties of the coating.

14 SOLAR ENERGY↗

DOE OSTI2020

Software-defined network for end-to-end networked science at the exascale

Domain science applications and workflow processes are currently forced to view the network as an opaque infrastructure into which they inject data and hope that it emerges at the destination with an acceptable Quality of Experience. There is little ability for applications to interact with the network to exchange information, negotiate performance parameters, discover expected performance metrics, or receive status/troubleshooting information in real time. The work presented here is motivated by a vision for a new smart network and smart application ecosystem that will provide a more deterministic and interactive environment for domain science workflows. The Software-Defined Network for End-to-end Networked Science at Exascale (SENSE) system includes a model-based architecture, implementation, and deployment which enables automated end- to-end network service instantiation across administrative domains. An intent based interface allows applications to express their high-level service requirements, an intelligent orchestrator and resource control systems allow for custom tailoring of scalability and real-time responsiveness based on individual application and infrastructure operator requirements. This allows the science applications to manage the network as a first-class schedulable resource as is the current practice for instruments, compute, and storage systems. Deployment and experiments on production networks and testbeds have validated SENSE functions and performance. Emulation based testing verified the scalability needed to support research and education infrastructures. Key contributions of this work include an architecture definition, reference implementation, and deployment. This provides the basis for further innovation of smart network services to accelerate scientific discovery in the era of big data, cloud computing, machine learning and artificial intelligence.

47 OTHER INSTRUMENTATION↗

DOE OSTI2020

Software-Defined Network for End-to-end Networked Science at the Exascale

Domain science applications and workflow processes are currently forced to view the network as an opaque infrastructure into which they inject data and hope that it emerges at the destination with an acceptable Quality of Experience. There is little ability for applications to interact with the network to exchange information, negotiate performance parameters, discover expected performance metrics, or receive status/troubleshooting information in real time. The work we presen here is motivated by a vision for a new smart network and smart application ecosystem that will provide a more deterministic and interactive environment for domain science workflows. The Software-Defined Network for End-to-end Networked Science at Exascale (SENSE) system includes a model-based architecture, implementation, and deployment which enables automated end-to-end network service instantiation across administrative domains. An intent based interface allows applications to express their high-level service requirements, an intelligent orchestrator and resource control systems allow for custom tailoring of scalability and real-time responsiveness based on individual application and infrastructure operator requirements. This allows the science applications to manage the network as a first-class schedulable resource as is the current practice for instruments, compute, and storage systems. Deployment and experiments on production networks and testbeds have validated SENSE functions and performance. Emulation based testing verified the scalability needed to support research and education infrastructures. Key contributions of this work include an architecture definition, reference implementation, and deployment. This provides the basis for further innovation of smart network services to accelerate scientific discovery in the era of big data, cloud computing, machine learning and artificial intelligence.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

Fiscal Year 2024 Software Quality Assurance Activities for the ARC Software

The continued goal of the ARC SQA project in the Advanced Reactor Technologies program of DOE is to resolve the QA gaps for the ARC software that limit, or prevent, commercialization of the software for industry users. This project started in earnest in fiscal year 2023 which saw the entire code system moved from a SVN repository to a GitLab repository and an associated software quality assurance plan (SQAP) developed and ratified. Most of the QA gaps in the ARC software were identified in collaboration with industry partners and work begin in fiscal year 2023 and continued in 2024. The primary documentation that is missing includes user manuals, user guides, software verification reports, and code coverage assessments. The SUMMAR manual was completed this fiscal year and work was started on creating manuals for SE2ANL, SE2RCT, and DASSH. Software verification work was carried out for DIF3D and REBUS in a previous program and the current fiscal year saw the completion of software verification reports for GAMSOR, GAMSRC, VARPOW, EvaluateFlux, and SUMMAR. The goal for the next fiscal year is to complete the PERSENT software verification work and begin planning the software verification work for DASSH, SE2ANL, and SE2RCT. The code coverage reports for DIF3D and MC2-3 were completed in the previous fiscal year and the goal is to generate code coverage reports for REBUS, GAMSOR, PERSENT, and DASSH in the coming fiscal year. A considerable amount of effort was spent in the current fiscal year working on the continuous integration capability for automated regression testing in GitLab. The first version of the testing was created in the previous fiscal year and applied to DIF3D and its utility programs. That testing was extended this year to cover GAMSOR, REBUS, and PERSENT. To accomplish this, the first version of the new testing methodology had to be updated to make a single output checking methodology viable for all of the ARC software. This will result in a single document to detail the automated regression testing methodology and minor documents to detail the tolerance settings that have been applied to the output for each ARC code. The previous methodology put into place with SVN would have required a separate document for each ARC code to detail the output checking methodology and the tolerance settings for the output from each code. Because some of our industry partners are providing funds to add new capabilities to the ARC software to meet their needs, all of which must be reviewed and approved by the SQA program funded by this project, a summary of that development work is detailed in this report. Overall progress on resolving the QA gaps has been good this year with the most impactful improvement for our industry partners in capability being the creation of a threaded version of DIF3D-VARIANT that allows the DIF3D, REBUS, and GAMSOR run times to be reduced by a factor of 4-6. The most impactful QA gap that was resolved was the software verification of GAMSRC and VARPOW.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2021

Developing and testing capabilities for simulating cases with heterogeneous land/water surfaces in a novel atmospheric large eddy simulation code

Large eddy simulations (LES) are the primary computational tool used to simulate high Reynolds number three-dimensional turbulent flows. In the context of earth system sciences, particularly atmospheric science, LES are uniquely able to resolve the scales of atmospheric motion that are key for building process-level understanding of boundary layer turbulence, atmosphere-surface interaction, clouds, and cloud-aerosol-chemistry interaction, and are a core limited-area modeling capability. Increasing demands are being placed on LES code bases as growing high performance computing resources allow LES to address a wider range of scientific problems. In addition, LES are emerging as a source of high-quality machine learning training data. These demands necessitate an agile and extensible code base that allows the model to quickly adapt to emergent needs. However, LES have largely relied on legacy Fortran code bases that lack flexibility. A new, Python-based LES capability called Predicting INteractions of Aerosol and Clouds in Large Eddy Simulation (PINACLES) has been developed as part of the Department of Energy’s Earth System Model Development (ESMD) program area’s Enabling Aerosol-cloud interactions at Global convection-permitting scalES (EAGLES) project. PINACLES was developed from the ground up with a philosophy of maximizing scientific throughput, by attempting to optimize for both model throughput and software extensibility. The initial development of PINACLES delivered a state-of-the-art idealized LES capability solving the non-hydrostatic anelastic equations of motion with doubly periodic boundary conditions and idealized homogenous surface boundary conditions. Here we provide a final report on the outcomes of a fiscal year 2021 Seed Laboratory Directed Research Project that extended PINACLES in two key ways. First, PINACLES was coupled to a state-of-the-art land surface model enabling it to simulate spatially inhomogeneous land-atmosphere interactions that are known to control key atmospheric processes. Second, the dynamical core of PINACLES was modified to permit non-periodic boundary conditions. This model enhancement enables simulation of realistic cases with boundary conditions prescribed from atmospheric reanalysis and enables nested simulations conducted on a hierarchy of computational domains with increasing resolution. Together, these extensions to PINACLES make it a formidable modeling capability and expand its potential application to diverse components of DOE’s atmospheric science portfolio.

42 ENGINEERING↗

DOE OSTI2023

Reassessing the MCNP Random Number Generator

Random number generators are integral components to Monte Carlo codes. They provide the pseudorandom number sequence used to actually sample the distributions of interest. As a result, they are one of the most important components to the software. The current recommended MCNP random number generator is a 63-bit linear congruential generator (LCG). This generator is quite fast, but it has some drawbacks. First, it only has a period of 2 63 . Due to the necessarily non-optimal usage of random numbers to ensure parallel reproducibility, this amount is too few to guarantee random number sequences are not reused in all configurations the code runs under. As simulation size increases, users will need to be aware of the limitations of the generator and tune configuration variables to best suit their simulations, or they will need to assume that reuse is not negatively affecting their answers. Neither of these are optimal. Second, small LCGs are fairly weak in bit generation quality, and this can have an unknown impact on the quality of the simulation. This paper is an investigation into whether or not more modern random number generators can supersede the current ones. The goal is to find a generator that is similar or superior in speed to the LCGs, has a state space large enough to make strong guarantees about random number reuse, and passes all modern random number test suites. If such a generator is found, it would eliminate the need for the user to even be aware of the limitations of the random number generator and would simplify the use of the code. This paper will be broken into several parts. Sec. 2 will discuss the evolution of the random number generator within the MCNP code. Sec. 3 will go over what a Monte Carlo code needs from a generator to be reproducible and portable and how the current generator behaves in that light. Sec. 4 goes through how each generator was tested. Finally, Sec. 5 will discuss improvements that could be made to the code.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2023

Advanced Simulation and Computing: ASC FY24 Implementation Plan

The DOE National Nuclear Security Administration (NNSA) Stockpile Stewardship Program (SSP) is an integrated technical program for maintaining the safety, security, and reliability of the U.S. nuclear stockpile. The SSP incorporates nuclear test data, computational modeling and simulation, and experimental facilities to advance understanding of nuclear weapons. The suite of data analyzed comes from activities including previous nuclear tests, stockpile surveillance, experimental research, and development and engineering programs. This integrated national program requires the continued use of experimental facilities and the computational capabilities to support the SSP missions. These component parts, in addition to an appropriately scaled production capability, enable NNSA to support stockpile requirements. The ultimate goal of the SSP, and thus of the Advanced Simulation and Computing (ASC) program, is to ensure that the U.S. maintains a safe, secure, and effective strategic deterrent. The ASC program is a cornerstone of the SSP, providing simulation capabilities and computational resources to support the annual stockpile assessment and certification process, study advanced nuclear weapons design and manufacturing processes, analyze accident scenarios and weapons aging, and provide the tools to enable stockpile Life Extension Programs (LEPs) and the resolution of Significant Finding Investigations (SFIs). This work requires a balance of resources, including technical staff, hardware, simulation software, and computer science solutions. The ASC program focuses on increasing the predictive capabilities in a three-dimensional (3D) simulation environment while maintaining support to the SSP. The Program continues to improve its unique tools for understanding and solving progressively more difficult stockpile problems (sufficient resolution, dimensionality, and scientific details), and quantifying critical margins and uncertainties. Resolving each issue requires increasingly difficult analyses because the aging process has progressively moved the stockpile further from the original test base. While the focus remains on the U.S. nuclear weapons program, where possible, the Program also enables the use of high-performance computing (HPC) and simulation tools to address broader national security needs, such as foreign nuclear weapon assessments and nuclear counterterrorism. The 2022 Nuclear Posture Review (NPR) calls for NNSA to “deliver a modern, adaptive nuclear security enterprise based on an integrated strategy for risk management, production-based resilience, science and technology innovation, and workforce initiatives.” Furthermore, “NNSA will establish a Science and Technology Innovation Initiative to accelerate the integration of science and technology (S&T) throughout its activities.” Executing this strategy necessitates the continued emphasis on developing and sustaining high-quality scientific and engineering staff, as well as supporting computational and experimental capabilities. These components constitute the foundation of the nuclear weapons program. The continued success of the SSP and LEPs is predicated upon the ability to credibly certify the stockpile, without a return to underground nuclear tests (UGTs). Shortly after the nuclear test moratorium entered into force in 1992, the Accelerated Strategic Computing Initiative (ASCI) was established to provide an extensive simulation capability to underpin stockpile certification. While computing and simulation have always been essential to the success of the nuclear weapons program, the program goal of ASCI was to execute NNSA’s vision of using these tools in support of the stockpile stewardship mission. The ASCI program was essential to the successful demonstration of the SSP, providing critical nuclear weapons simulation and modeling capabilities. ASCI officially evolved into the ASC program in fiscal year (FY) 2005, but the mission remains essentially the same: provide the simulation and computational capabilities that underpin the ability to maintain a safe, secure, effective nuclear weapon stockpile, without returning to underground nuclear testing. The capabilities that the ASC program provides at the national laboratories play a vital role in the nuclear security enterprise and are necessary for fulfilling the stockpile stewardship and life extension requirements outlined for NNSA. The Program develops modern simulation tools that provide insights into stockpile aging issues, provide the computational and simulation tools that enable designers and analysts to certify the current stockpile and life-extended nuclear weapons, and inform the decision-making process when any modifications in nuclear warheads or the associated manufacturing processes are deemed necessary. Furthermore, ASC is enhancing the predictive simulation capabilities that are essential to evaluate weapons effects, design experiments, and ensure test readiness. The ASC program continues to improve its unique tools to solve stockpile problems— with a focus on sufficient resolution, dimensionality, and scientific detail—to enable Quantification of Margins and Uncertainties (QMU) and to resolve the increasingly difficult analyses needed for stockpile stewardship. The needs of the Stockpile Management and Production Modernization programs (formerly Directed Stockpile Work) also drive the requirements for simulation and computational resources. These requirements include planned LEPs, stockpile support activities, and mitigation efforts against the potential for technical surprise. All of the weapons within the current stockpile are in some stage of the life extension process. The simulation and computational capabilities are crucial for successful execution of these life extensions and for ensuring NNSA can certify these life-extended weapons without conducting a UGT.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

Multi-Task with Procter and Gamble (CRADA No. NFE-10-02672)

The purpose of this Cooperative Research and Development Agreement (CRADA) between UT-Battelle, LLC (the “Contractor) and Procter & Gamble Company (the “Participant”) is the development of a research partnership to create new tools, tests and analytical methods to improve the performance, safety and/or environmental quality of chemicals, advanced materials, food products and manufacturing processes. The Participant operates in three global business units: Beauty, Health and Well-Being and Household Care. Some of its worldwide products include Head and Shoulders®, Pantene®, Gillette® razors and personal care products, Crest®, Dawn®, Tide®, Bounty®, Duracell® batteries; and Iams® pet food among others. At its core, however, the Participant is a science driven company. It supports one of the most robust industrial research and development (R&D) programs in the world. The Participant uses this rich foundation of science to drive innovation across all of its product lines. But the innovation process is not confined in-house The Participant pursues an “open innovation” policy, seeking partnerships with scientists and researchers in universities and national laboratories where it can contribute its extensive knowledge assets and collaborate to advance scientific understanding. The research under this multi-task CRADA was directed under the following general task areas and, throughout the duration of this CRADA the work statement was modified to match the needs of the Parties and the direction of the research. (1) Software modeling, simulation and development; (2) Manufacturing Technologies; (3) Supply Chain Optimization, (4) Advanced Materials.

36 MATERIALS SCIENCE↗

DOE OSTI2022

A High-Throughput Computing Infrastructure to Generate Custom, Open Community Geothermal Datasets

The most significant challenge facing geothermal research, development, and deployment is a lack of comprehensive datasets describing the geological and economical properties of North America. Automated knowledge base construction, the process of designing algorithms to analyze text and images to programmatically build new datasets, is one possible solution to this problem. The xDD library of full-text scientific articles (https://xdd.wisc.edu) is one of the largest collections of open and controlled-access scientific documents available for knowledge base construction in the world, but it has been underutilized by experts in geothermal research. The xDD development team attributed the lack of engagement by software developers and geothermal researchers to two perceived shortcomings of the system. First, the workflow for obtaining data from xDD for local development and testing of data mining applications was unnecessarily abstruse and required significant manual intervention by xDD systems administrators. Second, although xDD already held articles from a broad cross-section of scientific literature with an emphasis on the geosciences, it did not have an explicit set of geothermal research documents that could serve as the nucleus of a geothermal data mining application. To address these issues, the Automated Data Extraction PlaTform (ADEPT) was proposed to extend the data distribution capabilities of the xDD system. The ADEPT extension added the following four key features to xDD: 1) integration of National Geothermal Data System (NGDS) documents into the xDD library to provide an explicitly geothermally-themed collection; 2) improved RESTful (i.e., https-protocol driven) web services for external partners to access xDD data for machine learning application development; 3) a web platform for end-users and xDD administrators to coordinate the development of data mining applications from the initial step of browsing available documents to the final stage of deploying a production-quality machine learning application on high-throughput computing infrastructure; and 4) the development of demonstration data mining applications to illustrate the new workflow to potential collaborators. A total of 21,674 geothermal documents from NGDS were fully ingested into the xDD library and the associated metadata is publicly available through the xDD web services; furthermore, the ADEPT web platform is now publicly accessible and fully live at https://xdd.wisc.edu/adept/.

15 GEOTHERMAL ENERGY↗

DOE OSTI2025

ON THE EFFECTIVENESS OF LLMS IN UNIT TEST GENERATION FOR STRUCTURED TEXT PROGRAMS

The reliability of industrial automation systems heavily depends on the correctness of Programmable Logic Controller (PLC) programs, which are often written in Structured Text (ST). While Large Language Models (LLMs) have shown promise in automating test generation for mainstream programming languages, their effectiveness for the syntactically strict ST language remains underexplored. This thesis presents a systematic empirical evaluation of three state-of-the-art LLMs—GPT-4o, Gemini 2.5 Pro, and Claude Sonnet 4.5—for generating ST unit tests. We examine three prompting strategies: Natural Language (NL), Code Language (CL), and Chain-of-Thought (CoT), across a curated set of 11 ST function blocks. The quality of the generated tests is assessed using Compilation Success Rate (CSR), Statement Coverage (SC), and Branch Coverage (BC). In the zero-shot setting, Claude Sonnet 4.5 achieves the highest CSR, while Gemini 2.5 Pro consistently delivers the best statement and branch coverage, particularly under CL prompts. By incorporating a one-shot CL prompt, all models exhibit substantial improvements—most notably GPT-4o, whose CSR increases from 45.45% to 90.91%, with substantial gains in both SC and BC. To further contextualize these findings, we compare GPT-4o’s one-shot results with PLCAutoTester, a state-ofthe- art ST unit test generation tool, on an additional benchmark dataset. While LLMgenerated tests approach competitive coverage levels, PLCAutoTester maintains significantly higher and more stable coverage across programs. This study provides the first comprehensive benchmark of modern LLMs for ST unit testing, highlighting their strengths, limitations, and improvements through one-shot prompting, and positioning their performance relative to specialized automated testing tools in industrial automation.

42 ENGINEERING↗

DOE OSTI2022

Verification of the 3-Region Advanced Test Reactor MCNP Model

The verification of the 3-region homogenized fuel Advanced Test Reactor MCNP model. The 3-region model was compared to the 19-plate model found in the 94-CIC report. Flux tallies, energy deposition tallies, and quarter core mesh tallies were used to compare the two models. The 3-region model needed updating in order to make good comparisons between the models. The percent error from the flux and energy deposition tallies data shows that experiment positions inside the flux trap have higher errors than positions outside the fuel ring. The standard deviation data obtained from the mesh tallies shows that the two models agree within two standard deviations throughout the reactor. It is concluded that the model works adequately for what it is used for.

99 GENERAL AND MISCELLANEOUS↗

DOE OSTI2022

Verification of the 3-Region Advanced Test Reactor MCNP Model

99 GENERAL AND MISCELLANEOUS↗

DOE OSTI2021

Phasor-Measurement-Unit-Based Data Analytics Using Digital Twin and PhasorAnalytics Software

A major objective of this project was to apply GE’s commercial machine learning and data analytics toolsets to large-scale, real-world, anonymized Phasor Measurement Unit (PMU) datasets in order to extract signatures, correlated and/or causal factors, and precursor patterns associated with significant power system phenomena. The project had a particular emphasis on extraction of insights relevant to asset health monitoring, real-time load modeling and cybersecurity monitoring. Additionally, the team was directed to undertake a comprehensive data quality analysis for the provided datasets and encouraged to estimate the ‘machine-learning readiness’ of the datasets by documenting any major obstacles to the application of commercial machine learning algorithms. To accomplish the aforementioned objectives, the project team’s work centered around the identification of key event signatures and application of the identified event signatures for event detection and event classification. The industry-validated, semi-supervised machine learning strategy employed for event signature identification involved several major tasks, including data-preprocessing, generation of an overabundance of features, normal data identification, normality modeling, and event signature identification through a methodical, quantitative ranking of features in order of relevance to each studied event type. Throughout the project, data quality issues and mitigation techniques were investigated. In this report, insights are provided regarding the readiness of the provided synchrophasor datasets for application of machine learning and data analytics. The methodologies employed for this technical strategy are summarized in this report. With regards to data preprocessing and feature generation, the provided Training and Test Datasets were ingested into GE’s big data environment. Subsequently, the team applied bad data cleansing and data imputation scripts, event detection scripts, and application programming interfaces (APIs) to the datasets for convenient data access. The project team completed development and validation of dozens of physics-based, statistics-based and transformation-based feature functions used for the extraction of over 60 synchrophasor features. Using a new parallel feature generation technology developed on this project, over 60 features have been rapidly generated for the full two years’ worth of Training and Test Dataset data associated with both the Eastern and Western interconnects. Even accommodating for temporal down-sampling inherent to the feature extraction procedure, this parallel feature generation activity resulted in a massive feature set with a storage requirement approximately equal to that of the raw training dataset itself. With regards to normal data identification and normality modeling, a normality model was built using the feature data extracted from the Training Dataset and iteratively refined subsequent to incremental adjustments and expansions of the Training Dataset feature data. With respect to event characterization and signature identification, an event signature identification pipeline was developed and used in conjunction with the normality model to identify over 15 event signatures for key event categories within the Training Dataset. The identified event signatures were used to characterize hundreds of key events in terms of relative severity, duration, and location of the event. An investigation was undertaken to identify correlated and causal factors involved in transformer events. A separate investigation into temporal trends in ring-down analysis results was undertaken to determine possible associations between system dynamics and various other factors such as loading, season or year. To validate the identified event signatures, additional work was undertaken to develop signature-based anomaly detection and classification tools suitable for convenient application to the synchrophasor datasets. The anomaly detection and classification tools, suitable for online application, were then applied to the entirety of the Eastern Interconnect Training and Test Datasets. Performance of the event detection and classification tools was evaluated upon receipt of the Test Dataset event logs (i.e., the labels for events contained in the Test Dataset), and promising results were obtained despite several challenges (documented herein) associated with application of supervised or semi-supervised machine learning methods to large-scale, anonymized datasets. Finally, the detection and classification tools were used to detect, classify, and characterize thousands of new events not included in the original event logs provided by the DOE within both the Training and Test Datasets.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2020

VADER: A Tool for Criticality Safety Validation

The purpose of criticality safety is to prevent any inadvertent criticality from occurring during the handling or storage of fissile material. Calculations are frequently used to demonstrate that a sufficient subcritical margin exists. Validation is a key aspect of the evaluation process, establishing the suitability, accuracy, and associated uncertainty of the computational method and data to be used for the intended application. The validation process is performed by comparing the results of critical experiments with the calculated results from models of the experiments using the computational method to be validated. Laboratory critical experiments are controlled systems that achieve a k eff of approximately 1 in order to investigate the parameters at which such a critical condition is achieved. The validation parameters that are traditionally applied to safety analysis calculations are the bias and the bias uncertainty . The bias is the deviation of the average k eff of the validation suite from unity. The bias uncertainty accounts for the statistical uncertainty in the bias based on the standard deviation, sample size, and distribution of k eff values of the validation suite. The values of bias and bias uncertainty ensure that the systems predicted to be subcritical by the computational method will indeed be subcritical. The bias and bias uncertainty are often combined to determine an upper subcritical limit (USL) or computational margin that can then be applied to safety analysis calculations. Many methods have been developed by different organizations to calculate the bias and bias uncertainty for various types of criticality analyses. Each of these methods typically requires that the validity of various underpinning statistical assumptions be confirmed to demonstrate that the method is appropriate for the analysis of a given validation suite. An example of the validation decision making flow is shown in Fig.1. As shown in Fig. 1, the analyst performing the validation fits a trend line to the data and performs a test to determine if the trend was a statistically better representation of the data than if it were treated as an uncorrelated sample. If the trend line is a better representation of the data, then the analyst uses any one of a number of trending techniques to determine the bias and bias uncertainty. If a trend is not an appropriate representation of the data, then the analyst proceeds to perform a normality assessment for the data. If the normal assumption can be shown to be acceptable, then the analyst calculates the bias and bias uncertainty with the parametric technique. If the assumption of normality cannot be justified, then the nonparametric technique is used. Once the decision flow has been followed and the appropriate technique has been selected, the bias and bias uncertainty is typically combined with an administrative margin to determine a USL below which calculated values of k eff for safety analysis models can be considered subcritical. The calculations used in each decision are often performed with spreadsheets or with small programs available at various sites performing criticality analyses. Expertise in understanding and interpreting the results must be maintained to perform these calculations. This can often be an error-prone process. Oak Ridge National Laboratory (ORNL) is currently developing the Validation and Data Evaluation Resource (VADER) to simplify and automate the criticality safety validation process and to provide a software quality assurance pedigree to the calculational methods used. This paper discusses the use of the Fulcrum user interface with VADER, the anticipated initial capabilities of VADER to perform validation analyses, and the output from the code.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2024

Moving toward automated µFTIR spectra matching for microplastic identification: addressing false identifications and improving accuracy

Abstract Infrared spectroscopy is a widely used tool for studying microplastics and identifying microparticles. Researchers rely on spectral libraries to differentiate between synthetic and natural materials. Unfortunately, spectral library matching is not perfect, and best practices require researchers to use time consuming, manual peak matching to assess spectral matches. Moving toward automated matching requires increased confidence in the matching process. Using spectra matching software may increase the efficiency of particle identification, however some matching strategies may confuse natural materials such as cotton, silk, and plant matter with common classes of synthetics such as polyesters and polyamides. In this experiment, we prepared 22 pristine sample materials from natural and synthetic sources and measured micro-Fourier transform infrared (µFTIR) spectra in transmission mode for each sample using a Thermo Nicolet iN10 MX instrument. The collected spectra were then input into two spectral library matching systems (Omnic Picta and Open Specy), using a total of five identification routines. Next, we placed a subset of four pristine microplastic materials in a biologically active river system for two weeks to simulate environmental samples. These simulated environmental samples were processed using 10% hydrogen peroxide for 24 h to remove organic contamination and then identified using the strongest performing library. We found that libraries with fewer sample spectra produced lower correlation matches and that using derivative correction greatly reduced the number of inaccuracies in identifying materials as either natural or synthetic. We also found that environmental fouling reduced the correlation value of library matches when compared to pristine particles, however the effect was not consistent across the four materials tested. Overall, we found that the accuracy of automated library matching in the tested systems and processing routines varied from 64.1 to 98.0% for distinguishing between natural and synthetic materials, and that a high Hit Quality Index (HQI) did not always correlate with accuracy. These results are important for the microplastic field, demonstrating a need to rigorously test spectral libraries and processing routines with known materials to ensure identification accuracy.

Kozloski, Rachel↗