Establishing a Framework and Testbed for Evaluating and Infusing Software Assurance Tools
No abstract available
software assurance tools↗
SEARCH · Engineering Papers
Results for “Evaluation Framework”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Establishing a Framework and Testbed for Evaluating and Infusing Software Assurance Tools
No abstract available
tool evaluation↗
Resiliency in Future Cislunar Space Architectures
This work introduces and explores the concept of resiliency as it relates to future cislunar space architectures by 1) citing examples of its growing demand across government; 2) describing potential characteristics of resilient systems; 3) introducing a framework for evaluating the linkages between resilient capabilities and visions for future cislunar architectures; and 4) exercising the framework to identify and evaluate resiliency-enabling technical capabilities for cislunar space architectures. We assert that resiliency can emerge from a layered approach of deliberately chosen capabilities with overlap and flexibility that, in aggerate, result in a resilient system. The challenge is to identify capabilities that contribute to resiliency and to accurately characterize their value. Resiliency is discussed through the lens of future architecture planning, outlining how the National Aeronautics and Space Administration (NASA) can benefit from a shift in approach when transitioning focus to the cislunar environment.
Jason Hay↗
Complementary Roles of LEO and GEO IR Sounders in NWP Evaluated using an OSSE Framework
The United States is currently preparing for the next-generation weather satellite program. Building off the multi-decadal successes of the Low-Earth Orbit (LEO) infrared (IR) sounder program, NOAA and NASA are planning for the Geostationary eXtended Observations (GeoXO) Sounder (GXS) that will join international counterparts to form a global ring from geosynchronous orbit (GEO). To address questions about the complementarities and roles of these two platform types in the context of numerical weather prediction (NWP) and forecast accuracy, NASA’s Global Modeling and Assimilation Office (GMAO) Observing System Simulation Experiment (OSSE) framework was used to test their impact individually and in concert. Analyses, forecasts, and forecast sensitivity-based observations impact (FSOI) are examined. Overall, each has a role in forecast error reduction with the most beneficial impacts occurring when both LEO and GEO sounders are used together to address the needs of global skill and targeted regional weather phenomena.
Erica Mcgrath-Spangler↗
A common task framework for testing and evaluation at the Space Domain Awareness Tools, Applications, and Processing Lab
Explore the source record for details and available documents.
Astronomy and AstroPhysics↗
Benchmarking machine learning strategies for phase-field problems
Abstract We present a comprehensive benchmarking framework for evaluating machine-learning approaches applied to phase-field problems. This framework focuses on four key analysis areas crucial for assessing the performance of such approaches in a systematic and structured way. Firstly, interpolation tasks are examined to identify trends in prediction accuracy and accumulation of error over simulation time. Secondly, extrapolation tasks are also evaluated according to the same metrics. Thirdly, the relationship between model performance and data requirements is investigated to understand the impact on predictions and robustness of these approaches. Finally, systematic errors are analyzed to identify specific events or inadvertent rare events triggering high errors. Quantitative metrics evaluating the local and global description of the microstructure evolution, along with other scalar metrics representative of phase-field problems, are used across these four analysis areas. This benchmarking framework provides a path to evaluate the effectiveness and limitations of machine-learning strategies applied to phase-field problems, ultimately facilitating their practical application.
36 MATERIALS SCIENCE↗
Evaluating Biosignatures for Life Detection
The goal here is to develop a conceptual framework for evaluating the ability of different biosignatures to provide evidence for the presence of life in planned missions or observational studies. The focus is on intrinsic characteristics of biosignatures in a given space environment rather than on their detection, which depends on the current state of technology. Suitable evaluation procedures are based on an extensive body of previous work on related problems in which different options are evaluated to make decisions in business, engineering, medical fields and social or political arena. In these cases, three approaches have proven to be particularly useful. Two of them, signal detection theory (SDT) and Bayesian hypothesis testing are based on probabilities. The third approach, multi-attribute utility (MAU) is based on utility theory.
Pohorille, Andrew↗
Land Surface Verification Toolkit (LVT) - A Generalized Framework for Land Surface Model Evaluation
Model evaluation and verification are key in improving the usage and applicability of simulation models for real-world applications. In this article, the development and capabilities of a formal system for land surface model evaluation called the Land surface Verification Toolkit (LVT) is described. LVT is designed to provide an integrated environment for systematic land model evaluation and facilitates a range of verification approaches and analysis capabilities. LVT operates across multiple temporal and spatial scales and employs a large suite of in-situ, remotely sensed and other model and reanalysis datasets in their native formats. In addition to the traditional accuracy-based measures, LVT also includes uncertainty and ensemble diagnostics, information theory measures, spatial similarity metrics and scale decomposition techniques that provide novel ways for performing diagnostic model evaluations. Though LVT was originally designed to support the land surface modeling and data assimilation framework known as the Land Information System (LIS), it also supports hydrological data products from other, non-LIS environments. In addition, the analysis of diagnostics from various computational subsystems of LIS including data assimilation, optimization and uncertainty estimation are supported within LVT. Together, LIS and LVT provide a robust end-to-end environment for enabling the concepts of model data fusion for hydrological applications. The evolving capabilities of LVT framework are expected to facilitate rapid model evaluation efforts and aid the definition and refinement of formal evaluation procedures for the land surface modeling community.
Kumar, Sujay V.↗
A Hybrid Anomaly Detection Approach for Obfuscated Malware
With the rapid evolution of malicious software, cyber threats have become increasingly sophisticated, employing advanced obfuscation techniques to evade traditional detection methods. This study presents a hybrid anomaly detection approach applied to obfuscated malware. Even though there is a large body of research in this field, existing malware detection techniques have some drawbacks, such as requiring large amounts of data, trustworthiness (imprecise results) of algorithms, and advanced obfuscation. To overcome these challenges, there is a need to employ solid and efficient techniques for malware detection. This paper proposes a hybrid approach, combining an autoencoder with traditional machine-learning methods to create an efficient malware detection framework. We used the malware memory dataset (MalMemAnalysis-2022) to evaluate this framework. The results indicate that our proposed approach can detect obfuscated malware when a deep autoencoder used for feature learning is combined with logistic regression, and it is extremely fast with an Accuracy, Detection Rate (DR), Matthew Correlation Coefficient(MCC), and Statistical Parity Difference
malware detection, Hybrid Anomly Detection, Obfusc↗
Investigation of the applicability of using the triple redundant hydrogen sensor for methane sensing
Application specifications for the methane sensor were assembled and design guidelines, development goals and evaluation criteria were formulated. This was done to provide a framework to evaluate sensor performance and any design adjustments to the preprototype sensor that could be required to provide methane sensitivity. Good response to hydrogen was experimentally established for four hydrogen sensor elements to be later evaluated for methane response. Prior results were assembled and analyzed for other prototype hydrogen sensor performance parameters to form a comparison base. The four sensor elements previously shown to have good hydrogen response were experimentally evaluated for methane response in 2.5% methane-in-air. No response was obtained for any of the elements, despite the high methane concentration used (50% of the Lower Flammability Limit). It was concluded that the preprototype sensing elements were insensitive to methane and were hydrogen specific. Alternative sensor operating conditions and hardware design changes were considered to provide methane sensitivity to the preprototype sensor, including a variety of different methane sensing techniques. Minor changes to the existing sensor elements, sensor geometry and operating conditions will not make the preprototype hydrogen sensor respond to methane. New sensor elements that will provide methane and hydrogen sensitivity require replacement of the existing thermistor type elements. Some hydrogen sensing characteristics of the modified sensor will be compromised (larger in situ calibration gas volume and H2 nonspecificity). The preprototype hydrogen sensor should be retained for hydrogen monitoring and a separate methane sensor should be developed.
Lantz, J. B.↗
Extrapolation Domains for Aggregating Environmental Outcomes from Local to Regional Levels
Billions of dollars are invested every year to run field experiments to quantify the response of crops to new technologies and associated environmental outcomes. Given the diversity of environments where crop production occurs, conducting these research studies without a robust framework for research site selection and upscaling results to larger spatial scales is inefficient. In this project, researchers from the University of Nebraska-Lincoln used their unique Technology Extrapolation Domains framework (TED) as a tool to guide the selection of experimental sites and as basis for aggregating and validating environmental outcomes from local to regional levels. In the present project, we first used the TED framework to evaluate the current distribution and area coverage of the current SMARTFARM sites. The purpose was to showcase how it is possible to use the TED framework to guide site selection and extrapolate results over space. Additionally, the original TED framework was expanded to account for other factors influencing environmental outcomes by inclusion of three additional variables: soil organic matter, soil texture, and topographic wetness index. The resulting expanded framework (TED-E) was validated using nitrogen (N) losses from corn in the United States as a case study. To do so, we used N balance as a proxy to N losses and we evaluated the capacity of the framework to explain variation in N balance across fields and across countries. We found that the TED-E has substantially higher predictive power than the original TED framework to explain spatial variation in N balance. However, improvements in predictive power with the TED-E tool come at the expense of a higher number of TEDs needed to achieve a given crop area coverage compared with the original TED framework. We conclude that the new TED-E framework can help aggregate and extrapolate environmental outcomes from research sites to regional levels and improve the visualization of their spatial patterns across the United States. An online version of the framework is available at: https://www.toolted.org/
54 ENVIRONMENTAL SCIENCES↗
A relative value framework: Why do different electricity system technologies have different economic value?
Not Available
Mowers, Matthew (ORCID:0000000198789588)↗
Advances in Uncertainty and Sensitivity Analysis Methods and Applications in GDSA Framework.
The Spent Fuel and Waste Science and Technology (SFWST) Campaign of the U.S. Department of Energy (DOE) Office of Nuclear Energy (NE), Office of Fuel Cycle Technology (FCT) is conducting research and development (R&D) on geologic disposal of spent nuclear fuel (SNF) and high-level nuclear waste (HLW). Two high priorities for SFWST disposal R&D are design concept development and disposal system modeling. These priorities are directly addressed in the SFWST ''Geologic Disposal Safety Assessment'' (GDSA) control account, which is charged with developing a geologic repository system modeling and analysis capability, and the associated software, ''GDSA Framework'', for evaluating disposal system performance for nuclear waste in geologic media. ''GDSA Framework'' is supported by SFWST Campaign and its predecessor the Used Fuel Disposition (UFD) campaign. This report fulfills the GDSA Uncertainty and Sensitivity Analysis Methods work package (SF-20SN01030403) level 3 milestone — ''Advances in Uncertainty and Sensitivity Analysis Methods and Applications in GDSA Framework'' (M3SF-20SN010304032). It presents high level objectives and strategy for development of uncertainty and sensitivity analysis tools, demonstrates uncertainty quantification (UQ) and sensitivity analysis (SA) tools in GDSA Framework in FY20, and describes additional UQ/SA tools whose future implementation would enhance the UQ/SA capability of ''GDSA Framework''. This work was closely coordinated with the other Sandia National Laboratory GDSA work packages: the GDSA Framework Development work package (SF- 2051\101030404), the GDSA Repository Systems Analysis work package (SF-2051\101030405), and the GDSA PFLOTRAN Development work package (SF-20SN01030406). This report builds on developments reported in previous ''GDSA Framework'' milestones, particularly M2SF- 19SNO1030403.
11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
Uncertainty and Sensitivity Analysis Methods and Applications in the GDSA Framework (FY2021)
The Spent Fuel and Waste Science and Technology (SFWST) Campaign of the U.S. Department of Energy (DOE) Office of Nuclear Energy (NE), Office of Fuel Cycle Technology (FCT) is conducting research and development (R&D) on geologic disposal of spent nuclear fuel (SNF) and high-level nuclear waste (HLW). Two high priorities for SFWST disposal R&D are design concept development and disposal system modeling. These priorities are directly addressed in the SFWST Geologic Disposal Safety Assessment (GDSA) control account, which is charged with developing a geologic repository system modeling and analysis capability, and the associated software, GDSA Framework, for evaluating disposal system performance for nuclear waste in geologic media. GDSA Framework is supported by SFWST Campaign and its predecessor the Used Fuel Disposition (UFD) campaign. This report fulfills the GDSA Uncertainty and Sensitivity Analysis Methods work package (SF-21SN01030404) level 3 milestone, Uncertainty and Sensitivity Analysis Methods and Applications in GDSA Framework (FY2021) (M3SF-21SN010304042). It presents high level objectives and strategy for development of uncertainty and sensitivity analysis tools, demonstrates uncertainty quantification (UQ) and sensitivity analysis (SA) tools in GDSA Framework in FY21, and describes additional UQ/SA tools whose future implementation would enhance the UQ/SA capability of GDSA Framework. This work was closely coordinated with the other Sandia National Laboratory GDSA work packages: the GDSA Framework Development work package (SF-21SN01030405), the GDSA Repository Systems Analysis work package (SF-21SN01030406), and the GDSA PFLOTRAN Development work package (SF-21SN01030407). This report builds on developments reported in previous GDSA Framework milestones, particularly M3SF 20SN010304032.
12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
Uncertainty and Sensitivity Analysis Methods and Applications in the GDSA Framework (FY2022)
The Spent Fuel and Waste Science and Technology (SFWST) Campaign of the U.S. Department of Energy (DOE) Office of Nuclear Energy (NE), Office of Fuel Cycle Technology (FCT) is conducting research and development (R&D) on geologic disposal of spent nuclear fuel (SNF) and high-level nuclear waste (HLW). Two high priorities for SFWST disposal R&D are design concept development and disposal system modeling. These priorities are directly addressed in the SFWST Geologic Disposal Safety Assessment (GDSA) control account, which is charged with developing a geologic repository system modeling and analysis capability, and the associated software, GDSA Framework, for evaluating disposal system performance for nuclear waste in geologic media. GDSA Framework is supported by SFWST Campaign and its predecessor the Used Fuel Disposition (UFD) campaign.
12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
Regime-Based Evaluation of Cloudiness in CMIP5 Models
The concept of Cloud Regimes (CRs) is used to develop a framework for evaluating the cloudiness of 12 fifth Coupled Model Intercomparison Project (CMIP5) models. Reference CRs come from existing global International Satellite Cloud Climatology Project (ISCCP) weather states. The evaluation is made possible by the implementation in several CMIP5 models of the ISCCP simulator generating for each gridcell daily joint histograms of cloud optical thickness and cloud top pressure. Model performance is assessed with several metrics such as CR global cloud fraction (CF), CR relative frequency of occurrence (RFO), their product (long-term average total cloud amount [TCA]), cross-correlations of CR RFO maps, and a metric of resemblance between model and ISCCP CRs. In terms of CR global RFO, arguably the most fundamental metric, the models perform unsatisfactorily overall, except for CRs representing thick storm clouds. Because model CR CF is internally constrained by our method, RFO discrepancies yield also substantial TCA errors. Our findings support previous studies showing that CMIP5 models underestimate cloudiness. The multi-model mean performs well in matching observed RFO maps for many CRs, but is not the best for this or other metrics. When overall performance across all CRs is assessed, some models, despite their shortcomings, apparently outperform Moderate Resolution Imaging Spectroradiometer (MODIS) cloud observations evaluated against ISCCP as if they were another model output. Lastly, cloud simulation performance is contrasted with each model's equilibrium climate sensitivity (ECS) in order to gain insight on whether good cloud simulation pairs with particular values of this parameter.
cloud climatology↗
Error-controlled Progressive Retrieval of Scientific Data under Derivable Quantities of Interest
The unprecedented amount of scientific data has introduced heavy pressure on the current data storage and transmission systems. Progressive compression has been proposed to mitigate this problem, which offers data access with on-demand precision. However, existing approaches only consider precision control on primary data, leaving uncertainties on the quantities of interest (QoIs) derived from it. In this work, we present a progressive data retrieval framework with guaranteed error control on derivable QoIs. Our contributions are three-fold. (1) We carefully derive the theories to strictly control QoI errors during progressive retrieval. Our theory is generic and can be applied to any QoIs that can be composited by the basis of derivable QoIs proved in the paper. (2) We design and develop a generic progressive retrieval framework based on the proposed theories, and optimize it by exploring feasible progressive representations. (3) We evaluate our framework using five real-world datasets with a diverse set of QoIs. Experiments demonstrate that our framework can faithfully respect any user-specified QoI error bounds in the evaluated applications. This leads to over 2.02× performance gain in data transfer tasks compared to transferring the primary data while guaranteeing a QoI error that is less than 1E-5.
Wu, Xuan↗
Energy Storage Impacts in Resilience Hubs and Other Critical Infrastructure: An Assessment Guide for Developers and Practitioners
Battery energy storage systems (BESS) deployed behind the meter at resilience hubs and other critical infrastructure can provide economic and operational value during normal operations—such as lower and more predictable energy costs—as well as resilience and security benefits during power disruptions by maintaining essential services. However, existing evaluation approaches tend to focus narrowly on engineering performance or rely on broad socio-economic frameworks that are not well suited to behind-the-meter storage. As a result, developers, utilities, and funders often lack consistent methods for defining success, quantifying benefits, and comparing outcomes across projects. This report presents a practitioner-oriented impact assessment framework for evaluating behind-the-meter BESS at resilience hubs and critical infrastructure facilities. The framework is organized into five iterative components—developing an action plan, defining project goals, identifying metrics, collecting data and measuring outcomes, and reporting and using results—and includes a structured metric architecture spanning six impact categories. Designed for real-world constraints such as limited staffing and uneven data availability, the framework was developed, applied, and refined through real projects, and is illustrated with case studies across diverse deployment contexts.
Impact Assessment↗