Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Evaluation Framework”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

A Design for Remanufacturing Framework Incorporating Identification, Evaluation, and Validation: A Case Study of Hydraulic Manifold

In recent years, academic researchers and engineers in the industry have widely recognized the necessity of integrating remanufacturing considerations into product design iterations to advance sustainability objectives. Acknowledging the importance of design for remanufacturing (DfRem), efforts were made to develop tools and guidelines that could be implemented in practice. However, such methods largely rely upon experiential insights and qualitative assessments, leaving a gap in the ability to quantitatively assess the economic and environmental impacts of design choices. To bridge this gap, we investigate existing efforts and present a framework for DfRem that integrates established design and remanufacturing practices into a cohesive workflow with quantitative assessments. To demonstrate its efficacy for making practical design changes for remanufacturing, we apply the framework to a hydraulic manifold in a transmission system for heavy-duty tractors. Through this industry-relevant case study, we focus on showcasing the practical utility of our framework. Based on the identified design modifications from remanufacturability analysis, we estimate the reductions in life cycle costs, energy consumption, and emissions. Afterward, the modifications are tested using physical experiments with plans for integration into future iterations of the hydraulic manifold design and production. Here, we anticipate this framework can illustrate the process of remanufacturing that ensures improvements in sustainability while maintaining performance and reliability standards.

design for X↗
DOE OSTI2025

A Novel Framework for Performance Evaluation and Design Optimization of PCM Embedded Heat Exchangers for the Built Environment

This research sheds light on the performance evaluation and design optimization of PCM-HXs for the built environment, addressing several barriers to practical issues to PCM-HX commercialization such as modeling aspects (i.e., modeling expertise and computational / time investment, etc.), manufacturing aspects (i.e., at-scale manufacturing, cost assessments, etc.) and experimental performance assessment (i.e., reliable experimental data, assessment of multiple PCM-working fluid combinations, etc.). We present a novel, comprehensive, and experimentally-validated design optimization framework for PCM-HXs capable of simulating any PCM-HX geometry with reasonable accuracy and significant computational time savings when compared to traditional CFD-based design practices. The framework was validated for a wide range of PCM-HX configurations, including a design optimization for a domestic hot water heater application where TES partially replaces electrical heating input. The resulting PCM-HXs were found to deliver 34-68% of the total daily hot water supply with only 5-10% package volume increase from the water heater, thus within U.S. DOE targets for TES systems. To identify the most promising HXs for PCM applications, first-order geometry and cost analyses were conducted based on off-the-shelf HX products. As part of this work, 9 PCM-HX prototypes were manufactured using additive and conventional manufacturing methods. Detailed economy-of-scale assessments were conducted for the most promising PCM-HXs and were found to have a good outlook for the next 5-10 years. The PCM-HX design optimization framework was validated through comprehensive in-house experimental testing using newly-developed PCM-to-fluid test facilities. In total,10 total in-house component-level experiments were conducted using these prototypes, including 9 with water and 1 with refrigerant (R410A) as the working fluid. It was found that the framework can successfully predict experimental thermal-hydraulic performance within ±10-20% the first time without manual design changes, eliminating the need for time-consuming and expensive prototyping efforts as part of the design process. As part of this work, a publicly-available PCM web tool was released which includes a PCM property database (531 PCMs) and PCM-HX modeling tool to assist the design community on common PCM-HX use-cases, e.g., single/multiple flow path(s) fluid-to-PCM and air-to-fluid-to-PCM configurations (https://ceeeweb.umd.edu/pcmapp/). This work will accelerate the design and time to market for next generation PCM-HXs.

25 ENERGY STORAGE↗
NASA NTRS2002

Software Engineering Support of the Third Round of Scientific Grand Challenge Investigations: Earth System Modeling Software Framework Survey

One of the most significant challenges in large-scale climate modeling, as well as in high-performance computing in other scientific fields, is that of effectively integrating many software models from multiple contributors. A software framework facilitates the integration task, both in the development and runtime stages of the simulation. Effective software frameworks reduce the programming burden for the investigators, freeing them to focus more on the science and less on the parallel communication implementation. while maintaining high performance across numerous supercomputer and workstation architectures. This document surveys numerous software frameworks for potential use in Earth science modeling. Several frameworks are evaluated in depth, including Parallel Object-Oriented Methods and Applications (POOMA), Cactus (from (he relativistic physics community), Overture, Goddard Earth Modeling System (GEMS), the National Center for Atmospheric Research Flux Coupler, and UCLA/UCB Distributed Data Broker (DDB). Frameworks evaluated in less detail include ROOT, Parallel Application Workspace (PAWS), and Advanced Large-Scale Integrated Computational Environment (ALICE). A host of other frameworks and related tools are referenced in this context. The frameworks are evaluated individually and also compared with each other.

Talbot, Bryan↗
NASA NTRS2012

A Framework for Probabilistic Evaluation of Interval Management Tolerance in the Terminal Radar Control Area

Projections of future traffic in the national airspace show that most of the hub airports and their attendant airspace will need to undergo significant redevelopment and redesign in order to accommodate any significant increase in traffic volume. Even though closely spaced parallel approaches increase throughput into a given airport, controller workload in oversubscribed metroplexes is further taxed by these approaches that require stringent monitoring in a saturated environment. The interval management (IM) concept in the TRACON area is designed to shift some of the operational burden from the control tower to the flight deck, placing the flight crew in charge of implementing the required speed changes to maintain a relative spacing interval. The interval management tolerance is a measure of the allowable deviation from the desired spacing interval for the IM aircraft (and its target aircraft). For this complex task, Formal Methods can help to ensure better design and system implementation. In this paper, we propose a probabilistic framework to quantify the uncertainty and performance associated with the major components of the IM tolerance. The analytical basis for this framework may be used to formalize both correctness and probabilistic system safety claims in a modular fashion at the algorithmic level in a way compatible with several Formal Methods tools.

Hercencia-Zapana, Heber↗
NASA NTRS2023

A Multi-Satellite Framework to Rapidly Evaluate Extreme Biosphere Cascades: The Western US 2021 Drought and Heatwave

The increasing frequency and intensity of climate extremes and complex ecosystem responses motivate the need for integrated observational studies at low-latency to determine biosphere responses and carbon-climate feedbacks. Here, we develop a satellite-based rapid attribution workflow and demonstrate its use at a 1–2-month latency to attribute drivers of the carbon cycle feedbacks during the 2020-2021 Western US drought and heatwave. In the first half of 2021, concurrent negative photosynthesis anomalies and large positive column CO 2 anomalies were detected with satellites. Using a simple atmospheric mass balance approach, we estimate a surface carbon efflux anomaly of 132 TgC in June 2021, a magnitude corroborated 28 independently with a dynamic global vegetation model. Integrated satellite observations of hydrologic processes, representing the soil-plant-atmosphere continuum (SPAC), show that these surface carbon flux anomalies are largely due to substantial reductions in photosynthesis because of a spatially widespread moisture-deficit propagation through the SPAC between 2020 and 2021. A causal model indicates deep soil moisture stores partially drove photosynthesis, maintaining its values in 2020 and driving its declines throughout 2021. The causal model also suggests legacy effects may have amplified photosynthesis deficits in 2021 beyond the direct effects of environmental forcing. The integrated, observation framework presented here provides a valuable first assessment of a biosphere extreme response and an independent testbed for improving drought propagation and mechanisms in models. The rapid identification of extreme carbon anomalies and hotspots can also aid mitigation and adaptation decisions.

Causal model↗
DOE OSTI2022

Framework for Quantitative Evaluation of Resilience Solutions: An Approach to Determine the Value of Resilience for a Particular Site

The paper provides the approach to providing a benefit cost analysis of energy and water alternatives to provide resilience to extreme events. The approach estimates the costs and returns of providing greater resilience of water and energy infrastructure. Extreme events are defined as high impact, low-frequency events such as, but not limited to, hurricanes, floods, storm surges and earthquakes. The provides justification for hardening water and energy infrastructure. Resilience is defined as “the ability to prepare for and to withstand an extreme event with little or no damage, or to recover more quickly from an extreme event.” The approach can be summarized as follows. The approach requires the development of a baseline with which to compare alternatives. The baseline is used to evaluate the baseline’s resilience to hazards through the probability of the hazard(s), the likelihood of damage from that the hazard through a vulnerability analysis, and the consequence to calculate a cost of the damage. The approach then evaluates proposed mitigation alternatives that would improve the resilience of the system. Each alternative is evaluated based on probability of the hazard, probability of vulnerability and consequence to determine the reduced damage that each alternative presents. The approach includes any monetary and non-monetary benefits that can quantified for each of the alternatives. Non-quantifiable benefits are evaluated based on the relative importance of each alternative to the criteria used to determine how well the alternative meets the goals and objectives of the site/facility. Then, a life cycle cost analysis should be conducted for the baseline and alternatives. Finally, the results of the life cycle analysis should be presented in a decision matrix with cost, net present value, benefit/cost ratios, and any non-monetary criteria ranked to show how well the alternatives met the criteria, weighted with the decision maker’s weights and the results presented.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2018

(GO)2-SIM: a GCM-Oriented Ground-Observation Forward-Simulator Framework for Objective Evaluation of Cloud and Precipitation Phase

General circulation model (GCM) evaluation using ground-based observations is complicated by inconsistencies in hydrometeor and phase definitions. Here we describe (GO)2-SIM, a forward simulator designed for objective hydrometeor-phase evaluation, and assess its performance over the North Slope of Alaska using a 1-year GCM simulation. For uncertainty assessment, 18 empirical relationships are used to convert model grid-average hydrometeor (liquid and ice, cloud, and precipitation) water contents to zenith polarimetric micropulse lidar and Ka-band Doppler radar measurements, producing an ensemble of 576 forward-simulation realizations. Sensor limitations are represented in forward space to objectively remove from consideration model grid cells with undetectable hydrometeor mixing ratios, some of which may correspond to numerical noise.Phase classification in forward space is complicated by the inability of sensors to measure ice and liquid signals distinctly. However, signatures exist in lidar–radar space such that thresholds on observables can be objectively estimated and related to hydrometeor phase. The proposed phase-classification technique leads to misclassification in fewer than 8% of hydrometeor-containing grid cells. Such misclassifications arise because, while the radar is capable of detecting mixed-phase conditions, it can mistake water- for ice-dominated layers. However, applying the same classification algorithm to forward-simulated and observed fields should generate hydrometeor-phase statistics with similar uncertainty. Alternatively, choosing to disregard how sensors define hydrometeor phase leads to frequency of occurrence discrepancies of up to 40%. So, while hydrometeor-phase maps determined in forward space are very different from model "reality" they capture the information sensors can provide and thereby enable objective model evaluation.

Lamer, K.↗
NASA NTRSDate not supplied

Traffic Prediction for Uncommunicative Aircraft in Terminal Airspace: Development Framework and Performance Evaluations

This paper presents an air traffic prediction algorithm that takes observations of an aircraft and classifies aircraft type, estimates the aircraft's intent to and method of joining an airport traffic pattern, and predicts the aircaft's future trajectory. To develop algorithms that enable autonomous aircraft to safely insert into un-towered traffic patterns, several challenges need to be addressed. These challenges range from traffic detection to sensor fusion to own-ship trajectory replanning. Critical to a trajectory replanning algorithm is information regarding the future behavior of all traffic aircraft in the operational environment. The presented traffic prediction algorithm generates this information using regular measurements of traffic aircraft position and velocity to classify the aircraft by speed-class, estimate how the aircraft will approach the runway, and construct a predicted trajectory to the runway including future positions and velocities at specific times. The predictions of the presented algorithm are the necessary inputs for any downstream traffic pattern sequencing and own-ship trajectory planning routines. The presented algorithm is benchmarked using approximately 300 randomized traffic trajectories, spanning four vehicle weight classes and eight traffic entry types. While the algorithm can process multiple traffic vehicles in the terminal area, there is no prediction of traffic-on-traffic interaction. Each traffic vehicle is processed separately.

John D McMinn↗
DOE OSTI2024

Framework for Quantitative Evaluation of Resilience Solutions: An Approach to Determine the Value of Resilience for a Particular Site

This paper describes an approach to obtaining a dollar value for the improvement in the resilience of projects for a particular site. The paper provides an approach to valuing resilience to provide justification for hardening cyber facilities that can be used by expert users/economists in undertaking investment-grade valuations to validate the appropriateness of funding for resilience mitigation projects.

resilience, valuation, cyber security↗
NASA NTRSDate not supplied

The Inspectability Metric: A Formalized System Of Measurement Enabling The Design For Inspection Framework

Nondestructive evaluation (NDE) engineers are often confronted with structural design choices that present challenges to meeting inspection requirements. These challenges, at best, increase the resources needed to design an inspection solution and, at worst, require resource intensive redesign of the structure. If the inspectability of the structure can be determined early in the design cycle, these challenging inspection scenarios can be avoided. The emergence of additive manufacturing has further compounded this problem by enabling the creation of highly optimized structures with no regard to inspection constraints. Design for inspection (DFI) offers a framework to integrate nondestructive evaluation (NDE) into the design process to alleviate the mechanisms that produce uninspectable designs. DFI is the concept of including inspectability in a multi-objective optimization framework so that it can be considered in parallel to other metrics such as mass and manufacturability. This allows rapid evaluation of the trade-off between design metrics to find solutions that meet the inspection needs of a particular material system, structural concept, or vehicle program. To enable DFI, there must be a system by which the inspectability of a structure can be measured. This system must be agile to produce results quickly, it must be versatile to work with the type of incomplete information one would encounter early in the design process (such as lack of inspection requirements), and it must be delivered in a form that is easily understood by designers. To meet this need, this presentation introduces the novel inspectability metric as a system to measure inspectability. The inspectability metric is a standardized, automation friendly procedure that uses simulations to determine inspectability. Along with guidelines to properly process designs and integrate with existing workflows, the inspectability metric provides a suite of simulation tests to interrogate the ability to find defects and the sensitivity to variability. The testing rubric is designed to maximize the coverage of the parameter space while minimizing the number of simulations needed. The inspectability metric has been in development in collaboration with industry partners to ensure compatibility with modern simulation tools and aerospace design workflows. In this study, we will demonstrate how the inspectability metric is able to determine the inspectability of multiple types of structures, including aerospace composites and additively manufactured parts. We will then show how the inspectability score can be plugged into existing design optimization tasks, such as structural sizing algorithms or design for manufacturing (DFM) frameworks.

Design for inspection↗
DOE OSTI2023

A computational modeling framework for pre-clinical evaluation of cardiac mapping systems

There are a variety of difficulties in evaluating clinical cardiac mapping systems, most notably the inability to record the transmembrane potential throughout the entire heart during patient procedures which prevents the comparison to a relevant “gold standard”. Cardiac mapping systems are comprised of hardware and software elements including sophisticated mathematical algorithms, both of which continue to undergo rapid innovation. The purpose of this study is to develop a computational modeling framework to evaluate the performance of cardiac mapping systems. The framework enables rigorous evaluation of a mapping system’s ability to localize and characterize (i.e., focal or reentrant) arrhythmogenic sources in the heart. The main component of our tool is a library of computer simulations of various dynamic patterns throughout the entire heart in which the type and location of the arrhythmogenic sources are known. Our framework allows for performance evaluation for various electrode configurations, heart geometries, arrhythmias, and electrogram noise levels and involves blind comparison of mapping systems against a “silver standard” comprised of computer simulations in which the precise transmembrane potential patterns throughout the heart are known. A feasibility study was performed using simulations of patterns in the human left atria and three hypothetical virtual catheter electrode arrays. Activation times (AcT) and patterns (AcP) were computed for three virtual electrode arrays: two basket arrays with good and poor contact and one high-resolution grid with uniform spacing. The average root mean squared difference of AcTs of electrograms and those of the nearest endocardial action potential was less than 1 ms and therefore appears to be a poor performance metric. In an effort to standardize performance evaluation of mapping systems a novel performance metric is introduced based on the number of AcPs identified correctly and those considered spurious as well as misclassifications of arrhythmia type; spatial and temporal localization accuracy of correctly identified patterns was also quantified. This approach provides a rigorous quantitative analysis of cardiac mapping system performance. Proof of concept of this computational evaluation framework suggests that it could help safeguard that mapping systems perform as expected as well as provide estimates of system accuracy.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2020

Evaluation of Graph Analytics Frameworks Using the GAP Benchmark Suite

The analysis of connected data is an increasingly important application in high-performance computing. Such analyses can reveal fraudulent patterns in financial transactions, optimize telecommunications networks, predict information flow in social networks, etc. However, the landscape of graph analytics is highly diverse. Graph algorithms stress processor architectures differently, and no one graph can represent all topologies. Consequently, no single approach or framework is expected to be optimal for all graph analytics problems. To help make sense of this diverse landscape, we evaluated four approaches to graph analytics: GraphBLAS, Galois, BGL17, GraphIt; and compare them against hand-tuned implementations that take advantage of hardware features on our test platform. Graph- BLAS formulates graph analytics as sparse linear algebra. Galois provides syntactic constructs for data parallelism over irregular data structures. BGL17 is a generic C++ template library for implementing graph algorithms. GraphIt provides a domain- specific language to describe and optimize graph algorithms. We use the GAP Benchmark Suite to establish baseline performance and guide the side-by-side evaluation of each framework. GAP consists of 30 tests: six graph analytics algorithms (breadth- first search, single-source shortest path, PageRank, betweenness centrality, connected components, and triangle counting) run on five graphs, each with different topological characteristics (e.g., high diameter, skewed degree distribution, high average degree). High-performance reference implementations are included for each benchmark algorithm. Because a graph can be loaded into memory a number of ways (e.g., flat file on disk, compressed sparse format, data frames, retrieved from SQL or NoSQL databases), our evaluation focused on computational performance rather than I/O. Our results show the relative strengths of each framework.

Graph algorithms, Benchmarking, shared-memory prog↗
DOE OSTI2023

I Know I'm Right, But Does My Phone?

Transportation is the largest source of green-house gas emissions in the United States. Reducing transportation emissions depends on human travel behavior, which relies on local land use and planning. Travel diaries, consisting of sequences of trips between places for a particular individual, are typically used to instrument human travel behavior. However, these diaries are only as accurate as the underlying methods used to construct them. Travel diary algorithms have been a popular research topic since the advent of GPS tracking surveys. Mode inference algorithms in particular have been well represented in literature. However, these algorithms have typically been validated using prompted recall of pre-segmented trips, which doesn't account for segmentation error, thus disregarding the continuity of mode inference. Furthermore, phone operating systems and applications have adopted battery-conserving techniques, but we are not aware of prior work that has characterized the resulting data collection errors or evaluated procedures to mitigate them. We introduce a framework to evaluate accuracy of trip length computations and mode inference. We develop a temporal alignment procedure in analyzing continuous mode-segmented trajectories for groups of trips. We then apply our framework to evaluate an example set of travel diary algorithms from the open-source OpenPATH travel diary platform against MobilityNet, a public dataset containing information from three artificial timelines that cover 15 different travel modes. Our results show that inference based on an integration with map features results in weighted F_1 scores of 0.60 (iOS) and 0.74 (android). We also show that OpenPATH tends to under count trip length, with mean of signed relative error of -0.0438 on android and -0.0704 on iOS. We hope that other travel diary algorithms will be evaluated using this standardized process, and that the results used to understand and improve the state-of-the-art in this field.

ADVANCED PROPULSION SYSTEMS,ENERGY PLANNING, POLIC↗
DOE OSTI2023

Jay : A software framework for prototyping and evaluating offloading applications in hybrid edge clouds

Abstract We present Jay , a software framework for offloading applications in hybrid edge clouds. Jay provides an API, services, and tools that enable mobile application developers to implement, instrument, and evaluate offloading applications using configurable cloud topologies, offloading strategies, and job types. We start by presenting Jay 's job model and the concrete architecture of the framework. We then present the programming API with several examples of customization. Then, we turn to the description of the internal implementation of Jay instances and their components. Finally, we describe the Jay Workbench, a tool that allows the setup, execution, and reproduction of experiments with networks of hosts with different resource capabilities organized with specific topologies. The complete source code for the framework and workbench is provided in a GitHub repository.

Silva, Joaquim↗
NASA NTRS1977

New model framework and structure and the commonality evaluation model

The development of a framework and structure for shuttle era unmanned spacecraft projects and the development of a commonality evaluation model is documented. The methodology developed for model utilization in performing cost trades and comparative evaluations for commonality studies is discussed. The model framework consists of categories of activities associated with the spacecraft system's development process. The model structure describes the physical elements to be treated as separate identifiable entities. Cost estimating relationships for subsystem and program-level components were calculated.

Source record↗