Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “evaluation framework”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Assessing Customer Experience and Business Models around Price-to-Device Communication and Smart Control Pathways in CalFlexHub

California is facing three major challenges in electrical grid operation: renewable overgeneration, steep evening ramping, and growing peak demand. The state has identified dynamic retail price response as a key strategy evidenced by CPUC’s Dynamic Rates proceeding and CEC’s Load Management Standards. Furthermore, the CEC launched a $16M “California Load Flexibility Research and Deployment Hub (CalFlexHub)” administered by Berkeley Lab to accelerate price-response flexible load technologies in buildings and EV charging. There are more than 16 laboratory and field demonstration projects in CalFlexHub, each demonstrating innovative automated price-response technologies. CalFlexHub tests various pathways through which hourly price signals and triggered control commands are communicated to load-flexible devices such as smart thermostats, heat pumps, water heaters, and EVs. We identified seven unique communication and control pathways, which involve combinations of third-party cloud, device OEM’s cloud, building central gateway, and local controller in between the price server and the load-flexible devices. It is important for utilities and policy makers to understand the long-term implications of each pathway in designing future programs and creating related policies and mandates for market transformation. We propose an evaluation framework including the following aspects: ● Functionality: connectivity and uptime, resilience, and optimization; ● Customer experience: simplicity in setup, troubleshooting support, continuity, customer choice, first cost, and ongoing cost; ● Business model and scalability: advance interoperability, holistic solution, bridge unique gap, customer base, and value streams and pricing structures. In this paper, we identify emerging business models associated with each communication pathway and discuss their positive features and challenges from the above aspects.

Liu, Jingjing↗
DOE OSTI2021

A modeling framework for designing and evaluating curbside traffic management policies at Dallas-Fort Worth International Airport

Emerging mobility technologies are changing the transportation system landscape. This is especially evident at airports, such as the Dallas-Fort Worth International Airport (DFW). Without careful analysis, these changes could lead to inefficient and costly airport operations. This paper presents a modeling framework that integrates travel mode encoding, demand projection, and microsimulation to enable airports to develop, simulate, and evaluate curbside traffic managements policies and measure their impact. Here, the framework is utilized to analyze several traffic scenarios and policies for DFW: a baseline scenario which represents DFW traffic pattern as observed in 2018 and projected to 2045, a transit network company (TNC) electrification policy, a TNC queuing policy, a policy that increased transit ridership, a bus-only policy which considers the use of only buses inside DFW, an autonomous vehicle (AV) policy which investigates the impact of autonomous vehicle (AV) adoption on airport operations, and an example COVID-19 scenario which models the impact of the COVID19 pandemic. The simulations’ results demonstrate that: increasing the DFW transit ridership postpones the need for airport curbside expansion the most; encouraging shared-mobility with the bus-only policy produces the most savings in curbside congestion delays; automation and electrification for all passenger vehicle trips to/from DFW generates the most saving in fuel consumption and emissions; and uncontrolled AV adoption incurs the highest increase in fuel consumption, delay, and emissions and could require immediate airport capacity extension. Without policy intervention or investment in additional infrastructure capacity, these results predict the current operations would face significant congestion on high demand days starting as early as 2028. While derived in close partnership with DFW, the methodology presented here can be generalized to any airport.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2025

A Design for Remanufacturing Framework Incorporating Identification, Evaluation, and Validation: A Case Study of Hydraulic Manifold

In recent years, academic researchers and engineers in the industry have widely recognized the necessity of integrating remanufacturing considerations into product design iterations to advance sustainability objectives. Acknowledging the importance of design for remanufacturing (DfRem), efforts were made to develop tools and guidelines that could be implemented in practice. However, such methods largely rely upon experiential insights and qualitative assessments, leaving a gap in the ability to quantitatively assess the economic and environmental impacts of design choices. To bridge this gap, we investigate existing efforts and present a framework for DfRem that integrates established design and remanufacturing practices into a cohesive workflow with quantitative assessments. To demonstrate its efficacy for making practical design changes for remanufacturing, we apply the framework to a hydraulic manifold in a transmission system for heavy-duty tractors. Through this industry-relevant case study, we focus on showcasing the practical utility of our framework. Based on the identified design modifications from remanufacturability analysis, we estimate the reductions in life cycle costs, energy consumption, and emissions. Afterward, the modifications are tested using physical experiments with plans for integration into future iterations of the hydraulic manifold design and production. Here, we anticipate this framework can illustrate the process of remanufacturing that ensures improvements in sustainability while maintaining performance and reliability standards.

design for X↗
DOE OSTI2025

A Novel Framework for Performance Evaluation and Design Optimization of PCM Embedded Heat Exchangers for the Built Environment

This research sheds light on the performance evaluation and design optimization of PCM-HXs for the built environment, addressing several barriers to practical issues to PCM-HX commercialization such as modeling aspects (i.e., modeling expertise and computational / time investment, etc.), manufacturing aspects (i.e., at-scale manufacturing, cost assessments, etc.) and experimental performance assessment (i.e., reliable experimental data, assessment of multiple PCM-working fluid combinations, etc.). We present a novel, comprehensive, and experimentally-validated design optimization framework for PCM-HXs capable of simulating any PCM-HX geometry with reasonable accuracy and significant computational time savings when compared to traditional CFD-based design practices. The framework was validated for a wide range of PCM-HX configurations, including a design optimization for a domestic hot water heater application where TES partially replaces electrical heating input. The resulting PCM-HXs were found to deliver 34-68% of the total daily hot water supply with only 5-10% package volume increase from the water heater, thus within U.S. DOE targets for TES systems. To identify the most promising HXs for PCM applications, first-order geometry and cost analyses were conducted based on off-the-shelf HX products. As part of this work, 9 PCM-HX prototypes were manufactured using additive and conventional manufacturing methods. Detailed economy-of-scale assessments were conducted for the most promising PCM-HXs and were found to have a good outlook for the next 5-10 years. The PCM-HX design optimization framework was validated through comprehensive in-house experimental testing using newly-developed PCM-to-fluid test facilities. In total,10 total in-house component-level experiments were conducted using these prototypes, including 9 with water and 1 with refrigerant (R410A) as the working fluid. It was found that the framework can successfully predict experimental thermal-hydraulic performance within ±10-20% the first time without manual design changes, eliminating the need for time-consuming and expensive prototyping efforts as part of the design process. As part of this work, a publicly-available PCM web tool was released which includes a PCM property database (531 PCMs) and PCM-HX modeling tool to assist the design community on common PCM-HX use-cases, e.g., single/multiple flow path(s) fluid-to-PCM and air-to-fluid-to-PCM configurations (https://ceeeweb.umd.edu/pcmapp/). This work will accelerate the design and time to market for next generation PCM-HXs.

25 ENERGY STORAGE↗
DOE OSTI2022

Framework for Quantitative Evaluation of Resilience Solutions: An Approach to Determine the Value of Resilience for a Particular Site

The paper provides the approach to providing a benefit cost analysis of energy and water alternatives to provide resilience to extreme events. The approach estimates the costs and returns of providing greater resilience of water and energy infrastructure. Extreme events are defined as high impact, low-frequency events such as, but not limited to, hurricanes, floods, storm surges and earthquakes. The provides justification for hardening water and energy infrastructure. Resilience is defined as “the ability to prepare for and to withstand an extreme event with little or no damage, or to recover more quickly from an extreme event.” The approach can be summarized as follows. The approach requires the development of a baseline with which to compare alternatives. The baseline is used to evaluate the baseline’s resilience to hazards through the probability of the hazard(s), the likelihood of damage from that the hazard through a vulnerability analysis, and the consequence to calculate a cost of the damage. The approach then evaluates proposed mitigation alternatives that would improve the resilience of the system. Each alternative is evaluated based on probability of the hazard, probability of vulnerability and consequence to determine the reduced damage that each alternative presents. The approach includes any monetary and non-monetary benefits that can quantified for each of the alternatives. Non-quantifiable benefits are evaluated based on the relative importance of each alternative to the criteria used to determine how well the alternative meets the goals and objectives of the site/facility. Then, a life cycle cost analysis should be conducted for the baseline and alternatives. Finally, the results of the life cycle analysis should be presented in a decision matrix with cost, net present value, benefit/cost ratios, and any non-monetary criteria ranked to show how well the alternatives met the criteria, weighted with the decision maker’s weights and the results presented.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Framework for Quantitative Evaluation of Resilience Solutions: An Approach to Determine the Value of Resilience for a Particular Site

This paper describes an approach to obtaining a dollar value for the improvement in the resilience of projects for a particular site. The paper provides an approach to valuing resilience to provide justification for hardening cyber facilities that can be used by expert users/economists in undertaking investment-grade valuations to validate the appropriateness of funding for resilience mitigation projects.

resilience, valuation, cyber security↗
DOE OSTI2023

A computational modeling framework for pre-clinical evaluation of cardiac mapping systems

There are a variety of difficulties in evaluating clinical cardiac mapping systems, most notably the inability to record the transmembrane potential throughout the entire heart during patient procedures which prevents the comparison to a relevant “gold standard”. Cardiac mapping systems are comprised of hardware and software elements including sophisticated mathematical algorithms, both of which continue to undergo rapid innovation. The purpose of this study is to develop a computational modeling framework to evaluate the performance of cardiac mapping systems. The framework enables rigorous evaluation of a mapping system’s ability to localize and characterize (i.e., focal or reentrant) arrhythmogenic sources in the heart. The main component of our tool is a library of computer simulations of various dynamic patterns throughout the entire heart in which the type and location of the arrhythmogenic sources are known. Our framework allows for performance evaluation for various electrode configurations, heart geometries, arrhythmias, and electrogram noise levels and involves blind comparison of mapping systems against a “silver standard” comprised of computer simulations in which the precise transmembrane potential patterns throughout the heart are known. A feasibility study was performed using simulations of patterns in the human left atria and three hypothetical virtual catheter electrode arrays. Activation times (AcT) and patterns (AcP) were computed for three virtual electrode arrays: two basket arrays with good and poor contact and one high-resolution grid with uniform spacing. The average root mean squared difference of AcTs of electrograms and those of the nearest endocardial action potential was less than 1 ms and therefore appears to be a poor performance metric. In an effort to standardize performance evaluation of mapping systems a novel performance metric is introduced based on the number of AcPs identified correctly and those considered spurious as well as misclassifications of arrhythmia type; spatial and temporal localization accuracy of correctly identified patterns was also quantified. This approach provides a rigorous quantitative analysis of cardiac mapping system performance. Proof of concept of this computational evaluation framework suggests that it could help safeguard that mapping systems perform as expected as well as provide estimates of system accuracy.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2020

Evaluation of Graph Analytics Frameworks Using the GAP Benchmark Suite

The analysis of connected data is an increasingly important application in high-performance computing. Such analyses can reveal fraudulent patterns in financial transactions, optimize telecommunications networks, predict information flow in social networks, etc. However, the landscape of graph analytics is highly diverse. Graph algorithms stress processor architectures differently, and no one graph can represent all topologies. Consequently, no single approach or framework is expected to be optimal for all graph analytics problems. To help make sense of this diverse landscape, we evaluated four approaches to graph analytics: GraphBLAS, Galois, BGL17, GraphIt; and compare them against hand-tuned implementations that take advantage of hardware features on our test platform. Graph- BLAS formulates graph analytics as sparse linear algebra. Galois provides syntactic constructs for data parallelism over irregular data structures. BGL17 is a generic C++ template library for implementing graph algorithms. GraphIt provides a domain- specific language to describe and optimize graph algorithms. We use the GAP Benchmark Suite to establish baseline performance and guide the side-by-side evaluation of each framework. GAP consists of 30 tests: six graph analytics algorithms (breadth- first search, single-source shortest path, PageRank, betweenness centrality, connected components, and triangle counting) run on five graphs, each with different topological characteristics (e.g., high diameter, skewed degree distribution, high average degree). High-performance reference implementations are included for each benchmark algorithm. Because a graph can be loaded into memory a number of ways (e.g., flat file on disk, compressed sparse format, data frames, retrieved from SQL or NoSQL databases), our evaluation focused on computational performance rather than I/O. Our results show the relative strengths of each framework.

Graph algorithms, Benchmarking, shared-memory prog↗
DOE OSTI2023

I Know I'm Right, But Does My Phone?

Transportation is the largest source of green-house gas emissions in the United States. Reducing transportation emissions depends on human travel behavior, which relies on local land use and planning. Travel diaries, consisting of sequences of trips between places for a particular individual, are typically used to instrument human travel behavior. However, these diaries are only as accurate as the underlying methods used to construct them. Travel diary algorithms have been a popular research topic since the advent of GPS tracking surveys. Mode inference algorithms in particular have been well represented in literature. However, these algorithms have typically been validated using prompted recall of pre-segmented trips, which doesn't account for segmentation error, thus disregarding the continuity of mode inference. Furthermore, phone operating systems and applications have adopted battery-conserving techniques, but we are not aware of prior work that has characterized the resulting data collection errors or evaluated procedures to mitigate them. We introduce a framework to evaluate accuracy of trip length computations and mode inference. We develop a temporal alignment procedure in analyzing continuous mode-segmented trajectories for groups of trips. We then apply our framework to evaluate an example set of travel diary algorithms from the open-source OpenPATH travel diary platform against MobilityNet, a public dataset containing information from three artificial timelines that cover 15 different travel modes. Our results show that inference based on an integration with map features results in weighted F_1 scores of 0.60 (iOS) and 0.74 (android). We also show that OpenPATH tends to under count trip length, with mean of signed relative error of -0.0438 on android and -0.0704 on iOS. We hope that other travel diary algorithms will be evaluated using this standardized process, and that the results used to understand and improve the state-of-the-art in this field.

ADVANCED PROPULSION SYSTEMS,ENERGY PLANNING, POLIC↗
DOE OSTI2023

Jay : A software framework for prototyping and evaluating offloading applications in hybrid edge clouds

Abstract We present Jay , a software framework for offloading applications in hybrid edge clouds. Jay provides an API, services, and tools that enable mobile application developers to implement, instrument, and evaluate offloading applications using configurable cloud topologies, offloading strategies, and job types. We start by presenting Jay 's job model and the concrete architecture of the framework. We then present the programming API with several examples of customization. Then, we turn to the description of the internal implementation of Jay instances and their components. Finally, we describe the Jay Workbench, a tool that allows the setup, execution, and reproduction of experiments with networks of hosts with different resource capabilities organized with specific topologies. The complete source code for the framework and workbench is provided in a GitHub repository.

Silva, Joaquim↗
DOE OSTI2026

A framework and calculator for evaluating the impacts of shelf life extension and other food loss and waste reduction technologies

Optimization of the food supply chain (FSC) depends on reducing food waste, especially at the consumer stage, where a substantial portion of food is not eaten, but instead disposed of via landfill, incineration, or in-sink disposals. One key strategy is to increase the time that consumers have before food goes bad or expires. This study developed a framework to assess the efficacy of shelf-life extension (SLE) technologies for mitigating food loss and waste (FLW), such as packaging improvements. The impact flows through the entire FSC, reducing FLW, energy use, and other inputs at each stage. The framework and resulting calculator can be used to evaluate the impact of FLW reduction at any stage for any food commodity. As shown by two SLE cases, the calculator is valuable for policy-makers, government entities, and professionals, specifically those in marketing, business development, and capital projects teams, to comprehensively evaluate the impacts of FLW reduction technologies and practices. The framework and calculator are sensitive to the shape of the consumption curve, the fraction of inedible waste, and the current shelf life. The calculator was used to assess the impacts of the United States goal of reducing food waste by consumers through various SLE lengths. It was found that uptake of several near-ready-to-deploy SLE technologies would reduce annual food production demand by about 10–19 MMT and supply chain energy consumption by 240–410 PJ in the United States.

Food loss and waste (FLW)↗
DOE OSTI2025

Techno-Economic Analysis of Data-Driven and Transactive Approaches for Resilience Enhancement

As extreme weather events lead to more frequent power outages, understanding and enhancing grid resilience is critical to mitigating economic losses and non-energy impacts from service disruptions. Here, this study introduces a novel techno-economic analysis framework for evaluating resilience enhancement mechanisms. The framework combines grid response modeling with a co-simulation approach and valuation methodology to provide a comprehensive assessment. We apply this framework to a realistic case study of the Texas grid during Winter Storm Uri in February 2021. Two advanced resilience strategies are analyzed: a data-driven rolling outage mechanism and a transactive energy (TE) based allocation scheme. The rolling outage scheme selectively serves customers based on real-time curtailment needs, while the TE scheme allows customers to trade energy allocations according to their preferences. Our findings show that both the rolling outage and TE schemes significantly outperform conventional methods (i.e. controlled outages) by reducing the amount of energy not supplied to customers by 41% and 64%, respectively. These approaches also enhance flexibility and customer satisfaction, while improving energy utilization for greater resilience. Additionally, they maintain thermal comfort about 3.5 times better and substantially lower customer risk exposure. A key contribution of this study is addressing both utility and customer perspectives while considering both energy and non-energy impacts. The techno-economic analysis indicates that implementing these resilience enhancement strategies would incur an additional 1.1Bto1.6B in utility costs but has the potential to avoid 17.3Bto18B of customer losses as compared to existing solutions, thereby underscoring the value of investing in advanced resilience, as it provides significant societal benefits to customers.

42 ENGINEERING↗
DOE OSTI2020

Evaluating long-term model-based scenarios of the energy system

Energy-economic models are used to provide science-based decision support in a variety of contexts. Analyses using these tools often involve defining a “reference” scenario, which serves as a counterfactual against which alternative scenarios are compared. Evaluating scenarios, including reference scenarios, is critical for establishing the credibility of the analyses these models support. We propose a framework for evaluating energy system scenarios which consists of three parts – a qualitative storyline, quantitative metrics, and evaluation criteria. We apply this framework to the reference scenario for GCAM-USA, a version of the global human-Earth system model GCAM (Global Change Assessment Model) with state-level detail in the United States, focusing on the evolution of the electric power sector. We develop new visual analytic tools to facilitate the evaluation of model outcomes in 51 sub-national regions, and demonstrate how scenario performance can be tracked and compared across four quantifications of the GCAM-USA reference scenario.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2020

Preparing Distribution Utilities for Utility-Scale Storage and Electric Vehicles: A Novel Analytical Framework

Emerging distributed energy resources (DERs)—such as solar photovoltaics (PV), battery energy storage systems (BESS), and electric vehicles (EVs)—are expected to increase substantially in India in the coming years following policy-driven targets of the Government of India to modernize its electricity system, reduce greenhouse gas emissions (GHGs), and improve air quality. These emerging technologies can pose challenges to distribution utilities, forcing overhauls in planning and operational practices. They can also create challenges in power system infrastructure planning and cause more frequent system operational violations (e.g., network voltage bounds and loading thresholds) if not properly integrated. The impacts on the localized power distribution grid from these emerging technologies manifest in increased infrastructure investments and erratic shifts in demand patterns. These impacts are not yet well understood, and analytic solutions are not readily available. To address these challenges, the National Renewable Energy Laboratory (NREL), in collaboration with BSES Rajdhani Power Ltd. (BRPL), developed an advanced power distribution system impact analysis framework of BRPL’s distribution system. This framework helps analyze the readiness of the power distribution network to accommodate emerging technologies and the potential opportunities they might introduce. The framework has been predominantly set up to evaluate distributed PV, BESS, and EVs. In this collaboration between NREL and BRPL, we developed and evaluated the framework on two distribution feeders in the BRPL territory for various scenarios of BESS and EVs. BESS are evaluated for their effectiveness on the grid to mitigate present and future feeder overloading scenarios, and they are subsequently analyzed for their costs compared to the costs of traditional upgradation measures. Scenarios include assessing the effects of EV density on grid infrastructure upgrades and interlinking EV management with BESS integration.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2021

How different power plant types contribute to electric grid reliability, resilience, and vulnerability: a comparative analytical framework

Abstract This work explores the dependability tradeoffs provided by the most common types of central power plants in the United States. Historically, the electricity sector has lacked consensus on how reliability , resilience , and vulnerability differ and how those metrics change depending on the power plant fleet composition. We propose distinct definitions for these metrics and an analytical framework to evaluate power plant fleet dependability. Using data analysis and literature review, we identify fifteen dependability attributes across which we rank eleven power plant types relative to natural gas combined-cycle (NGCC) plants. We use NGCC as the benchmark because it is common to many locations and is of relatively recent vintage. The framework shows that each power plant type has unique dependability benefits and drawbacks. We provide examples of how researchers may use the framework to evaluate grid dependability qualitatively under different scenarios. We find that assuming all attributes that contribute to grid dependability are equally important and additive, electric grid dependability is best supported when power plant fleets include a mixture of power generation technologies. Then, we discuss scenario characteristics that could alter the prioritization and relationships of attributes. We also find that if current capacity installation trends continue to favor low- and zero-carbon power plants, US power grids may benefit from increased resilience and reduced vulnerability at the cost of decreased reliability. We conclude by recommending methods for adapting the framework and quantifying relationships between attributes in individual scenarios.

Ramirez-Meyers, K. (ORCID:0000000291216952)↗