Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Evaluation Framework”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

A Multi-Dimensional Benefit Assessment of Automated Mobility Platforms (AMP) for Large Facilities: Mobility, Energy, Equity, and Facility Management & Design

The goal of the automated mobility platforms (AMPs) initiative is to raise the bar of service regarding equity and sustainability for public mobility systems that are crucial to large facilities, and doing so using electrified, energy efficient technology. Using airports as an example, the rapid growth in air travel demand has led to facility expansions and congested terminals, which directly impacts equity (e.g., increased challenges for Passengers with Reduced Mobility [PRMs]) and sustainability - both of which are important metrics often overlooked during the engineering design process. Therefore, to evaluate systems and inform critical near- and long-term decisions more effectively, a holistic evaluation framework is proposed focused on four key areas: (1) mobility, with emphasis on travel time and accessibility within an airport, (2) environment, focused on energy consumption and greenhouse gas (GHG) emissions associated with intra-airport mobility, (3) equity, specifically to the PRM community, but with an eye to the whole of society, and (4) built environment, or the fundamental changes in building design enabled by different mobility systems for larger and more flexible, functional, and energy-efficient structures. Below, AMPs are defined, and each metric is discussed further, all with a focus on airport mobility.

ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2026

Antarctic ice sheet model comparison with uncurated geological constraints shows that higher spatial resolution improves deglacial reconstructions

Accurately reconstructing past changes to the shape and volume of the Antarctic ice sheet relies on the use of physically based and thus internally consistent ice sheet modeling, benchmarked against spatially limited geologic data. The challenge in model benchmarking against geologic data is diagnosing whether model-data misfits are the result of an inadequate model, inherently noisy or biased geologic data, and/or incorrect association between modeled quantities and geologic observations. In this work we address this challenge by (i) the development and use of a new model-data evaluation framework applied to an uncurated data set of geologic constraints, and (ii) nested high-spatial-resolution modeling designed to test the hypothesis that model resolution is an important limitation in matching geologic data. While previous approaches to model benchmarking employed highly curated datasets, our approach applies an automated screening and quality control algorithm to an uncurated public dataset of geochronological observations (specifically, cosmogenic-nuclide exposure-age measurements from glacial deposits in ice-free areas). This optimizes data utilization by including more geological constraints, reduces potential interpretive bias, and allows unsupervised assimilation of new data as they are collected. We also incorporate a nested model framework in which high-resolution domains are downscaled from a continent-wide ice sheet model. We highlight the application of this framework by applying these methods to a small ensemble of deglacial ice-sheet model simulations, and demonstrate that the nested approach improves the ability of model simulations to match exposure age data collected from areas of complex topography and ice flow. We develop a range of diagnostic model-data comparison metrics to provide more insight into model performance than possible from a single-valued misfit statistic, showing that different metrics capture different aspects of ice sheet deflation.

Geosciences↗
DOE OSTI2021

Call to Action for Global Access to and Harmonization of Quality Information of Individual Earth Science Datasets

Knowledge about the quality of data and metadata is important to support informed decisions on the (re)use of individual datasets and is an essential part of the ecosystem that supports open science. Quality assessments reflect the reliability and usability of data. They need to be consistently curated, fully traceable, and adequately documented, as these are crucial for sound decision- and policy-making efforts that rely on data. Quality assessments also need to be consistently represented and readily integrated across systems and tools to allow for improved sharing of information on quality at the dataset level for individual quality attribute or dimension. Although the need for assessing the quality of data and associated information is well recognized, methodologies for an evaluation framework and presentation of resultant quality information to end users may not have been comprehensively addressed within and across disciplines. Global interdisciplinary domain experts have come together to systematically explore needs, challenges and impacts of consistently curating and representing quality information through the entire lifecycle of a dataset. This paper describes the findings of that effort, argues the importance of sharing dataset quality information, calls for community action to develop practical guidelines, and outlines community recommendations for developing such guidelines. Practical guidelines will allow for global access to and harmonization of quality information at the level of individual Earth science datasets, which in turn will support open science.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
NASA NTRS1974

Electron beam, laser beam and plasma arc welding studies

This program was undertaken as an initial step in establishing an evaluation framework which would permit a priori selection of advanced welding processes for specific applications. To this end, a direct comparison of laser beam, electron beam and arc welding of Ti-6Al-4V alloy was undertaken. Ti-6Al-4V was selected for use in view of its established welding characteristics and its importance in aerospace applications.

Banas, C. M.↗
NASA NTRS1978

Forest resource information system

The author has identified the following significant results. A benchmark classification evaluation framework was implemented. The FRIS preprocessing activities were refined. Potential geo-based referencing systems were identified as components of FRIS.

Mroczynski, R. P.↗
NASA NTRS1997

Why are Formal Methods Not Used More Widely?

Despite extensive development over many years and significant demonstrated benefits, formal methods remain poorly accepted by industrial practitioners. Many reasons have been suggested for this situation such as a claim that they extent the development cycle, that they require difficult mathematics, that inadequate tools exist, and that they are incompatible with other software packages. There is little empirical evidence that any of these reasons is valid. The research presented here addresses the question of why formal methods are not used more widely. The approach used was to develop a formal specification for a safety-critical application using several specification notations and assess the results in a comprehensive evaluation framework. The results of the experiment suggests that there remain many impediments to the routine use of formal methods.

Knight, John C.↗
NASA NTRS2020

COSMIC: Content-based Onboard Summarization to Monitor Infrequent Change

Interplanetary exploration occurs at vast distancesthat severely limit communication bandwidth to spacecraft exploringother planets. It is possible to collect much morescientific data than can ever be downlinked given current communicationcapabilities. Therefore, we are developing a systemcalled COSMIC (Content-based Onboard Summarization toMonitor Infrequent Change) that will opportunistically analyzedata onboard a Mars orbiter to alert scientists when meaningfulchanges have occurred. COSMIC will allow future spacecraftto continuously collect data to search for rare, transient phenomenasuch as fresh impacts or seasonally changing polarlandforms under a constrained downlink budget. In this paper,we describe the overall goals and architecture of COSMIC,plans to enable specific scientific studies, label acquisition toenable supervised approaches to surface landform classification,a new machine learning evaluation framework for analyzingthe trade-offs between classifier accuracy and computationalrequirements, and lessons learned about constraints that COSMICwill face operating onboard a spacecraft. In particular, wediscuss design considerations surrounding computational andstorage constraints, change detection strategies, and localizingdetected landforms of interest within a global coordinate frame.Finally, we describe challenges and open research questions thatmust be addressed prior to deploying COSMIC.

Trockman, Asher↗
NASA NTRSDate not supplied

Better Constraining Supercooled Clouds Could Reduce Projected Warming Spread

The increase of climate sensitivity to rising greenhouse gas concentrations in the coupled model intercomparison project phase 6 (CMIP6) Earth system models (ESMs) compared to CMIP5 ESMs is primarily attributed to a larger extratropical cloud response to climate change, referred to as cloud feedback. The ratio of supercooled liquid cloud water relative to all cloud water, termed liquid phase ratio (LPR), which has also notably increased in many recent ESMs, is thought to be a primary driver of the extratropical cloud feedback increase. Unlike the preponderance of previous studies that compare native model LPR directly with observations, here we evaluate LPR over three ESM generations against Cloud–Aerosol Lidar and Infrared Pathfinder Satellite Observations (CALIPSO) using an instrument simulator approach, which mimics instrument limitations and uses consistent cloud definitions and resolutions. We find that current coupled model intercomparison project (CMIP) ESMs collectively simulate greater LPR than previous CMIP generations and overestimate LPR compared to observations, contrary to previous findings, likely driven by past inconsistent comparisons of ESM outputs with CALIPSO observations. We further show that greater LPR in ESM present-day climate is unexpectedly correlated with a smaller extratropical cloud feedback, attributable to a decrease of cloud amount exceeding the increase of cloud optical depth. Finally, our results suggest that improving constraints on model LPR using our evaluation framework would likely reduce the spread in CMIP6 climate sensitivities, owing to its effect on extratropical cloud feedback from supercooled clouds.

greenhouse gas concentrations↗
NASA NTRSDate not supplied

Critical Progressive Damage Analysis Model Features for Low-Velocity Impact on Thermoset and Thermoplastic Panels

A previous verification and validation framework evaluated the ability of various progressive damage analysis (PDA) tools to predict the mechanical response and damage state in post-buckled stiffened aerospace structures composed of fiber-reinforced thermoset composite materials. Herein, the ability of PDA tools to predict damage in thermoplastic composite materials subject to low-velocity impact (LVI) loading has been assessed. LVI tests and analyses have been conducted for similar panels, comprising thermoset or thermoplastic composite materials. Preliminary test/analysis correlations revealed that using the same combination of PDA model features and finite element model-building guidelines for the thermoset and thermoplastic LVI specimens resulted in underprediction of matrix crack lengths and delaminated area in the thermoplastic analyses despite yielding good correlations for the thermoset specimens. In this paper, the quality of the test/analysis correlations for analyses of thermoplastic LVI specimens performed with different combinations of PDA model features using the NASA CompDam material model are presented. Model features that were assessed in terms of effect on the quality of the test/analysis correlation include the representation of damageable interfaces, residual thermal stresses, and fiber damage.

Frank Leone↗
NASA NTRSDate not supplied

Critical Progressive Damage Analysis Model Features for Low-Velocity Impact on Thermoset and Thermoplastic Panels

A previous verification and validation framework evaluated the ability of various progressive damage analysis (PDA) tools to predict the mechanical response and damage state in post-buckled stiffened aerospace structures composed of fiber-reinforced thermoset composite materials. Herein, the ability of PDA tools to predict damage in thermoplastic composite materials subject to low-velocity impact (LVI) loading has been assessed. LVI tests and analyses have been conducted for similar panels, comprising thermoset or thermoplastic composite materials. Preliminary test/analysis correlations revealed that using the same combination of PDA model features and finite element model-building guidelines for the thermoset and thermoplastic LVI specimens resulted in underprediction of matrix crack lengths and delaminated area in the thermoplastic analyses despite yielding good correlations for the thermoset specimens. In this paper, the quality of the test/analysis correlations for analyses of thermoplastic LVI specimens performed with different combinations of PDA model features using the NASA CompDam material model are presented. Model features that were assessed in terms of effect on the quality of the test/analysis correlation include the representation of damageable interfaces, residual thermal stresses, and fiber damage.

Frank Leone↗
NASA NTRSDate not supplied

Simulated Multipath Using Software Generated GPS Signals

Depending on the environment, multipath can be one of the largest error sources contributing to degradation in Global Navigation Satellite System (GNSS) (e.g., GPS) performance. Multipath is a phenomenon that occurs as radio signals reflect off of surfaces, such as buildings, producing multiple copies of the original signal. When this occurs with GPS signals, it results in one or more delayed signals arriving at the receiver with or without the on-time/direct GPS signal. The receiver measures the composite of these signals which, depending on the severity of the multipath, can substantially degrade the accuracy of the receiver's calculated position. Multipath is commonly experienced in cities due to tall buildings and its mitigation is an ongoing area of study. This research demonstrates a novel approach for simulating GPS multipath through the modification of an open-source tool, GPS-SDR-SIM. The resulting additional testing capability could allow for improved development of multipath mitigating technologies. Currently, open-source tools for simulating GPS signals are available and can be used in the testing and evaluation of GPS receiver equipment. These tools can generate GPS signals that, when used by a GPS receiver, result in computation of a position solution that was pre-determined at the time of signal generation. That is, the signals produced are properly formed for the pre-determined location and result in the receiver reporting that position. This allows for a GPS receiver under test to be exposed to various simulated locations and conditions without having to be physically subjected to them. Additionally, while these signals are generated by a software simulation, they can be processed by real or software defined GPS receivers. This work utilizes the GPS-SDR-SIM software tool for GPS signal generation and while this tool does implement some sources of error that are inherent to GPS, it cannot inject multipath. GPS-SDR-SIM was modified in this effort to produce additional copies of signals with pre-determined delays. These additional delayed signals mimic multipath and represent what happens to GPS signals in the real world as they reflect off of surfaces and arrive at a receiver in place of or alongside the direct GPS signal. A successful proof of concept was prototyped and demonstrated using this modified version of GPS-SDR-SIM to produce simulated GPS signals as well as additional simulated multipath signals. The generated data was processed using a software defined GPS receiver and it was found that the introduction of simulated multipath signals successfully produced the expected characteristics of a composite multipath signal. Further maturation of this work could allow for the development of a GPS receiver testing and evaluation framework and aid in the development of multipath mitigating technologies.

GPS↗
DOE OSTI2020

Are You Resilient? Site Energy and Water Resilience Indicators

There is growing interest among operators of individual or co-located facilities to plan for utility disruptions caused by natural hazards and threats. Energy and water resilience at the site level can be characterized as the ability to withstand, adapt, and recover from an energy and/or water supply disruption. But how can a site’s energy and water resilience posture be measured? An evaluation framework with clear performance measures or indicators can help organizations establish what it means to be energy and water resilient and assess progress towards meeting resilience goals across sites and over time. Tracking a consistent set of indicators can help organizations identify problem areas that should be addressed and ascertain if energy and water resilience has improved as a result of investments or operational or policy changes. Resilience indicators also provide a means for organizations with geographically distributed sites to identify common problem areas that may require higher-level attention. This paper outlines an approach that organizations can use to measure site energy and water resilience. This approach was developed by Pacific Northwest National Laboratory for the U.S. Department of Energy’s Federal Energy Management Program as part of an effort to help federal agencies plan for energy and water resilience in a more systematic and rigorous way. The paper defines a general structure for defining energy and water resilience indicators, and provides example indicators and potential rating methods.

resilience, metrics for resilience, site resilienc↗
DOE OSTI2022

Evaluation of WRF-Solar Cloud Forecast Using the NSRDB: Preprint

Cloud forecast is a crucial component in predicting solar irradiance from numerical weather prediction (NWP) models. Assessing cloud properties from the NWP models requires significant work due to the need for high-quality data, spatial analysis covering model extent, and detailed analysis of model performance for different types of clouds. This study presents an evaluation of the WRF-Solar cloud forecast using the National Solar Radiation Database (NSRDB). We propose an evaluation framework applied to a single model prediction as well as ensemble-based forecasts. Various cloud detection metrics are calculated when comparing with the satellite-derived dataset. The mismatched clouds from the WRF-Solar model are quantified using nine cloud types classified by cloud top height and cloud optical depth. The results based on the WRF-Solar forecasts covering the entire U.S. for the full year of 2018 shows mismatched cloud frequency in the range of 8% - 46% for thick and high-level (deep convective) to thin and low-level (cumulus) clouds.

cloud mask forecast↗
DOE OSTI2024

A Multi-Dimensional Benefit Assessment of Automated Mobility Platforms (AMP) for Large Facilities

Automated Mobility Platform (AMP) is a general term for intelligently managed passenger mobility systems that leverage automation and electrification to improve system performance and sustainability through the integration of real-time data streams with advanced decision-making processes tailored for use in large facilities. Using airports as an example, the rapid growth in air travel demand has led to facility expansions and congested terminals, which directly impacts equity (e.g., increased challenges for passengers with reduced mobility [PRMs]) and sustainability - both of which are important metrics often neglected during the engineering design process. Therefore, to evaluate systems and inform critical decisions more effectively, a holistic evaluation framework is proposed that includes equity, sustainability, and infrastructure perspectives - referred to as a multidimensional benefit assessment. More specifically, the following analysis focuses on: (1) mobility, with emphasis on travel time and accessibility within an airport, contrasting existing systems with the proposed AMP system; (2) equity analysis, specifically to the PRM community, but with an eye to the whole of society; (3) energy consumption and greenhouse gas (GHG) emissions associated with intra-airport mobility, comparing and contrasting modes as well as impacts on overall airport operations efficiency; and (4) fundamental changes in building design enabled by AMPs for larger and more flexible, functional, and energy-efficient structures.

ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2024

Assessing Customer Experience and Business Models around Price-to-Device Communication and Smart Control Pathways in CalFlexHub

California is facing three major challenges in electrical grid operation: renewable overgeneration, steep evening ramping, and growing peak demand. The state has identified dynamic retail price response as a key strategy evidenced by CPUC’s Dynamic Rates proceeding and CEC’s Load Management Standards. Furthermore, the CEC launched a $16M “California Load Flexibility Research and Deployment Hub (CalFlexHub)” administered by Berkeley Lab to accelerate price-response flexible load technologies in buildings and EV charging. There are more than 16 laboratory and field demonstration projects in CalFlexHub, each demonstrating innovative automated price-response technologies. CalFlexHub tests various pathways through which hourly price signals and triggered control commands are communicated to load-flexible devices such as smart thermostats, heat pumps, water heaters, and EVs. We identified seven unique communication and control pathways, which involve combinations of third-party cloud, device OEM’s cloud, building central gateway, and local controller in between the price server and the load-flexible devices. It is important for utilities and policy makers to understand the long-term implications of each pathway in designing future programs and creating related policies and mandates for market transformation. We propose an evaluation framework including the following aspects: ● Functionality: connectivity and uptime, resilience, and optimization; ● Customer experience: simplicity in setup, troubleshooting support, continuity, customer choice, first cost, and ongoing cost; ● Business model and scalability: advance interoperability, holistic solution, bridge unique gap, customer base, and value streams and pricing structures. In this paper, we identify emerging business models associated with each communication pathway and discuss their positive features and challenges from the above aspects.

Liu, Jingjing↗
DOE OSTI2021

A modeling framework for designing and evaluating curbside traffic management policies at Dallas-Fort Worth International Airport

Emerging mobility technologies are changing the transportation system landscape. This is especially evident at airports, such as the Dallas-Fort Worth International Airport (DFW). Without careful analysis, these changes could lead to inefficient and costly airport operations. This paper presents a modeling framework that integrates travel mode encoding, demand projection, and microsimulation to enable airports to develop, simulate, and evaluate curbside traffic managements policies and measure their impact. Here, the framework is utilized to analyze several traffic scenarios and policies for DFW: a baseline scenario which represents DFW traffic pattern as observed in 2018 and projected to 2045, a transit network company (TNC) electrification policy, a TNC queuing policy, a policy that increased transit ridership, a bus-only policy which considers the use of only buses inside DFW, an autonomous vehicle (AV) policy which investigates the impact of autonomous vehicle (AV) adoption on airport operations, and an example COVID-19 scenario which models the impact of the COVID19 pandemic. The simulations’ results demonstrate that: increasing the DFW transit ridership postpones the need for airport curbside expansion the most; encouraging shared-mobility with the bus-only policy produces the most savings in curbside congestion delays; automation and electrification for all passenger vehicle trips to/from DFW generates the most saving in fuel consumption and emissions; and uncontrolled AV adoption incurs the highest increase in fuel consumption, delay, and emissions and could require immediate airport capacity extension. Without policy intervention or investment in additional infrastructure capacity, these results predict the current operations would face significant congestion on high demand days starting as early as 2028. While derived in close partnership with DFW, the methodology presented here can be generalized to any airport.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
NASA NTRS1998

A Catchment-Based Land Surface Model for GCMs and the Framework for its Evaluation

A new GCM-scale land surface modeling strategy that explicitly accounts for subgrid soil moisture variability and its effects on evaporation and runoff is now being explored. In a break from traditional modeling strategies, the continental surface is disaggregated into a mosaic of hydrological catchments, with boundaries that are not dictated by a regular grid but by topography. Within each catchment, the variability of soil moisture is deduced from TOP-MODEL equations with a special treatment of the unsaturated zone. This paper gives an overview of this new approach and presents the general framework for its off-line evaluation over North-America.

Ducharen, A.↗