Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “COMPARATIVE EVALUATIONS”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,153 records · Page 64
All sourcesNASA NTRSDOE OSTI
NASA NTRS2021

Evaluating the Impact of Model Resolutions and Cumulus Parameterization on Precipitation in NU-WRF: A Case Study in the Central Great Plains

Regional climate models are expected to exhibit improved skill at finer spatial resolutions due to improved representation of land surface heterogeneity. However, at spatial scales between 1 to 10 km (grey scales), these improvements are often illusive due to the competing benefits from spatial resolution and cumulus parameterization. This study provides insights into the impact of model resolution and cumulus parameterization on precipitation prediction in the Central Great Plains by using an object-based evaluation method. Our results show limited improvement solely from finer resolution but larger improvement without using the cumulus scheme at a 4-kmresolution. Compared to traditional evaluation methods, the object-based analysis shows that without the cumulus scheme the spatial properties of precipitation are better represented. In contrast, all model configurations show a dry bias in precipitation days and a tendency to produce widespread precipitation but with fewer hours with precipitationwhich indicates other shortcomings in the model.

Yuqi Zhang↗
NASA NTRS2009

Evaluation of Fast-Time Wake Vortex Prediction Models

Current fast-time wake models are reviewed and three basic types are defined. Predictions from several of the fast-time models are compared. Previous statistical evaluations of the APA-Sarpkaya and D2P fast-time models are discussed. Root Mean Square errors between fast-time model predictions and Lidar wake measurements are examined for a 24 hr period at Denver International Airport. Shortcomings in current methodology for evaluating wake errors are also discussed.

Proctor, Fred H.↗
DOE OSTI2022

Comparison and validation of the QuEChERSER mega-method for determination of per- and polyfluoroalkyl substances in foods by liquid chromatography with high-resolution and triple quadrupole mass spectrometry

Instances of food contamination with per- and polyfluoroalkyl substances (PFAS) continue to occur globally, but sample preparation and analytical methods are quite limited and often monitor for a small percentage of known PFAS. This study aimed to evaluate, validate, and compare performance of two instruments with the recently developed “quick, easy, cheap, effective, rugged, safe, efficient, and robust” (QuEChERSER) sample preparation mega-method – a method developed to monitor chemicals over a broad range of physicochemical properties. Initial evaluation of the QuEChERSER mega-method for determination of PFAS in food demonstrated recoveries, matrix interferences, and co-extractive removal comparable to (or better than) US Food and Drug Administration (FDA) and USDA Food Safety and Inspection Service (FSIS) methods. Subsequent validation of QuEChERSER in beef, catfish, chicken, pork, liquid eggs, and powdered eggs on a high-resolution mass spectrometer achieved acceptable recoveries (70–120%) and precision (RSDs ≤20%) for all 33 target analytes at the 1 and 5 ng g –1 levels and 67–88% of analytes at the 0.1 ng g –1 level, depending on the matrix. Additional validation was performed by tandem mass spectrometry on a triple quadrupole instrument. This approach provided no non-detects and better recoveries at the 0.1 ng g –1 level than the HRMS method but exhibited more variability at 1 and 5 ng g –1 spiking levels. Analysis of NIST SRMs 1946 and 1947 gave accuracies of 70–117%. Furthermore, these results demonstrate the capability of combining PFAS analysis with a mega-method previously validated for 350 analytes, while collecting non-target data for future retrospective analysis of emerging alternatives with a high-resolution mass spectrometry method.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
NASA NTRS2010

Initial Assessment of a Rapid Method of Calculating CEV Environmental Heating

An innovative method for rapidly calculating spacecraft environmental absorbed heats in planetary orbit is described. The method employs reading a database of pre-calculated orbital absorbed heats and adjusting those heats for desired orbit parameters. The approach differs from traditional Monte Carlo methods that are orbit based with a planet centered coordinate system. The database is based on a spacecraft centered coordinated system where the range of all possible sun and planet look angles are evaluated. In an example case 37,044 orbit configurations were analyzed for average orbital heats on selected spacecraft surfaces. Calculation time was under 2 minutes while a comparable Monte Carlo evaluation would have taken an estimated 26 hours

Pickney, John T.↗
NASA NTRS2013

Evaluating Global Emission Inventories of Biogenic Bromocarbons

Emissions of halogenated very short-lived substances (VSLS) are poorly constrained. However, their inclusion in global models is required to simulate a realistic inorganic bromine (Bry) loading in both the troposphere, where bromine chemistry perturbs global oxidizing capacity, and in the stratosphere, where it is a major sink for ozone (O3). We have performed simulations using a 3-D chemical transport model (CTM) including three top-down and a single bottom-up derived emission inventory of the major brominated VSLS bromoform (CHBr3) and dibromomethane (CH2Br2). We perform the first concerted evaluation of these inventories, comparing both the magnitude and spatial distribution of emissions. For a quantitative evaluation of each inventory, model output is compared with independent long-term observations at National Oceanic and Atmospheric Administration (NOAA) ground-based stations and with aircraft observations made during the NSF (National Science Foundation) HIAPER Pole-to-Pole Observations (HIPPO) project. For CHBr3, the mean absolute deviation between model and surface observation ranges from 0.22 (38 %) to 0.78 (115 %) parts per trillion (ppt) in the tropics, depending on emission inventory. For CH2Br2, the range is 0.17 (24 %) to 1.25 (167 %) ppt. We also use aircraft observations made during the 2011 Stratospheric Ozone: Halogen Impacts in a Varying Atmosphere (SHIVA) campaign, in the tropical western Pacific. Here, the performance of the various inventories also varies significantly, but overall the CTM is able to reproduce observed CHBr3 well in the free troposphere using an inventory based on observed sea-to-air fluxes. Finally, we identify the range of uncertainty associated with these VSLS emission inventories on stratospheric bromine loading due to VSLS (Br(VSLS/y)). Our simulations show Br(VSLS/y) ranges from approximately 4.0 to 8.0 ppt depending on the inventory. We report an optimized estimate at the lower end of this range (approximately 4 ppt) based on combining the CHBr3 and CH2Br2 inventories which give best agreement with the compilation of observations in the tropics.

bromocarbons↗
NASA NTRS2005

Advanced Method to Estimate Fuel Slosh Simulation Parameters

The nutation (wobble) of a spinning spacecraft in the presence of energy dissipation is a well-known problem in dynamics and is of particular concern for space missions. The nutation of a spacecraft spinning about its minor axis typically grows exponentially and the rate of growth is characterized by the Nutation Time Constant (NTC). For launch vehicles using spin-stabilized upper stages, fuel slosh in the spacecraft propellant tanks is usually the primary source of energy dissipation. For analytical prediction of the NTC this fuel slosh is commonly modeled using simple mechanical analogies such as pendulums or rigid rotors coupled to the spacecraft. Identifying model parameter values which adequately represent the sloshing dynamics is the most important step in obtaining an accurate NTC estimate. Analytic determination of the slosh model parameters has met with mixed success and is made even more difficult by the introduction of propellant management devices and elastomeric diaphragms. By subjecting full-sized fuel tanks with actual flight fuel loads to motion similar to that experienced in flight and measuring the forces experienced by the tanks these parameters can be determined experimentally. Currently, the identification of the model parameters is a laborious trial-and-error process in which the equations of motion for the mechanical analog are hand-derived, evaluated, and their results are compared with the experimental results. The proposed research is an effort to automate the process of identifying the parameters of the slosh model using a MATLAB/SimMechanics-based computer simulation of the experimental setup. Different parameter estimation and optimization approaches are evaluated and compared in order to arrive at a reliable and effective parameter identification process. To evaluate each parameter identification approach, a simple one-degree-of-freedom pendulum experiment is constructed and motion is induced using an electric motor. By applying the estimation approach to a simple, accurately modeled system, its effectiveness and accuracy can be evaluated. The same experimental setup can then be used with fluid-filled tanks to further evaluate the effectiveness of the process. Ultimately, the proven process can be applied to the full-sized spinning experimental setup to quickly and accurately determine the slosh model parameters for a particular spacecraft mission. Automating the parameter identification process will save time, allow more changes to be made to proposed designs, and lower the cost in the initial design stages.

Schlee, Keith↗
NASA NTRSDate not supplied

Evaluating the Trustworthiness of Deep Neural Networks in Deployment – A Comparative Study (Replicability Study)

As deep neural networks (DNNs) are increasingly used in safety critical applications, there is a growing concern for their trustworthiness. Even highly trained, high-performant networks are not 100% accurate. However, it is very difficult to predict their behaviour during deployment without ground truth. In this paper, we provide a comparative and replicability study on recent approaches that have been proposed to evaluate the trustworthiness of DNNs. We find that it is very difficult to run and reproduce the results for these approaches on their replication packages, and it is even more difficult to run the tools on artifacts other than their own. Further, it is difficult to compare the effectiveness of the tools, due to lack of clearly defined evaluation metrics. Our results indicate that more effort is needed in our research community to obtain sound techniques for evaluating the trustworthiness of neural networks in safety-critical domains. To this end, we contribute an evaluation framework that incorporates the considered approaches and enables evaluation on common benchmarks, using common metrics. Using this framework, we run a comparative study of the three approaches.

Trustworthy AI↗
DOE OSTI2023

Are System Baselines within OT Environments Feasible?

Critical infrastructure stakeholders need to baseline their systems to understand expected protocol communications.Baseline behaviors may vary based on operational context.Expected operations during a maintenance window, for example, may be different from normal operations.Furthermore, constructing system baselines for Industrial Control Systems (ICS) is difficult and time-consuming.ICS processes generate artifacts expressed across heterogeneous data sources such as network and device logs. There needs to be a corpus of data in order to develop and compare methods that evaluate the feasibility, performance, and generality of approaches to construct baselines for ICS events. Standalone repositories of network packet captures are insufficient to develop methods to classify or recognize operational events expressed across multiple data sources. Moreover, static data corpora do not enable researchers to compare the impact of changing the underlying system for which a baseline is being constructed and this limits the ability to evaluate the performance of system baselines given system changes (e.g. patches, configuration, maintenance events). In order to address these limitations within the community, this talk intends to promote discussion about the state of the practice of constructing baselines. In this manner, we can continue to understand requirements within industry that are not being met by current approaches to baseline construction. This talk builds on two previous talks on the topic of system baselines for OT environments. First, Weaver co-presented at the RSA Conference ICS Sandbox with Dan Gunter. The talk confirmed the need within industry to construct baselines across multiple types of data sources relative to the semantics of specific business processes. Second, Weaver presented at IEEE Security and Privacy Workshop on Language-Theoretic Security.

02 PETROLEUM↗
DOE OSTI2025

Insights from Optimizing HPL Performance on Exascale Systems: A Comparative Analysis of Panel Factorization

High performance LINPACK (HPL) remains the primary benchmark for evaluating supercomputing performance. It includes many parts with substantial internal complexity, and its performance is affected by a large number of parameters that interact in ways that are difficult to predict on large-scale heterogeneous supercomputer systems. We present a comprehensive performance analysis of HPL on Frontier, the world’s first exascale supercomputer, which achieved HPL performance of 1.35 exaflops. Through empirical parameter tuning, detailed modeling, and comparative evaluation, we uncover critical performance insights, share lessons learned, and outline best practices for effective parameter tuning on exascale systems. We introduce and evaluate two novel PDFACT strategies: a dedicated-thread (DT) variant and a GPU-based variant (GPUPDFACT) implementation using HIP cooperative groups, demonstrating that GPU-based factorization outperforms conventional CPU-based PDFACT on Frontier’s architecture. Our findings establish key performance factors for HPL on exascale systems and offer valuable guidance for future high-performance computing and benchmarking efforts.

Lu, Hao [ORNL] (ORCID:000000018941870X)↗
DOE OSTI2023

Machine Learning 5G Attack Detection in Programmable Logic

Machine learning-assisted network security may significantly contribute to securing 5G components. However, machine learning network security inference speeds generally require tens to hundreds of milliseconds thereby introducing significant latency in 5G operations. The inference latency can be reduced by deploying the machine learning model to programmable logic in a field programmable gate array (FPGA) at the cost of a small loss in accuracy. In order to quantify this loss, as well as to establish baseline performance inference speeds for programmable logic implementations, this work explores an autoencoder and a ß-variational autoencoder deployed on two different FPGA evaluation boards and compares accuracy and performance against an NVIDIA A100 GPU implementation. A publicly available 5G dataset containing 10 types of attacks along with normal traffic is introduced as part of the evaluation.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Do Neural Networks Trained with Topological Features Learn Different Internal Representations?

There is a growing body of work that leverages features extracted via topological data analysis to train machine learning models. While this field, sometimes known as topological machine learning (TML), has seen some notable successes, an understanding of how the process of learning from topological features differs from the process of learning from raw data is still limited. In this work, we begin to address one component of this larger issue by asking whether a model trained with topological features learns internal representations of data that are fundamentally different than those learned by a model trained with the original raw data. To quantify ``different'', we exploit two popular metrics that can be used to measure the similarity of the hidden representations of data within neural networks, neural stitching and centered kernel alignment. From these we draw a range of conclusions about how training with topological features does and does not change the representations that a model learns. Perhaps unsurprisingly, we find that structurally, the hidden representations of models trained and evaluated on topological features differ substantially compared to those trained and evaluated on the corresponding raw data. On the other hand, our experiments show that in some cases, these representations can be reconciled (at least to the degree required to solve the corresponding task) using a simple affine transformation. We conjecture that this means that neural networks trained on raw data may extract some limited topological features in the process of making predictions.

McGuire, Sarah L.↗
DOE OSTI2021

Analytical gradients for molecular-orbital-based machine learning

We report molecular-orbital-based machine learning (MOB-ML) enables the prediction of accurate correlation energies at the cost of obtaining molecular orbitals. Here, we present the derivation, implementation, and numerical demonstration of MOB-ML analytical nuclear gradients, which are formulated in a general Lagrangian framework to enforce orthogonality, localization, and Brillouin constraints on the molecular orbitals. The MOB-ML gradient framework is general with respect to the regression technique (e.g., Gaussian process regression or neural networks) and the MOB feature design. We show that MOB-ML gradients are highly accurate compared to other ML methods on the ISO17 dataset while only being trained on energies for hundreds of molecules compared to energies and gradients for hundreds of thousands of molecules for the other ML methods. The MOB-ML gradients are also shown to yield accurate optimized structures at a computational cost for the gradient evaluation that is comparable to a density-corrected density functional theory calculation.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
NASA NTRS1999

Situation Awareness and Workload Measures for SAFOR

The present research was conducted in support of the NASA Safe All-Weather Flight Operations for Rotorcraft (SAFOR) program. The purpose of the work was to investigate the utility of two measurement tools developed by the British Defense Evaluation Research Agency. These tools were a subjective workload assessment scale, the DRA Workload Scale (DRAWS), and a situation awareness measurement tool in which the crews self-evaluation of performance is compared against actual performance. These two measurement tools were evaluated in the context of a test of an innovative approach to alerting the crew by way of a helmet mounted display. The DRAWS was found to be usable, but it offered no advantages over extant scales, and it had only limited resolution. The performance self-evaluation metric of situation awareness was found to be highly effective.

DeMaio, Joe↗
NASA NTRS2019

Performance of Landsat-8 and Sentinel-2 Surface Reflectance Products for River Remote Sensing Retrievals of Chlorophyll-A and Turbidity

Rivers and other freshwater systems play a crucial role in ecosystems, industry, transportation and agriculture. Despite the more than 40 years of inland water observations made possible by optical remote sensing, a standardized reflectance product for inland waters is yet forthcoming. The aim of this work is to compare the standard USGS land surface reflectance product to two Landsat-8 and Sentinel-2 aquatic remote sensing reflectance products over the Amazon, Columbia and Mississippi rivers. Landsat-8 reflectance products from all three routines are then evaluated for their comparative performance in retrieving chlorophyll-a and turbidity in reference to shipborne, underway in situ validation measurements. The land surface product shows the best agreement (4 percent Mean Absolute Percent Difference) with field measurements of radiometry collected on the Amazon River and generates 36 percent higher reflectance values in the visible bands compared to aquatic methods (ACOLITE (Atmospheric Correction for OLI (Operational Land Imager) 'lite') and SeaDAS (Sea-viewing Wide Field-of-View Sensor (SeaWiFS) Data Analysis System)) with larger differences between land and aquatic products observed in Sentinel-2 (0.01 per steraradian) compared to Landsat-8 (0.001 per steraradian). Choice of atmospheric correction routine can bias Landsat-8 retrievals of chlorophyll-a and turbidity by as much as 59 percent and 35 percent respectively. Using a more restrictive time window for matching in situ and satellite imagery can reduce differences by 5–31 percent depending on correction technique. This work highlights the challenges of satellite retrievals over rivers and underscores the need for future optical and biogeochemical research aimed at improving our understanding of the absorbing and scattering properties of river water and their relationships to remote sensing reflectance.

Landsat-8↗
DOE OSTI2025

Optimizing Traffic Signal Control to Enhance Transportation Efficiency and Maximize Pedestrian Benefits in the Road Network

Increasing urban mobility requirements demand efficient transportation system strategies for both vehicular and pedestrian movement. This study enhances the Decentralized Graph-based Multi-Agent Reinforcement Learning (DGMARL) approach, originally tailored for vehicular traffic signal timing, to incorporate pedestrian traffic dynamics. The improved algorithm considers crucial metrics such as Eco_PI, assesses vehicle fuel consumption by factoring in stops and delays, and addresses pedestrian waiting time, crucial for system efficiency while acknowledging driver waiting time impact. Utilizing Digital Twin simulation along the MLK Smart Corridor in Chattanooga, Tennessee, the algorithm's performance is compared for various pedestrian control scenarios. To evaluate the effectiveness of DGMARL, this study compared DGMARL-enabled signal management with automated pedestrian traffic detection and an actuated signal management system (real-word baseline) with pedestrian recall, which predetermingly enforces a pedestrian phase every cycle. Findings indicate substantial improvements with DGMARL, showing a 28.29% enhancement in vehicle Eco_PI, a 60.55 % reduction in pedestrian waiting time, and a 55.74% decrease in driver stop delay, on average, compared to the baseline actuated signal timing plan.

Kumarasamy, Vijayalakshmi K [The University of Ten↗
NASA NTRS1981

Analysis and evaluation in the production process and equipment area of the low-cost solar array project

The effect of solar cell metallization pattern design on solar cell performance and the costs and performance effects of different metallization processes are discussed. Definitive design rules for the front metallization pattern for large area solar cells are presented. Chemical and physical deposition processes for metallization are described and compared. An economic evaluation of the 6 principal metallization options is presented. Instructions for preparing Format A cost data for solar cell manufacturing processes from UPPC forms for input into the SAMIC computer program are presented.

Wolf, M.↗
DOE OSTI2023

PARFUME/BISON Fission Product Release Predictions versus AGR-3/4 Heating Test Measurements

The fuel performance modeling codes PARFUME (PARticle FUel ModEl) and BISON were used to predict the release of fission products silver, cesium, and strontium from as-irradiated fuel compacts containing tristructural isotropic (TRISO) coated particles during heating tests post irradiation. The AGR-3/4 fuel compacts were irradiated as part of the third and fourth series of planned experiments to support the Advanced Gas Reactor (AGR) program. The heating tests were conducted at temperatures between 1200°C and 1700°C to simulate reactor accident conditions. The measured fission product release fractions from the heating tests were compared to modeling predictions calculated by PARFUME and BISON to evaluate how the codes compare to experimental results. Comparisons between the experimental measured fission product release fractions from silver, cesium and strontium indicate that both modeling codes overpredict the fission product release fractions demonstrating that the diffusivities used in the codes are overestimated. This results in a conservative estimate predicted by the codes when evaluating the fission product release relative to experimental data as it pertains to silver, cesium, and strontium.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2022

Inference for Nonparanormal Partial Correlation via Regularized Rank-Based Nodewise Regression

Abstract Partial correlation is a common tool in studying conditional dependence for Gaussian distributed data. However, partial correlation being zero may not be equivalent to conditional independence under non-Gaussian distributions. In this paper, we propose a statistical inference procedure for partial correlations under the high-dimensional nonparanormal (NPN) model where the observed data are normally distributed after certain monotone transformations. The NPN partial correlation is the partial correlation of the normal transformed data under the NPN model, which is a more general measure of conditional dependence. We estimate the NPN partial correlations by regularized nodewise regression based on the empirical ranks of the original data. A multiple testing procedure is proposed to identify the nonzero NPN partial correlations. The proposed method can be carried out by a simple coordinate descent algorithm for lasso optimization. It is easy-to-implement and computationally more efficient compared to the existing methods for estimating NPN graphical models. Theoretical results are developed to show the asymptotic normality of the proposed estimator and to justify the proposed multiple testing procedure. Numerical simulations and a case study on brain imaging data demonstrate the utility of the proposed procedure and evaluate its performance compared to the existing methods. Data used in preparation of this article were obtained from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database.

97 MATHEMATICS AND COMPUTING↗