SEARCH · Engineering Papers

Results for “evaluation datasets”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 721 records · Page 40

All sources NASA NTRS DOE OSTI

DOE OSTI2021

Evaluating State Space Discovery by Persistent Cohomology in the Spatial Representation System

Persistent cohomology is a powerful technique for discovering topological structure in data. Strategies for its use in neuroscience are still undergoing development. We comprehensively and rigorously assess its performance in simulated neural recordings of the brain's spatial representation system. Grid, head direction, and conjunctive cell populations each span low-dimensional topological structures embedded in high-dimensional neural activity space. We evaluate the ability for persistent cohomology to discover these structures for different dataset dimensions, variations in spatial tuning, and forms of noise. We quantify its ability to decode simulated animal trajectories contained within these topological structures. We also identify regimes under which mixtures of populations form product topologies that can be detected. Our results reveal how dataset parameters affect the success of topological discovery and suggest principles for applying persistent cohomology, as well as persistent homology, to experimental neural recordings.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2023

Evaluation of Four Ground-based Retrievals of Cloud Droplet Number Concentration in Marine Stratocumulus with Aircraft In Situ Measurements

Cloud droplet number concentration (N d ) is crucial for understanding aerosol-cloud interactions (ACI) and associated radiative effects. We present evaluations of four ground-based N d retrievals based on comprehensive datasets from the Atmospheric Radiation Measurements (ARM) Aerosol and Cloud Experiments in the Eastern North Atlantic (ACE-ENA) field campaign. The N d retrieval methods use ARM ENA observatory ground-based remote sensing observations from a Micropulse lidar, Raman lidar, cloud radar, and the ARM NDROP Value-added Product (VAP), all of which also retrieve cloud effective radius (r e ). The retrievals are compared against aircraft measurements from the Fast-Cloud Droplet Probe (FCDP) and the Cloud and Aerosol Spectrometer (CAS) obtained from low-level marine boundary layer clouds on 12 flight days during summer and winter seasons. Additionally, the in situ measurements are used to validate the assumptions and characterizations used in the retrieval algorithms. Statistical comparisons of the probability distribution function (PDF) of the N d and cloud r e retrievals with aircraft measurements demonstrate that these retrievals align well with in situ measurements for overcast clouds, but they may substantially differ for broken clouds or clouds with low liquid water path (LWP). The retrievals are applied to four years of ground-based remote sensing measurements of overcast marine boundary layer clouds at the ARM ENA observatory to find that N d (r e ) values exhibit seasonal variations, with higher (lower) values during the summer season and lower (higher) values during the winter season. The ensemble of various retrievals using different measurements and retrieval algorithms such as those in this paper can help to quantify N d retrieval uncertainties and identify reliable N d retrieval scenarios. Of the retrieval methods, we recommend using the using the Micropulse lidar-based method given its good agreement with in situ measurements, it has less sensitivity to issues arising from precipitation and low cloud LWP/optical depth, and it has broad applicability by functioning for both day and nighttime conditions.

54 ENVIRONMENTAL SCIENCES↗

NASA NTRS1996

The Character of the Beaufort Sea Ice Cover From Active and Passive Microwave Observations

By definition, ice which survives the summer is classified as multiyear ice. Thus, multiyear ice concentration during the winter should be nearly equivalent to the ice concentration during the previous summer's minima. This condition provides a reasonable criterion for evaluation of ice concentration and ice type retrieval algorithms from remote sensing datasets. From SSM/I data, the NASA Team algorithm estimates the multiyear, first-year and total ice concentrations throughout the sea ice season using combinations of the polarization and spectral gradient ratios.

Beaufort Sea↗

NASA NTRS2019

Cellular Statistical Models of Broken Cloud Fields. Part IV: Effects of Pixel Size on Idealized Satellite Observations

In the fourth part of our “Cellular Statistical Models of Broken Cloud Fields” series we use the binary Markov processes framework for quantitative investigation of the effects of low resolution of idealized satellite observations on the statistics of the retrieved cloud masks. We assume that the cloud fields are Markovian and are characterized by the “actual” cloud fraction (CF) and scale length. We use two different models of observations: a simple discrete-point sampling and a more realistic “pixel” protocol. The latter is characterized by a state attribution function (SAF) which has the meaning of the probability that the pixel with a certain CF is declared cloudy in the observed cloud mask. The stochasticity of the SAF means that the cloud/clear attribution is not ideal and can be affected by external or unknown factors. We show that the observed cloud masks can be accurately described as Markov chains of pixels and use the master-matrix formalism (introduced in Part III of the series) for analytical computation of their parameters: the “observed” CF and scale length. This procedure allows us to establish a quantitative relationship (which is pixel-size dependent) between the actual and the observed cloud-field statistics. The feasibility of restoring the former from the latter is considered. The adequacy of our analytical approach to idealized observations is evaluated using numerical simulations. Comparison of the observed parameters of the simulated datasets with their theoretical expectations showed an agreement within 0.005 for the CF, while for the scale length it is within 1% in the sampling case and within 4% in the pixel case.

Mikhail D. Alexandrov↗

NASA NTRS2020

Kenya Food Security & Agriculture II: Utilizing NASA Earth Observations to Enhance Drought Warning Systems and Develop Capacity to Use the RHEAS Model in Kenya

Twenty-three counties in Kenya experience frequent drought, which damages agricultural productivity and threatens the health and well being of millions. NASA DEVELOP partnered with NASA SERVIR, the Regional Centre for Mapping of Resources for Development (RCMRD) and Kenya’s National Drought Management Authority (NDMA) to enhance drought-detection capacity using NASA Earth observations. Currently, the NDMA publishes monthly Early Warning Bulletins with drought conditions for each arid or semi-arid county in Kenya. These bulletins utilize the Vegetation Condition Index (VCI) from the Moderate Resolution ImagingSpectroradiometer (MODIS) as well as a variety of biophysical, social and economic drought indicators, but do not allow for advanced forecasting. To improve drought-monitoring capabilities, the team created a Combined Drought Indicator (CDI) using the Regional Hydrologic Extremes Assessment System (RHEAS) model and data from Aqua and Terra MODIS and the National Centers for Environmental Prediction. The CDI combined precipitation anomalies, soil moisture anomalies, evaporative stress, and VCI according to weights determined by principal component analysis. To evaluate the performance of the CDI across Kenya, the team compiled a dataset of drought events from historical reports and the NDMA’s records. Results suggested that the CDI detected drought earlier thanVCI alone. However, more validation is needed to ensure that the CDI accurately and consistently detects drought earlier than current warning systems. In order to better understand the behavior of individual indices and the potential for earlier drought detection, the team also conducted a time-lag analysis. The team found that VCI responds to drought, on average, one month later than most other indices included in the CDI, suggesting that incorporating additional indices could improve early drought warning systems in Kenya.

Food Security & Agriculture↗

NASA NTRS2020

Kenya Food Security & Agriculture II: Utilizing NASA Earth Observations to Enhance Drought Warning Systems and Develop Capacity to Use the RHEAS Model in Kenya

Twenty-three counties in Kenya experience frequent drought, which damages agricultural productivity and threatens the health and well being of millions. NASA DEVELOP partnered with NASA SERVIR, the Regional Centre for Mapping of Resources for Development (RCMRD) and Kenya’s National Drought Management Authority (NDMA) to enhance drought-detection capacity using NASA Earth observations. Currently, the NDMA publishes monthly Early Warning Bulletins with drought conditions for each arid or semi-arid county in Kenya. These bulletins utilize the Vegetation Condition Index (VCI) from the Moderate Resolution Imaging Spectroradiometer (MODIS) as well as a variety of biophysical, social and economic drought indicators, but do not allow for advanced forecasting. To improve drought-monitoring capabilities, the team created a Combined Drought Indicator (CDI) using the Regional Hydrologic Extremes Assessment System (RHEAS) model and data from Aqua and Terra MODIS and the National Centers for Environmental Prediction.The CDI combined precipitation anomalies, soil moisture anomalies, evaporative stress, and VCI according to weights determined by principal component analysis. To evaluate the performance of the CDI across Kenya, the team compiled a dataset of drought events from historical reports and the NDMA’s records. Results suggested that the CDI detected drought earlier than VCI alone. However, more validation is needed to ensure that the CDI accurately and consistently detects drought earlier than current warning systems. In order to better understand the behavior of individual indices and the potential for earlier drought detection, the team also conducted a time-lag analysis. The team found that VCI responds to drought, on average, one month later than most other indices included in the CDI, suggesting that incorporating additional indices could improve early drought warning systems in Kenya.

Food Security & Agriculture↗

NASA NTRSDate not supplied

A Survey of Earth Observation Data Needed by U.S. Federal Agencies

The Satellite Needs Working Group (SNWG), part of the U.S. Group on Earth Observations (USGEO), surveys U.S. Federal civil agencies every two years regarding the Earth observation data they need in order to accomplish their high-priority agency objectives. The 2020 SNWG survey marked the third assessment cycle (after 2016 and 2018). Information solicited by the survey includes the nature of the need, features and phenomena being observed, data measurement characteristics, satellites and data products the agency uses or plans to use, agency satisfaction with currently available data, and limitations to agency use of satellite data. For analysis purposes, agency needs are assigned to science Focus Areas such as atmospheric composition, carbon/ecosystems, solid Earth, water, etc. Analysis of the survey responses yields insights on the current landscape of Earth observation needs across the U.S. Government agencies. The evolution over multiple survey cycles has been analyzed by the SNWG Assessment team at the Interagency Implementation and Advanced Concepts Team (IMPACT) at Marshall Space Flight Center (MSFC). Key trends emerged in the agency-agnostic analysis that can be helpful to NASA decision-makers evaluating the ongoing, evolving, and emerging needs for NASA missions and instrument datasets expressed by participating agencies in the SNWG.

Katrina Virts↗

NASA NTRS2016

Detection of Chlorophyll and Leaf Area Index Dynamics from Sub-weekly Hyperspectral Imagery

Temporally rich hyperspectral time-series can provide unique time critical information on within-field variations in vegetation health and distribution needed by farmers to effectively optimize crop production. In this study, a dense time series of images were acquired from the Earth Observing-1 (EO-1) Hyperion sensor over an intensive farming area in the center of Saudi Arabia. After correction for atmospheric effects, optimal links between carefully selected explanatory hyperspectral vegetation indices and target vegetation characteristics were established using a machine learning approach. A dataset of in-situ measured leaf chlorophyll (Chll) and leaf area index (LAI), collected during five intensive field campaigns over a variety of crop types, were used to train the rule-based predictive models. The ability of the narrow-band hyperspectral reflectance information to robustly assess and discriminate dynamics in foliar biochemistry and biomass through empirical relationships were investigated. This also involved evaluations of the generalization and reproducibility of the predictions beyond the conditions of the training dataset. The very high temporal resolution of the satellite retrievals constituted a specifically intriguing feature that facilitated detection of total canopy Chl and LAI dynamics down to sub-weekly intervals. The study advocates the benefits associated with the availability of optimum spectral and temporal resolution spaceborne observations for agricultural management purposes.

Houborg, Rasmus↗

DOE OSTI2023

pyEGAF: An open-source Python library for the Evaluated Gamma-ray Activation File

The Evaluated Gamma-ray Activation File (EGAF) is one of the most comprehensive resources for thermal neutron-capture data. This database contains data from prompt gamma activation analysis measurements carried out in a consistent manner using the same experimental configuration at the Budapest Research Reactor for 245 isotopes. Although these valuable datasets have been freely available for many years, one of the drawbacks is the outdated and cryptic Evaluated Nuclear Structure Data File (ENSDF) format that is currently adopted for dissemination, making it difficult for users unfamiliar with the format to access and utilize the data contained therein. Furthermore, the ENSDF format does not readily lend itself to modern computational technologies and a parser is required to interpret the complicated mixed-record format. To help overcome these challenges, we have developed a translator to convert the ENSDF-formatted datasets into an open standard JavaScript Object Notation (JSON) format enabling accessibility to applications using different programming languages running in different environments. To compliment this effort, we have also developed an open-source software package implemented in Python, pyEGAF, that is designed to interact with the JSON data structures for general purpose access, manipulation, and analysis of the neutron-capture $\gamma$-ray data in EGAF. The new format, together with the pyEGAF library, greatly enhances access to the wider applications community where EGAF data may be useful or is required.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2020

Uncertainty Quantification of Machine Learning Predicted Creep Property of Alumina-Forming Austenitic Alloys

The development of machine learning (ML) approaches in materials science offers the opportunity to exploit existing engineering and developmental alloy datasets, such as Oak Ridge National Laboratory (ORNL)’s consistently measured creep-rupture dataset for alumina-forming austenitic (AFA) alloys, to accelerate their further development. As a first step toward achieving ML insights for improved alloy design, the potential sources of uncertainty and their impacts on ML output are examined. It is observed that the selection of algorithms and features as well as data sampling significantly affects the performance of ML models, either positively or negatively. Further, the performance of various ML models in predicting the creep properties of AFA alloys is compared, with further evaluation by assessment of a small set of new developmental AFA alloys that were not part of the training dataset. The present study demonstrates that uncertainty quantification (UQ) is essential in materials science for evaluating the performance of ML algorithms with specifically selected feature sets and obtaining a comprehensive understanding of their limitations and the resultant capability of effective prediction in complex materials systems.

36 MATERIALS SCIENCE↗

DOE OSTI2026

Public water supply infrastructure extensification and diversification in surface waters is insufficient to meet future demands in Texas

The data were developed to evaluate the capacity of existing and potential new surface water supply infrastructure to meet projected public water demands across districts in Texas under multiple future socioeconomic and climate scenarios. The database integrates hydrologic, water quality, infrastructure, energy, cost, demographic, and demand-projection information for candidate surface water supply locations. Candidate sites include stream reaches, waterbodies, reservoir surplus locations, and potential new reservoir sites. Water availability is characterized using historical and projected flow conditions, while site suitability is evaluated using five indicators: Water Availability Index (WAI), Water Quality Index (WQI), Energy Requirement Index (ERI), Water Treatment Cost (WTC), and Water Infrastructure Cost (WIC). The datasets include statewide candidate-site information, district-level demand projections under Shared Socioeconomic Pathways (SSPs), runoff-based allocation constraints, climate-stress metrics, and optimization outputs evaluating alternative infrastructure planning strategies. Optimization results compare Business-as-Usual (BAU) and All Surface Water (AllSW) demand-management approaches under both scaled and fixed cost-cap strategies. Associated validation datasets provide district-level feasibility assessments, infrastructure selection outcomes, cost-cap utilization, demand satisfaction metrics, and constraint diagnostics. Additional datasets quantify projected changes in storage and flow conditions as well as water availability stress for both existing and newly selected intake locations under the SSP5 scenario for mid-century and late-century climate conditions. Together, these datasets support assessment of the extent to which surface-water infrastructure expansion and diversification strategies can satisfy future public water demands while accounting for hydrologic, economic, and planning constraints across Texas. Dataset(s) Description Dataset_preoptimization.xlsx Comprehensive pre-optimization dataset containing candidate water-supply sites and associated hydrologic, water-quality, infrastructure, climate, demographic, runoff, and demand-projection variables used as inputs to the optimization analyses. Includes variable descriptions and the full statewide candidate-site database. District_level_site_selection.zip - Compressed archive containing all SSP-specific district-level optimization result files MESIO_ssp1_results.xlsx District-level site selection results for SSP1 (MESIO). Includes variable descriptions, BAU and AllSW site-selection results under scaled and fixed cost strategies, and district-level validation diagnostics. MESID_ssp2_results.xlsx District-level site selection results for SSP2 (MESID). Includes variable descriptions, BAU and AllSW site-selection results under scaled and fixed cost strategies, and district-level validation diagnostics. LCMRD_ssp3_results.xlsx District-level site selection results for SSP3 (LCMRD). Includes variable descriptions, BAU and AllSW site-selection results under scaled and fixed cost strategies, and district-level validation diagnostics. IRDev-Low_ssp4l_results.xlsx District-level site selection results for SSP4-Low (IRDev-Low). Includes variable descriptions, BAU and AllSW site-selection results under scaled and fixed cost strategies, and district-level validation diagnostics. IRDev-High_ssp4h_results.xlsx District-level site selection results for SSP4-High (IRDev-High). Includes variable descriptions, BAU and AllSW site-selection results under scaled and fixed cost strategies, and district-level validation diagnostics. RSIM_ssp5_results.xlsx District-level site selection results for SSP5 (RSIM). Includes variable descriptions, BAU and AllSW site-selection results under scaled and fixed cost strategies, and district-level validation diagnostics. tx_hydrological_stress.xlsx Hydrological stress dataset for existing and newly selected intake locations. Includes projected mid-century and late-century changes, gain/loss classifications, planning strategy information, and accompanying variable descriptions. Also includes water-stress metrics derived from historical and projected low-flow conditions.

Okoye, Perpetua I. (ORCID:0000000215545033)↗

DOE OSTI2020

Comparison of linear regression, k-nearest neighbour and random forest methods in airborne laser-scanning-based prediction of growing stock

Abstract In this study, for five sites around the world, we look at the effects of different model types and variable selection approaches on forest yield modelling performances in an area-based approach (ABA). We compared ordinary least squares regression (OLS), k-nearest neighbours (kNN) and random forest (RF). Our objective was to test if there are systematic differences in accuracy between OLS, kNN and RF in ABA predictions of growing stock volume. The analyses are based on a 5-fold cross-validation at five study sites: an eucalyptus plantation, a temperate forest and three different boreal forests. Two completely independent validation datasets were also available for two of the boreal sites. For the kNN, we evaluated multiple measures of distance including Euclidean, Mahalanobis, most similar neighbour (MSN) and an RF-based distance metric. The variable selection approaches we examined included a heuristic approach (for OLS, kNN and RF), exhaustive search among all combinations (OLS only) and all variables together (RF only). Performances varied by model type and variable selection approaches among sites. OLS and RF had similar accuracies and were more efficient than any of the kNN variants. Variable selection did not affect RF performance. Heuristic and exhaustive variable selection performed similarly for OLS. kNN fared the poorest amongst model types, and kNN with RF distance was prone to overfitting when compared with a validation dataset. Additional caution is therefore required when building kNN models for volume prediction though ABA, being preferable instead to opt for models based on OLS with some variable selection, or RF with all variables together.

Cosenza, Diogo N.↗

DOE OSTI2026

Heterogeneous Multi-Domain Dataset Synthesis to Facilitate Privacy and Risk Assessments in Smart City IoT

The emergence of the Smart Cities paradigm and the rapid expansion and integration of Internet of Things (IoT) technologies within this context have created unprecedented opportunities for high-resolution behavioral analytics, urban optimization, and context-aware services. However, this same proliferation intensifies privacy risks, particularly those arising from cross-modal data linkage across heterogeneous sensing platforms. To address these challenges, this paper introduces a comprehensive, statistically grounded framework for generating synthetic, multimodal IoT datasets tailored to Smart City research. The framework produces behaviorally plausible synthetic data suitable for preliminary privacy risk assessment and as a benchmark for future re-identification studies, as well as for evaluating algorithms in mobility modeling, urban informatics, and privacy-enhancing technologies. As part of our approach, we formalize probabilistic methods for synthesizing three heterogeneous and operationally relevant data streams—cellular mobility traces, payment terminal transaction logs, and Smart Retail nutrition records—capturing the behaviors of a large number of synthetically generated urban residents over a 12-week period. The framework integrates spatially explicit merchant selection using K-Dimensional (KD)-tree nearest-neighbor algorithms, temporally correlated anchor-based mobility simulation reflective of daily urban rhythms, and dietary-constraint filtering to preserve ecological validity in consumption patterns. In total, the system generates approximately 116 million mobility pings, 5.4 million transactions, and 1.9 million itemized purchases, yielding a reproducible benchmark for evaluating multimodal analytics, privacy-preserving computation, and secure IoT data-sharing protocols. To show the validity of this dataset, the underlying distributions of these residents were successfully validated against reported distributions in published research. We present preliminary uniqueness and cross-modal linkage indicators; comprehensive re-identification benchmarking against specific attack algorithms is planned as future work. This framework can be easily adapted to various scenarios of interest in Smart Cities and other IoT applications. By aligning methodological rigor with the operational needs of Smart City ecosystems, this work fills critical gaps in synthetic data generation for privacy-sensitive domains, including intelligent transportation systems, urban health informatics, and next-generation digital commerce infrastructures.

IoT↗

DOE OSTI2021

Modeling Buried Object Brightness and Visibility for Ground Penetrating Radar

Comparing the observed brightness of various buried objects is a straightforward way to characterize the performance of a ground penetrating radar (GPR) system. However, a limitation arises. A simple comparison of buried object brightness values does not disentangle the effects of the GPR system itself from the system's operating environment and the objects being observed. Therefore, with brightness values exhibiting an unknown synthesis of systemic, environmental, and object factors, GPR system analysis becomes a convoluted affair. In this work, we use an experimentally collected dataset of over 25,000 object observations from five different multistatic radar arrays to develop models of buried object brightness and control for these various effects. Additionally, our modeling efforts provide a means for quantifying the relative brightness of GPR systems, the objects they detect, and the physical properties of those objects that influence observed brightness. To evaluate the models' performance on new object observations, we repeatedly simulate fitting them to half the dataset and predicting the observed brightness values of the unseen half. In addition, we introduce a method for estimating the probability that individual observations constitute a visible object, which aids in failure analysis, performance characterization, and dataset cleaning.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2026

wa-hls4ml: A Benchmark and Surrogate Models for hls4ml Resource and Latency Estimation

As machine learning (ML) is increasingly implemented in hardware to address real-time challenges in scientific applications, the development of advanced toolchains has significantly reduced the time required to iterate on various designs. These advancements have solved major obstacles, but also exposed new challenges. For example, processes that were not previously considered bottlenecks, such as hardware synthesis, are becoming limiting factors in the rapid iteration of designs. To mitigate these emerging constraints, multiple efforts have been undertaken to develop an ML-based surrogate model that estimates resource usage of ML accelerator architectures. We introduce wa-hls4ml, a benchmark for ML accelerator resource and latency estimation, and its corresponding initial dataset of over 680,000 fully connected and convolutional neural networks, all synthesized using hls4ml and targeting Xilinx FPGAs. The benchmark evaluates the performance of resource and latency predictors against several common ML model architectures, primarily originating from scientific domains, as exemplar models, and the average performance across a subset of the dataset. Additionally, we introduce GNN- and transformer-based surrogate models that predict latency and resources for ML accelerators. We present the architecture and performance of the models and find that the models generally predict latency and resources for the 75% percentile within several percent of the synthesized resources on the synthetic test dataset.

Hawks, Benjamin [Fermilab] (ORCID:0000000157000288↗

DOE OSTI2025

wa-hls4ml: A Benchmark and Surrogate Models for hls4ml Resource and Latency Estimation

As machine learning (ML) is increasingly implemented in hardware to address real-time challenges in scientific applications, the development of advanced toolchains has significantly reduced the time required to iterate on various designs. These advancements have solved major obstacles, but also exposed new challenges. For example, processes that were not previously considered bottlenecks, such as hardware synthesis, are becoming limiting factors in the rapid iteration of designs. To mitigate these emerging constraints, multipleefforts have been undertaken to develop an ML-based surrogate model that estimates resource usage of synthesized ML accelerator architectures. We introduce wa-hls4ml, a benchmark for ML accelerator resource and latency estimation, and its corresponding initial dataset of over 680 000 fully connected and convolutional neural networks, all synthesized using hls4ml and targeting Xilinx FPGAs. The benchmark evaluates the performance of resource and latency predictors against several common ML model architectures, primarily originating from scientific domains, as exemplar models, and the average performance across a subset of the dataset. Additionally, we introduce GNN- and transformer-based surrogate models that predict latency and resources for ML accelerators. We present the architecture and performance of the models and find that the models generally predict latency and resources for the 75% percentile within several percent of the synthesized resources on the synthetic test dataset.

Hawks, Benjamin G. [Fermilab]↗

NASA NTRS2015

Climate Model Diagnostic Analyzer

The comprehensive and innovative evaluation of climate models with newly available global observations is critically needed for the improvement of climate model current-state representation and future-state predictability. A climate model diagnostic evaluation process requires physics-based multi-variable analyses that typically involve large-volume and heterogeneous datasets, making them both computation- and data-intensive. With an exploratory nature of climate data analyses and an explosive growth of datasets and service tools, scientists are struggling to keep track of their datasets, tools, and execution/study history, let alone sharing them with others. In response, we have developed a cloud-enabled, provenance-supported, web-service system called Climate Model Diagnostic Analyzer (CMDA). CMDA enables the physics-based, multivariable model performance evaluations and diagnoses through the comprehensive and synergistic use of multiple observational data, reanalysis data, and model outputs. At the same time, CMDA provides a crowd-sourcing space where scientists can organize their work efficiently and share their work with others. CMDA is empowered by many current state-of-the-art software packages in web service, provenance, and semantic search.

cloud computing↗

DOE OSTI2021

Deep-Learning-Derived Evaluation Metrics Enable Effective Benchmarking of Computational Tools for Phosphopeptide Identification

Tandem mass spectrometry (MS/MS)-based phosphoproteomics is a powerful technology for global phosphorylation analysis. However, applying four computational pipelines to a typical mass spectrometry (MS)-based phosphoproteomic dataset from a human cancer study, we observed a large discrepancy among the reported phosphopeptide identification and phosphosite localization results, underscoring a critical need for benchmarking. While efforts have been made to compare performance of computational pipelines using data from synthetic phosphopeptides, evaluations involving real application data have been largely limited to comparing the numbers of phosphopeptide identifications due to the lack of appropriate evaluation metrics. We investigated three deep learning-derived features as potential evaluation metrics: phosphosite probability, Delta RT and spectral similarity. Predicted phosphosite probability is computed by MusiteDeep, which provides high accuracy as previously reported; Delta RT is defined as the absolute retention time (RT) difference between RTs observed and predicted by AutoRT; and spectral similarity is defined as the Pearson’s correlation coefficient between spectra observed and predicted by pDeep2. Using a synthetic peptide dataset, we found that both Delta RT and spectral similarity provided excellent discrimination between correct and incorrect peptide-spectrum matches (PSMs) both when incorrect PSMs involved wrong peptide sequences and even when incorrect PSMs were caused by only incorrect phosphosite localization. Based on these results, we used all the three deep learning-derived features as evaluation metrics to compare different computational pipelines on diverse set of phosphoproteomic datasets and showed their utility in benchmarking performance of the pipelines. The benchmark metrics demonstrated in this study will enable users to select computational pipelines and parameters for routine analysis of phosphoproteomics data and will offer guidance for developers to improve computational methods.

59 BASIC BIOLOGICAL SCIENCES↗