Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data science workflows”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 253 records · Page 14
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Evaluation & Development of Algorithms & Techniques for Streaming Detector Readout

The advancement in microelectronics capabilities, computing, and data science in the last decade has been remarkable. With the start of the 12 GeV Science Program at Jefferson Lab (JLab) and the conceptual detector design for the upcoming Ion Collider (EIC) in mind, we have evaluated and developed the possibility of evolving and improving the existing nuclear science research workflow based on these advances. Specifically, we have prototyped components of streaming readout.

43 PARTICLE ACCELERATORS↗
DOE OSTI2022

Metabolome patterns identify active dechlorination in bioaugmentation consortium SDC-9™

Ultra-high performance liquid chromatography–high-resolution mass spectrometry (UPHLC–HRMS) is used to discover and monitor single or sets of biomarkers informing about metabolic processes of interest. The technique can detect 1000’s of molecules (i.e., metabolites) in a single instrument run and provide a measurement of the global metabolome, which could be a fingerprint of activity. Despite the power of this approach, technical challenges have hindered the effective use of metabolomics to interrogate microbial communities implicated in the removal of priority contaminants. Herein, our efforts to circumvent these challenges and apply this emerging systems biology technique to microbiomes relevant for contaminant biodegradation will be discussed. Chlorinated ethenes impact many contaminated sites, and detoxification can be achieved by organohalide-respiring bacteria, a process currently assessed by quantitative gene-centric tools (e.g., quantitative PCR). This laboratory study monitored the metabolome of the SDC-9™ bioaugmentation consortium during cis-1,2-dichloroethene (cDCE) conversion to vinyl chloride (VC) and nontoxic ethene. Untargeted metabolomics using an UHPLC-Orbitrap mass spectrometer and performed on SDC-9™ cultures at different stages of the reductive dechlorination process detected ~10,000 spectral features per sample arising from water-soluble molecules with both known and unknown structures. Multivariate statistical techniques including partial least squares-discriminate analysis (PLSDA) identified patterns of measurable spectral features (peak patterns) that correlated with dechlorination (in)activity, and ANOVA analyses identified 18 potential biomarkers for this process. Statistical clustering of samples with these 18 features identified dechlorination activity more reliably than clustering of samples based only on chlorinated ethene concentration and Dhc 16S rRNA gene abundance data, highlighting the potential value of metabolomic workflows as an innovative site assessment and bioremediation monitoring tool.

environmental monitoring↗
DOE OSTI2023

How Climate and Data Quality Impact Photovoltaic Performance Loss Rate Estimations

Different data pipelines and statistical methods are applied to photovoltaic (PV) performance datasets to quantify the performance loss rate (PLR). Since the real values of PLR are unknown, a variety of unvalidated values are reported. As such, the PV industry commonly assumes PLR based on statistically extracted ranges from the literature. However, the accuracy and uncertainty of PLR depend on several parameters including seasonality, local climatic conditions, and the response of a particular PV technology. In addition, the specific data pipeline and statistical method used affect the accuracy and uncertainty. To provide insights, a framework of (≈200 million) synthetic simulations of PV performance datasets using data from different climates is developed. Time series with known PLR and data quality are synthesized, and large parametric studies are conducted to examine the accuracy and uncertainty of different statistical approaches over the contiguous US, with an emphasis on the publicly available and “standardized” library, RdTools . In the results, it is confirmed that PLRs from RdTools are unbiased on average, but the accuracy and uncertainty of individual PLR estimates vary with climate zone, data quality, PV technology, and choice of analysis workflow. Best practices and improvement recommendations based on the findings of this study are provided.

14 SOLAR ENERGY↗
DOE OSTI2020

Volumetric analysis and mesh generation of real and artificial microstructural geometries

Producing a viable finite element mesh of realistic microstructural structural geometry is a critical step in analyzing the thermo-mechanical behavior of complex multi-material composites. Advancements in imaging technology such as micro computed tomography have allowed modelers to access high resolution mesoscale geometries for direct numerical simulation. However, converting from voxel based 3D images to usable finite element meshes has been challenging. A robust method including algorithms and software scripts for generating finite element meshes from 3D imaged microstructures is presented in this paper. It includes a routine for inserting cohesive elements around material interfaces to enable modeling of interface properties including delamination and damage. The algorithms and procedures presented in this method leverage currently available software packages for processing surface based geometry into volume based meshes. In addition to converting real geometry from physical imaging systems, algorithms for producing numerically generated and statistically equivalent microstructural geometry are also included. These artificial microstructures can be a valuable resource for modelers when physical specimens do not exist or are limited in quantity: Method establishes a workflow from voxel data to viable finite element mesh including interface information, and Includes a method for synthetic geometry generation based on metrics from real microstructures.

36 MATERIALS SCIENCE↗
DOE OSTI2026

JGI Archive and Metadata Organizer (JAMO) v2.0.0

JAMO (JGI Archive and Metadata Organizer) helps researchers keep large collections of scientific files organized, findable, and safe. It lets you submit files with consistent, template-driven metadata, bundle related files into sets, and track them as a group instead of one by one. As data ages, JAMO automatically moves it from fast disk to cost-saving tape and can bring it back when needed, keeping storage lean without losing access. A simple web/CLI workflow supports submitting, checking status, retrying, and updating metadata. Compared with generic storage, JAMO's strengths are: clear, searchable metadata tuned for science; set-level organization that mirrors real projects; and built-in lifecycle care (archive, purge, restore) so you don't have to manage those steps yourself.

Cassol, Daniela [Lawrence Berkeley National Labora↗
DOE OSTI2022

Scanning structural mapping at the Life Science X-ray Scattering Beamline

This work describes the instrumentation and software for microbeam scattering and structural mapping at the Life Science X-ray Scattering (LiX) beamline at NSLS-II. Using a two-stage focusing scheme, an adjustable beam size between a few micrometres and a fraction of a millimetre is produced at the sample position. Scattering data at small and wide angles are collected simultaneously on multiple Pilatus detectors. A recent addition of an in-vacuum Pilatus 900k detector, with the detector modules arranged in a C-shaped configuration, has improved the azimuthal angle coverage in the wide-angle data. As an option, fluorescence data can be collected simultaneously. Fly scans have been implemented to minimize the time interval between scattering patterns and to avoid unnecessary radiation damage to the sample. For weakly scattering samples, an in-vacuum sample environment has been developed here to minimize background scattering. Data processing for these measurements is highly sample-specific. To establish a generalized data process workflow, first the data are reduced to reciprocal coordinates at the time of data collection. The users can then quantify features of their choosing from these intermediate data and construct structural maps. As examples, results from in-vacuum mapping of onion epidermal cell walls and 2D tomographic sectioning of an intact poplar stem are presented.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Morphologically-Directed Raman Spectroscopy as an Analytical Method for Subvisible Particle Characterization in Therapeutic Protein Product Quality

Abstract Subvisible particles (SVPs) are a critical quality attribute of injectable therapeutic proteins (TPs) that needs to be controlled due to potential risks associated with drug product quality. The current compendial methods routinely used to analyze SVPs for lot release provide information on particle size and count. However, chemical identification of individual particles is also important to address root-cause analysis. Herein, we introduce Morphologically-Directed Raman Spectroscopy (MDRS) for SVP characterization of TPs. The following particles were used for method development: (1) polystyrene microspheres, a traditional standard used in industry; (2) photolithographic (SU-8); and (3) ethylene tetrafluoroethylene (ETFE) particles, candidate reference materials developed by NIST. In our study, MDRS rendered high-resolution images for the ETFE particles (> 90%) ranging from 19 to 100 μm in size, covering most of SVP range, and generated comparable morphology data to flow imaging microscopy. Our method was applied to characterize particles formed in stressed TPs and was able to chemically identify individual particles using Raman spectroscopy. MDRS was able to compare morphology and transparency properties of proteinaceous particles with reference materials. The data suggests MDRS may complement the current TPs SVP analysis system and product quality characterization workflow throughout development and commercial lifecycle.

Science & Technology - Other Topics↗
DOE OSTI2026

Data for Yield from Iowa’s first commercial miscanthus fields: implications of spatial variability for productivity and sustainability beyond research plots

This dataset contains biomass yield measurements and associated vegetation index data collected from commercial Miscanthus × giganteus fields in eastern Iowa during the 2022–2023 growing seasons. The data support the analyses presented in the article: “Yield From Iowa's First Commercial Miscanthus Fields: Implications of Spatial Variability for Productivity and Sustainability Beyond Research Plots.” We collected 105 ground-truth biomass samples from four mature commercial fields (>4 years old) covering 92.81 ha. Samples were taken from 3 m² quadrats that were hand-harvested in alignment with commercial harvest timing. Stem biomass (excluding leaves) was weighed, moisture-corrected, and converted to dry-matter yield expressed in Mg DM ha⁻¹. Sampling locations were selected to capture spatial variability visible in aerial imagery and were recorded using RTK GPS. Each biomass observation was paired with vegetation indices derived from high-resolution PlanetScope satellite imagery (3 m resolution). Images were acquired throughout the growing season, and indices were calculated to evaluate their ability to predict end-of-season biomass yield. Statistical and machine learning approaches were used to identify key predictors, and a linear regression model based on end-of-July Green Normalized Difference Vegetation Index (GNDVI) was developed and evaluated. This repository includes the data used in that modeling workflow. Management practices, economic data, full imagery time series, and additional methodological details are described in the associated publication and are not included here. The dataset consists of three comma-separated value (CSV) files: 1. Combine_Groundtruth_Yield_VI_22_23.csv This file contains ground-truth biomass yield measurements and associated key vegetation index values collected during the 2022 and 2023 growing seasons. Rows: 105 observations Columns: Year — Year of observation (2022 or 2023) Field — Field location identifier Sample_number — Unique sample identifier GNDVI_End_Jul — Green Normalized Difference Vegetation Index calculated at end of July GNDVI_End_Aug — Green Normalized Difference Vegetation Index calculated at end of August NDRE_End_Aug — Normalized Difference Red Edge index calculated at end of August Biomass_Stem_Yield_MgDM/ha — Measured stem biomass yield (megagrams dry matter per hectare) 2. trainData_GNDVI.csv This file contains the subset of observations used to train the predictive relationship between July GNDVI and biomass yield. Rows: 76 observations Columns: Unnamed: 0 — Row index retained from the original data processing workflow GNDVI_End_Jul — GNDVI at end of July Stem_Yield_MgDM/ha — Observed stem biomass yield (Mg DM ha⁻¹) 3. testData_GNDVI.csv This file contains the test dataset used to evaluate model performance. Rows: 29 observations Columns: Unnamed: 0 — Row index retained from the original data processing workflow GNDVI_End_Jul — GNDVI at end of July Predicted_Yield_MgDM/ha — Model-predicted stem biomass yield (Mg DM ha⁻¹) Observed_Yield_MgDM/ha — Measured stem biomass yield (Mg DM ha⁻¹)

Potential yield, yield gap, in-field management, y↗
DOE OSTI2022

Enabling FAIR data in Earth and environmental science with community-centric (meta)data reporting formats

Abstract Research can be more transparent and collaborative by using Findable, Accessible, Interoperable, and Reusable (FAIR) principles to publish Earth and environmental science data. Reporting formats—instructions, templates, and tools for consistently formatting data within a discipline—can help make data more accessible and reusable. However, the immense diversity of data types across Earth science disciplines makes development and adoption challenging. Here, we describe 11 community reporting formats for a diverse set of Earth science (meta)data including cross-domain metadata (dataset metadata, location metadata, sample metadata), file-formatting guidelines (file-level metadata, CSV files, terrestrial model data archiving), and domain-specific reporting formats for some biological, geochemical, and hydrological data (amplicon abundance tables, leaf-level gas exchange, soil respiration, water and sediment chemistry, sensor-based hydrologic measurements). More broadly, we provide guidelines that communities can use to create new (meta)data formats that integrate with their scientific workflows. Such reporting formats have the potential to accelerate scientific discovery and predictions by making it easier for data contributors to provide (meta)data that are more interoperable and reusable.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

A science paradigm shift is needed for Earth and Environmental Systems Sciences (EESS) to integrate Knowledge-Guided Artificial Intelligence (KGAI) and lead new EESS-KGAI theories

The focal area of this white paper is learning from complex data through the use of AI techniques and AI-derived model components. Specifically, we advocate for research programs to develop knowledge-guided AI (KGAI) in the Earth and Environmental Systems sciences (EESS) as a basic research paradigm that is separate from (but supports) any specific Earth system model, modeling components, and modeling workflows, and even separates from specific hypothesis-driven questions about individual Earth system processes.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

RTN-117: Image Calibration and Instrument Signal Removal for the First Year of the LSST

The NSF-DOE Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST) requires calibration products that provide uniform, stable, and accurate photometric and astrometric performance across the 3.2 gigapixel focal plane and throughout the 10-year survey. This paper details the algorithms and workflows used to produce instrument calibrations and remove instrumental artifacts for Data Preview 2 (DP2)---the first end-to-end processing demonstration using on-sky data with the LSST Camera (LSSTCam). We describe the verification, acceptance, and certification framework used to assess calibration quality and quantify residual systematics. We show baseline metrics on calibrated science images to evaluate the robustness of the calibration and instrument signature removal (ISR) data processing pipelines for DP2. Finally, we summarize the known limitations observed in DP2 production and outline expected algorithmic improvements for the first public LSST data release (Data Release~1, DR1).

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

MyCrunchGPT: A LLM Assisted Framework for Scientific Machine Learning

Scientific machine learning (SciML) has advanced recently across many different areas in computational science and engineering. Here, the objective is to integrate data and physics seamlessly without the need of employing elaborate and computationally taxing data assimilation schemes. However, preprocessing, problem formulation, code generation, postprocessing, and analysis are still time- consuming and may prevent SciML from wide applicability in industrial applications and in digital twin frameworks. Here, we integrate the various stages of SciML under the umbrella of ChatGPT, to formulate MyCrunchGPT, which plays the role of a conductor orchestrating the entire workflow of SciML based on simple prompts by the user. Specifically, we present two examples that demonstrate the potential use of MyCrunchGPT in optimizing airfoils in aerodynamics, and in obtaining flow fields in various geometries in interactive mode, with emphasis on the validation stage. To demonstrate the flow of the MyCrunchGPT, and create an infrastructure that can facilitate a broader vision, we built a web app based guided user interface, that includes options for a comprehensive summary report. The overall objective is to extend MyCrunchGPT to handle diverse problems in computational mechanics, design, optimization and controls, and general scientific computing tasks involved in SciML, hence using it as a research assistant tool but also as an educational tool. While here the examples focus on fluid mechanics, future versions will target solid mechanics and materials science, geophysics, systems biology, and bioinformatics.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Energy Material Network Data Hubs

In early 2015 the United States Department of Energy conceived of a consortium of collaborative bodies based on shared expertise, data, and resources that could be targeted towards the more difficult problems in energy materials research. The concept of virtual laboratories had been envisioned and discussed earlier in the decade in response to the advent of the Materials Genome Initiative and similar scientific thrusts. To be effective, any virtual laboratory needed a robust method for data management, communication, security, data sharing, dissemination, and demonstration to work efficiently and effectively for groups of remote researchers. With the accessibility of new, easily deployed cloud technology and software frameworks, such individual elements could be integrated, and the required collaboration architecture is now possible. The developers have leveraged open-source software frameworks, customized them, and merged them into a platform to enable collaborative energy materials science, regardless of the geographic dispersal of the people and resources. After five years in operations, the systems are demonstratively an effective platform for enabling research within the Energy Material Networks (EMN). This paper will show the design and development of a secured scientific data sharing platform, the ability to customize the system to support diverse workflows, and examples of the enabled research and results connected with some of the Energy Material Networks.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Enabling pan-repository reanalysis for big data science of public metabolomics data

Public untargeted metabolomics data is a growing resource for metabolite and phenotype discovery; however, accessing and utilizing these data across repositories pose significant challenges. Therefore, here we develop pan-repository universal identifiers and harmonized cross-repository metadata. This ecosystem facilitates discovery by integrating diverse data sources from public repositories including MetaboLights, Metabolomics Workbench, and GNPS/MassIVE. Our approach simplified data handling and unlocks previously inaccessible reanalysis workflows, fostering unmatched research opportunities.

El Abiead, Yasin↗
DOE OSTI2025

The microbiologist's guide to metaproteomics

Metaproteomics is an emerging approach for studying microbiomes, offering the ability to characterize proteins that underpin microbial functionality within diverse ecosystems. As the primary catalytic and structural components of microbiomes, proteins provide unique insights into the active processes and ecological roles of microbial communities. By integrating metaproteomics with other omics disciplines, researchers can gain a comprehensive understanding of microbial ecology, interactions, and functional dynamics. This review, developed by the Metaproteomics Initiative (www.metaproteomics.org), serves as a practical guide for both microbiome and proteomics researchers, presenting key principles, state-of-the-art methodologies, and analytical workflows essential to metaproteomics. Topics covered include experimental design, sample preparation, mass spectrometry techniques, data analysis strategies, and statistical approaches.

bioinformatics↗
NASA NTRS2026

Hydrology Copilot: A Cloud-Native Ai System for Hydrological Data Analysis

The emergence of AI-driven Earth observation systems promises to broaden access to petabyte-scale geospatial data beyond domain specialists. However, translating this vision into operational scientific infrastructure requires addressing fundamental challenges in data virtualization, code transparency, and domain-specific reasoning. We present Hydrology Copilot, a cloud-native AI framework for natural-language-driven analysis of Earth observation data. To demonstrate operational capabilities at scale, we implement the system using NASA's North American Land Data Assimilation System version 3 (NLDAS-3), which provides surface meteorological forcing and land-surface model output across North and Central America at 1-km resolution, from which drought diagnostics are derived. The system integrates five core contributions: (1) scalable data virtualization using Kerchunk-based cloud optimized access, achieving a 1.5 to 4.6 times improvement in I/O latency across benchmark queries spanning regional single-day extractions (4.6 times speedup) to continental monthly aggregations (1.5 times speedup); (2) transparent code generation through Microsoft Azure AI Foundry agents that expose executable Python workflows for scientific verification; (3) persistent conversational memory enabling multi-turn analytical discourse across sessions; (4) intelligent query validation that enforces dataset boundaries and resolves ambiguous requests before execution; and (5) a multi-agent architecture coordinating query parsing, code generation, and visualization. We evaluate the system through drought-monitoring workflows, demonstrating reliable code generation, accurate results validated against reference computations and the operational U.S. Drought Monitor, and efficient operation across increasingly complex tasks. By bridging natural-language interfaces with rigorous hydrological analysis, Hydrology Copilot advances beyond proof-of-concept demonstrations to provide a deployable framework for operational Earth science applications.

Data virtualization↗