Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Science Pipeline”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 505 records · Page 28
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

REDI – Readiness Engine for Data Integration

The Readiness Engine for Data Integration (REDI) is an open-source framework for automating, standardizing, and assessing the process of preparing scientific data for AI training. REDI implements a five-stage pipeline (ingest, preprocess, transform, structure, output) with per-stage provenance instrumentation via Flowcept, domain-aware transformation logic (PII anonymization, regridding, graph encoding, and more), and built-in readiness assessment and validation modes. REDI has been evaluated across climate, proteomics, materials science, and nuclear fusion datasets, demonstrating near-ideal parallel scaling to 100 nodes on OLCF's Frontier system. REDI is deployable as an agent-callable skill in coding environments such as Claude Code and OpenAI Codex, and is complemented by SetGo for FAIR compliance and catalog publication.

Brewer, Wesley [Oak Ridge National Laboratory (ORN↗
DOE OSTI2017

Algorithm sensitivity analysis and parameter tuning for tissue image segmentation pipelines

Motivation: Sensitivity analysis and parameter tuning are important processes in large-scale image analysis. They are very costly because the image analysis workflows are required to be executed several times to systematically correlate output variations with parameter changes or to tune parameters. An integrated solution with minimum user interaction that uses effective methodologies and high performance computing is required to scale these studies to large imaging datasets and expensive analysis workflows. Results: The experiments with two segmentation workflows show that the proposed approach can (i) quickly identify and prune parameters that are non-influential; (ii) search a small fraction (about 100 points) of the parameter search space with billions to trillions of points and improve the quality of segmentation results (Dice and Jaccard metrics) by as much as 1.42 x compared to the results from the default parameters; (iii) attain good scalability on a high performance cluster with several effective optimizations. Conclusions: Our work demonstrates the feasibility of performing sensitivity analyses, parameter studies and auto-tuning with large datasets. The proposed framework can enable the quantification of error estimations and output variations in image segmentation pipelines.

Biochemistry & Molecular Biology↗
DOE OSTI2023

In-depth analysis on parallel processing patterns for high-performance Dataframes

The Data Science domain has expanded monumentally in both research and industry communities during the past decade, predominantly owing to the Big Data revolution. Artificial Intelligence (AI) and Machine Learning (ML) are bringing more complexities to data engineering applications, which are now integrated into data processing pipelines to process terabytes of data. Typically, a significant amount of time is spent on data preprocessing in these pipelines, and hence improving its efficiency directly impacts the overall pipeline performance. The community has recently embraced the concept of Dataframes as the de-facto data structure for data representation and manipulation. However, the most widely used serial Dataframes today (R, pandas) experience performance limitations while working on even moderately large data sets. We believe that there is plenty of room for improvement by taking a look at this problem from a high-performance computing point of view. In a prior publication, we presented a set of parallel processing patterns for distributed dataframe operators and the reference runtime implementation, Cylon. In this paper, we are expanding on the initial concept by introducing a cost model for evaluating the said patterns. Furthermore, we evaluate the performance of Cylon on the ORNL Summit supercomputer.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

First detection of the BAO signal from early DESI data

We present the first detection of the baryon acoustic oscillations (BAOs) signal obtained using unblinded data collected during the initial 2 months of operations of the Stage-IV ground-based Dark Energy Spectroscopic Instrument (DESI). From a selected sample of 261 291 luminous red galaxies spanning the redshift interval 0.4 < z < 1.1 and covering 1651 square degrees with a 57.9 per cent completeness level, we report a ∼5σ level BAO detection and the measurement of the BAO location at a precision of 1.7 per cent. Using a bright galaxy sample of 109 523 galaxies in the redshift range 0.1 < z < 0.5, over 3677 square degrees with a 50.0 per cent completeness, we also detect the BAO feature at ∼3σ significance with a 2.6 per cent precision. These first BAO measurements represent an important milestone, acting as a quality control on the optimal performance of the complex robotically actuated, fibre-fed DESI spectrograph, as well as an early validation of the DESI spectroscopic pipeline and data management system. Based on these first promising results, we forecast that DESI is on target to achieve a high-significance BAO detection at sub-per cent precision with the completed 5-yr survey data, meeting the top-level science requirements on BAO measurements. This exquisite level of precision will set new standards in cosmology and confirm DESI as the most competitive BAO experiment for the remainder of this decade.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2025

Using Large Language Models to help customers monitor global threat data

Large Language Models have proven adept at answering general knowledge questions. To make these generative AI tools useful to our mission customers for monitoring global threats, the data sciences team at Sandia is utilizing retrieval augmented generation (RAG) techniques to customize these models with local data. The local data we use consists of data such as research articles and patent abstracts that we've collected over the last several years using automated pipelines.

Herzer, John Andrew [Sandia National Laboratories ↗
DOE OSTI2020

Viral Annotation Pipeline in KBase

This data analysis workflow demonstrates how to process a metagenome to identify viruses using VirSorter. Then, provides taxonomic classification using vConTACT2.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2018

sppIDer: A Species Identification Tool to Investigate Hybrid Genomes with High-Throughput Sequencing

The genomics era has expanded our knowledge about the diversity of the living world, yet harnessing high-throughput sequencing data to investigate alternative evolutionary trajectories, such as hybridization, is still challenging. Here we present sppIDer, a pipeline for the characterization of interspecies hybrids and pure species, that illuminates the complete composition of genomes. sppIDer maps short-read sequencing data to a combination genome built from reference genomes of several species of interest and assesses the genomic contribution and relative ploidy of each parental species, producing a series of colorful graphical outputs ready for publication. As a proof-of-concept, we use the genus Saccharomyces to detect and visualize both interspecies hybrids and pure strains, even with missing parental reference genomes. Through simulation, we show that sppIDer is robust to variable reference genome qualities and performs well with low-coverage data. We further demonstrate the power of this approach in plants, animals, and other fungi. sppIDer is robust to many different inputs and provides visually intuitive insight into genome composition that enables the rapid identification of species and their interspecies hybrids. sppIDer exists as a Docker image, which is a reusable, reproducible, transparent, and simple-to-run package that automates the pipeline and installation of the required dependencies (https://github.com/GLBRC/sppIDer; last accessed September 6, 2018).

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

Quantitative cross-species translators of cardiac myocyte electrophysiology: Model training, experimental validation, and applications

Animal experimentation is key in the evaluation of cardiac efficacy and safety of novel therapeutic compounds. However, interspecies differences in the mechanisms regulating excitation-contraction coupling can limit the translation of experimental findings from animal models to human physiology and undermine the assessment of drugs’ efficacy and safety. Here, we built a suite of translators for quantitatively mapping electrophysiological responses in ventricular myocytes across species. We trained these statistical operators using a broad dataset obtained by simulating populations of our biophysically detailed computational models of action potential and Ca 2+ transient in mouse, rabbit, and human. We then tested our translators against experimental data describing the response to stimuli, such as ion channel block, change in beating rate, and β-adrenergic challenge. We demonstrate that this approach is well suited to predicting the effects of perturbations across different species or experimental conditions and suggest its integration into mechanistic studies and drug development pipelines.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2022

Machine learning pipeline to predict defect behavior in metallic alloy systems

The interaction between defect and solute atoms is critical to the thermodynamic and kinetic behavior of metallic alloys under exposure to high-energy radiation, causing irradiation damage in materials. Radiation can generate non-equilibrium concentrations of point defects such as vacancies and interstitials. The excess point defects not only accelerate diffusional processes such as precipitation that cause radiation embrittlement, but also change the pathway of phase transformations, including nucleation processes. Understanding these defect behaviors is complicated by the challenge and complexity of addressing each possible local and discrete distribution of environments and chemical interactions around targeted defects-solute or solute-solute complexes. To resolve the challenge, machine learning regression techniques have emerged as powerful tools that can train and construct an energy model to accurately describe the chemical interactions of solutes and defects. In Fiscal Year 2022, the work focused on the workflow development and demonstration using machine learning regression, density functional theory, cluster expansion, and Monte Carlo simulation to predict the effects of ternary solute elements (e.g., aluminum and molybdenum) and point defects on the Cr-rich $\alpha^{\prime}$ precipitation in multicomponent FeCr model alloys. The computational outcomes include the prediction of the ternary phase diagram, vacancy formation energy for different compositions, and the effect of vacancies on the nucleation of Cr-rich clusters. The simulations predict a pronounced change of Cr solubility in bcc Fe by the addition of Al and the rejection of Al atoms from $\alpha^{\prime}$ precipitates. Additionally, the simulations show the formation of Cr-vacancy clusters as the initial nuclei for stable nucleation and growth of $\alpha^{\prime}$ particles. The results demonstrate important outcomes and applications of using machine learning pipeline to study model or commercial alloys with multicomponent solute species and point defects.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2021

The Last Journey. I. An Extreme-scale Simulation on the Mira Supercomputer

The Last Journey is a large-volume, gravity-only, cosmological N-body simulation evolving more than 1.24 trillion particles in a periodic box with a side length of 5.025 Gpc. It was implemented using the HACC simulation and analysis framework on the BG/Q system Mira. The cosmological parameters are chosen to be consistent with the results from the Planck satellite. A range of analysis tools have been run in situ to enable a diverse set of science projects and, at the same time, keep the resulting data amount manageable. Analysis outputs have been generated starting at redshift z ~ 10 to allow for construction of synthetic galaxy catalogs using a semianalytic modeling approach in postprocessing. As part of our in situ analysis pipeline, we employ a new method for tracking halo substructures, introducing the concept of subhalo cores. The production of multiwavelength synthetic sky maps is facilitated by generating particle light cones in situ, also beginning at z ~ 10. We provide an overview of the simulation setup and generated data products; a first set of analysis results is presented. A subset of the data is publicly available.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2018

DESCQA: An Automated Validation Framework for Synthetic Sky Catalogs

The use of high-quality simulated sky catalogs is essential for the success of cosmological surveys. The catalogs have diverse applications, such as investigating signatures of fundamental physics in cosmological observables, understanding the effect of systematic uncertainties on measured signals and testing mitigation strategies for reducing these uncertainties, aiding analysis pipeline development and testing, and survey strategy optimization. The list of applications is growing with improvements in the quality of the catalogs and the details that they can provide. Given the importance of simulated catalogs, it is critical to provide rigorous validation protocols that enable both catalog providers and users to assess the quality of the catalogs in a straightforward and comprehensive way. For this purpose, we have developed the DESCQA framework for the Large Synoptic Survey Telescope Dark Energy Science Collaboration as well as for the broader community. The goal of DESCQA is to enable the inspection, validation, and comparison of an inhomogeneous set of synthetic catalogs via the provision of a common interface within an automated framework. Here in this paper, we present the design concept and first implementation of DESCQA. In order to establish and demonstrate its full functionality we use a set of interim catalogs and validation tests. We highlight several important aspects, both technical and scientific, that require thoughtful consideration when designing a validation framework, including validation metrics and how these metrics impose requirements on the synthetic sky catalogs.

79 ASTRONOMY AND ASTROPHYSICS↗
NASA NTRS2020

Data Accountability and Uncertainty Analysis for the Mars Science Laboratory

This paper presents machine learning-based approaches to automate and optimize the detection of volume loss for the downlink process of telemetry data from the Mars Curiosity Rover. The Curiosity observes volume loss and data corruption, requiring re-transmits from the rover and Ground Data System Analysts (GDSA) to monitor the data flow. To resolve this issue, we created a data pipeline to accumulate data from various data sources in the downlink process and detect where the data is missed. In this paper, we benchmarked different methodologies based on the accuracy and excitability of them to identify whether a downlink data that is received to the ground system is complete or incomplete. Our results show that machine learning methods can improve the performance of the GDSA by 55% while the user can diagnose why data is missed and provide an explanation for the data accountability problem.

Chowdhury, Ameera↗
DOE OSTI2020

Hypergraph Analytics of Domain Name System Relationships

We report on the use of novel mathematical methods in hypergraph analytice over a large quantity of DNS data. Hypergraphs generalize graphs, as used in network science, to better model complex multiway relations in cyber data. Specifically, casting DNS data from Georgia Tech's ActiveDNS repository as hypergraphs allows us to fully represent the interactions between {\em collections} of domains and IP addresses. To facilitate large-scale analytics, we fielded an analytical pipeline of two capabilities. HyperNetX (HNX) is a Python package for the exploration and visualization of hypergraphs, acting as a frontend. For the backend, the Chapel HyperGraph Library (CHGL) is a library for high performance hypergraph analytics written in the exascale programming language Chapel. CHGL was used to process gigascale DNS data, performing compute-intensive calculations for data reduction and segmentation. Identified portions are then sent to HNX for both exploratory analysis and knowledge discovery targeting known tactics, techniques, and procedures.

Joslyn, Cliff A.↗
NASA NTRS2001

ROSAT Science Data Center

This report provides a summary of the Smithsonian Astrophysical Observatory (SAO) ROSAT SCIENCE DATA CENTER (RSDC) activities for the recent years of our contract. Details have already been reported in the monthly reports. The SAO was responsible for the High Resolution Imager (HRI) detector on ROSAT. We also provided and supported the HRI standard analysis software used in the pipeline processing (SASS). Working with our colleagues at the Max Planck in Garching Germany (MPE), we fixed bugs and provided enhancements. The last major effort in this area was the port from VMS/VAX to VMS/ALPHA architecture. In 1998, a timing bug was found in the HRI standard processing system which degraded the positional accuracy because events accessed incorrect aspect solutions. The bug was fixed and we developed off-line correction routines and provided them to the community. The Post Reduction Off-line Software (PROS) package was developed by SAO and runs in the IRAF environment. Although in recent years PROS was not a contractual responsibility of the RSDC, we continued to maintain the system and provided new capabilities such as the ability to deal with simulated AXAF data in preparation for the NASA call for proposals for Chandra. Our most recent activities in this area included the debugging necessary for newer versions of IRAF which broke some of our software. At SAO we have an operating version of PROS and hope to release a patch even though almost all functionality that was lost was subsequently recovered via an IRAF patch (i.e. most of our problems were caused by an IRAF bug).

Murray, Stephen↗
DOE OSTI2022

Toward designing effective exascale scientific computing workflows: experiences and best practices

Many fields within scientific computing have embraced advances in big-data analysis and machine learning, which often requires the deployment of large, distributed and complicated workflows that may combine training neural networks, performing simulations, running inference, and performing database queries and data analysis in asynchronous, parallel and pipelined execution frameworks. Such a shift has brought into focus the need for scalable, efficient workflow management solutions with reproducibility, error and provenance handling, traceability, and checkpoint-restart capabilities, among other needs. Here, we discuss challenges and best-practices for deploying exascale-generation computational science workflows on resources at the Oak Ridge Leadership Computing Facility (OLCF). We present our experiences with large-scale deployment of distributed workflows on the Summit supercomputer, including for bioinformatics and computational biophysics, materials science, and deep learning model optimization. We also present problems and solutions created by working within a Python-centric software base on traditional HPC systems, and discuss steps that will be required before the convergence of HPC, AI, and data science can be fully realized. Our results point to a wealth of exciting new possibilities for harnessing this convergence to tackle new scientific challenges.

Coletti, Mark↗
DOE OSTI2023

A Joint Roman Space Telescope and Rubin Observatory Synthetic Wide-Field Imaging Survey

We present and validate 20 deg(2) of overlapping synthetic imaging surveys representing the full depth of the Nancy Grace Roman Space Telescope High-Latitude Imaging Survey (HLIS) and 5 yr of observations of the Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST). The two synthetic surveys are summarized, with reference to the existing 300 deg(2) of LSST simulated imaging produced as part of Dark Energy Science Collaboration (DESC) Data Challenge 2 (DC2). Both synthetic surveys observe the same simulated DESC DC2 universe. For the synthetic Roman survey, we simulate for the first time fully chromatic images along with the detailed physics of the Sensor Chip Assemblies derived from lab measurements using the flight detectors. The simulated imaging and resulting pixel-level measurements of photometric properties of objects span a wavelength range of similar to 0.3-2.0 mu m. We also describe updates to the Roman simulation pipeline, changes in how astrophysical objects are simulated relative to the original DC2 simulations, and the resulting simulated Roman data products. We use these simulations to explore the relative fraction of unrecognized blends in LSST images, finding that 20-30 per cent of objects identified in LSST images with i-band magnitudes brighter than 25 can be identified as multiple objects in Roman images. These simulations provide a unique testing ground for the development and validation of joint pixel-level analysis techniques of ground- and space-based imaging data sets in the second half of the 2020s - in particular the case of joint Roman-LSST analyses.

gravitational lensing↗