Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Computational pipelines”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 415 records · Page 23
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Fisher Forecasting for the DESC with $\texttt{Augur}$

The Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST) has begun its ten-year survey of the entire visible southern hemisphere. To ensure robust cosmological measurements, computationally inexpensive investigations of modeling choices must be made to gauge the performance of proposed cosmological analyses. In this paper, we introduce the $\texttt{Augur}$ tool of the Dark Energy Science Collaboration (DESC), which provides Fisher forecasts for cosmological inference for the LSST using software frameworks designed for DESC science. We test the pipeline by comparing it to forecasts produced by external code and direct sampling of the posterior via nested sampling methods, finding good agreement between all methods. We additionally investigate a range of modeling and hyperparameter choices for a 3$\times$2pt investigation in harmonic space, providing users with diagnostics to obtain reliable forecasts. $\texttt{Augur}$ will be continually updated to be compatible with the other tools in the DESC software ecosystem as additional probes and functionality become available.

Rogozenski, Paul [Carnegie Mellon U.; Arizona U.] ↗
DOE OSTI2020

Hypergraph Analytics of Domain Name System Relationships

We report on the use of novel mathematical methods in hypergraph analytice over a large quantity of DNS data. Hypergraphs generalize graphs, as used in network science, to better model complex multiway relations in cyber data. Specifically, casting DNS data from Georgia Tech's ActiveDNS repository as hypergraphs allows us to fully represent the interactions between {\em collections} of domains and IP addresses. To facilitate large-scale analytics, we fielded an analytical pipeline of two capabilities. HyperNetX (HNX) is a Python package for the exploration and visualization of hypergraphs, acting as a frontend. For the backend, the Chapel HyperGraph Library (CHGL) is a library for high performance hypergraph analytics written in the exascale programming language Chapel. CHGL was used to process gigascale DNS data, performing compute-intensive calculations for data reduction and segmentation. Identified portions are then sent to HNX for both exploratory analysis and knowledge discovery targeting known tactics, techniques, and procedures.

Joslyn, Cliff A.↗
DOE OSTI2021

Time-Resolved Geometric Feature Tracking Elucidates Laser-Induced Keyhole Dynamics

In this report, during laser melting of metals, localized metal evaporation resulting in the formation of a keyhole shaped cavity can occur if processing conditions are chosen with high power density. An unstable keyhole can have deleterious effects in certain applications (e.g., laser powder bed fusion) as it increases the likelihood of producing defects such as porosity. In this work, we propose a pipeline that enables complete segmentation and extraction of various geometric features in keyholing conditions. In situ synchrotron high-speed X-ray visualization at the Advanced Photon Source provides large datasets of experimental images with a high spatio-temporal resolution across a range of laser parameters for Ti-6Al- 4 V. Computer vision image processing techniques were used to extract time-resolved quantitative geometric features (e.g., depth, width, front wall angle) throughout keyhole evolution which were subsequently analyzed to understand the relationship between the variation of local keyhole geometry and processing conditions. This analysis is the first to employ a data-driven approach to further our understanding of the keyholing process regime.

36 MATERIALS SCIENCE↗
DOE OSTI2024

Understanding Mixed Precision GEMM with MPGemmFI: Insights into Fault Resilience

Emerging deep learning workloads urgently need fast general matrix multiplication (GEMM). Thus, one of the critical features of machine-learning-specific accelerators such as NVIDIA Tensor Cores, AMD Matrix Cores, and Google TPUs is the support of mixed-precision enabled GEMM. For DNN models, lower-precision FP data formats and computation offer acceptable correctness but significant performance, area, and memory footprint improvement. While promising, the mixed-precision computation on error resilience remains unexplored. To this end, we develop a fault injection framework that systematically injects fault into the mixed-precision computation results. We investigate how the faults affect the accuracy of machine learning applications. Based on the characteristics of error resilience, we offer lightweight error detection and correction solutions that significantly improve the overall model accuracy by 75% if the models experience hardware faults. The solutions can be efficiently integrated into the accelerator's pipelines.

Fang, Bo↗
DOE OSTI2022

Naturally-meaningful and efficient descriptors: machine learning of material properties based on robust one-shot ab initio descriptors

Establishing a data-driven pipeline for the discovery of novel materials requires the engineering of material features that can be feasibly calculated and can be applied to predict a material’s target properties. Here we propose a new class of descriptors for describing crystal structures, which we term Robust One-Shot Ab initio (ROSA) descriptors. ROSA is computationally cheap and is shown to accurately predict a range of material properties. These simple and intuitive class of descriptors are generated from the energetics of a material at a low level of theory using an incomplete ab initio calculation. We demonstrate how the incorporation of ROSA descriptors in ML-based property prediction leads to accurate predictions over a wide range of crystals, amorphized crystals, metal–organic frameworks and molecules. We believe that the low computational cost and ease of use of these descriptors will significantly improve ML-based predictions.

36 MATERIALS SCIENCE↗
DOE OSTI2026

BICEP/Keck. XX. Component-separated Maps of the Polarized Cosmic Microwave Background and Thermal Dust Emission Using Planck and BICEP/Keck Observations through the 2018 Observing Season

We present component-separated polarization maps of the cosmic microwave background (CMB) and Galactic thermal dust emission, derived using data from the BICEP/Keck experiments through the 2018 observing season and Planck. By employing a maximum-likelihood method that utilizes observing matrices, we produce unbiased maps of the CMB and dust signals. We outline the computational challenges and demonstrate an efficient implementation of the component map estimator. We show methods to compute and characterize power spectra of these maps, opening up an alternative way to infer the tensor-to-scalar ratio from our data. We compare the results of this map-based separation method with the baseline BICEP/Keck analysis. Our analysis demonstrates consistency between the two methods, finding an 84% correlation between the pipelines.

cosmic inflation↗
DOE OSTI2026

FitCache: A Transparent Drop-In Framework for Multi-Tier Caching to Accelerate Distributed Deep Learning Workloads

Training in Deep learning (DL) remains highly compute- and data-intensive, with I/O becoming a critical bottleneck as models and datasets scale. Recent studies report that data loading can dominate training time, especially on large-scale HPC systems with shared parallel file systems (PFS). Existing caching approaches either rely on single-tier designs or require intrusive modifications to training pipelines, limiting their portability and effectiveness. In this work, we present FitCache, a transparent drop-in framework for multi-tier caching to accelerate distributed DL training by coordinating fast local memory (e.g., DRAM, Persistent Memory (PMem)) and NVMe as hierarchical caches atop PFS. Our design adapts to hardware diversity, i.e., if NVMe is missing, memory transparently acts as a caching tier, ensuring stable performance. FitCache transparently intercepts I/O requests and issues concurrent fetches across all tiers, returning data from the fastest responder without centralized metadata or static redirection paths. FitCache adapts to dynamic workloads and heterogeneous clusters while maintaining POSIX compatibility. Experiments on Frontier (2048 GPUs) and smaller research clusters show that FitCache reduces training time by up to 40% and per-batch I/O latency by up to 71.6% compared to Lustre Orion PFS, offering a drop-in solution for scalable DL training.

Hu, Guangxing [ORNL] (ORCID:0009000283203614)↗
DOE OSTI2025

Scalable Hybrid Learning Techniques for Scientific Data Compression

Data compression is becoming critical for storing scientific data because many scientific applications need to store large amounts of data and post process this data for scientific discovery. Unlike image and video compression algorithms that limit errors to primary data (PD), scientists require compression techniques that accurately preserve derived quantities of interest (QoIs). Here, this article presents a physics-informed compression technique implemented as an end-to-end, scalable, GPU-based pipeline for data compression that addresses this requirement. Our hybrid compression technique combines machine learning techniques and standard compression methods. Specifically, we combine an autoencoder, an error-bounded lossy compressor to provide guarantees on raw data error, and a constraint satisfaction post-processing step to preserve the QoIs within a minimal error (generally less than floating point error). The effectiveness of the data compression pipeline is demonstrated by compressing nuclear fusion simulation data generated by a large-scale fusion code, XGC, which produces hundreds of terabytes of data in a single day. Our approach works within the ADIOS framework and results in compression by a factor of more than 150 while requiring only a few percent of the computational resources necessary for generating the data, making the overall approach highly effective for practical scenarios.

ITER↗
DOE OSTI2021

Watermarks in stream processing systems: semantics and comparative analysis of Apache Flink and Google cloud dataflow

Streaming data processing is an exercise in taming disorder: from oftentimes huge torrents of information, we hope to extract powerful and timely analyses. But when dealing with streaming data, the unbounded and temporally disordered nature of real-world streams introduces a critical challenge: how does one reason about the completeness of a stream that never ends? In this paper, we present a comprehensive definition and analysis of watermarks, a key tool for reasoning about temporal completeness in infinite streams.First, we describe what watermarks are and why they are important, highlighting how they address a suite of stream processing needs that are poorly served by eventually-consistent approaches:• Computing a single correct answer, as in notifications.• Reasoning about a lack of data, as in dip detection.• Performing non-incremental processing over temporal subsets of an infinite stream, as in statistical anomaly detection with cubic spline models.• Safely and punctually garbage collecting obsolete inputs and intermediate state.• Surfacing a reliable signal of overall pipeline health.Second, we describe, evaluate, and compare the semantically equivalent, but starkly different, watermark implementations in two modern stream processing engines: Apache Flink and Google Cloud Dataflow.

Akidau, Tyler↗
DOE OSTI2022

Machine learning pipeline to predict defect behavior in metallic alloy systems

The interaction between defect and solute atoms is critical to the thermodynamic and kinetic behavior of metallic alloys under exposure to high-energy radiation, causing irradiation damage in materials. Radiation can generate non-equilibrium concentrations of point defects such as vacancies and interstitials. The excess point defects not only accelerate diffusional processes such as precipitation that cause radiation embrittlement, but also change the pathway of phase transformations, including nucleation processes. Understanding these defect behaviors is complicated by the challenge and complexity of addressing each possible local and discrete distribution of environments and chemical interactions around targeted defects-solute or solute-solute complexes. To resolve the challenge, machine learning regression techniques have emerged as powerful tools that can train and construct an energy model to accurately describe the chemical interactions of solutes and defects. In Fiscal Year 2022, the work focused on the workflow development and demonstration using machine learning regression, density functional theory, cluster expansion, and Monte Carlo simulation to predict the effects of ternary solute elements (e.g., aluminum and molybdenum) and point defects on the Cr-rich $\alpha^{\prime}$ precipitation in multicomponent FeCr model alloys. The computational outcomes include the prediction of the ternary phase diagram, vacancy formation energy for different compositions, and the effect of vacancies on the nucleation of Cr-rich clusters. The simulations predict a pronounced change of Cr solubility in bcc Fe by the addition of Al and the rejection of Al atoms from $\alpha^{\prime}$ precipitates. Additionally, the simulations show the formation of Cr-vacancy clusters as the initial nuclei for stable nucleation and growth of $\alpha^{\prime}$ particles. The results demonstrate important outcomes and applications of using machine learning pipeline to study model or commercial alloys with multicomponent solute species and point defects.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2024

ULTRA-effective labeling of tandem repeats in genomic sequence

In the age of long read sequencing, genomics researchers now have access to accurate repetitive DNA sequence (including satellites) that, due to the limitations of short read-sequencing, could previously be observed only as unmappable fragments. Tools that annotate repetitive sequence are now more important than ever, so that we can better understand newly uncovered repetitive sequences, and also so that we can mitigate errors in bioinformatic software caused by those repetitive sequences. To that end, we introduce the 1.0 release of our tool for identifying and annotating locally repetitive sequence, ULTRA Locates Tandemly Repetitive Areas (ULTRA). ULTRA is fast enough to use as part of an efficient annotation pipeline, produces state-of-the-art reliable coverage of repetitive regions containing many mutations, and provides interpretable statistics and labels for repetitive regions.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Machine Learning Techniques for Data Reduction of Climate Applications

Scientists conduct large-scale simulations to compute derived quantities-of-interest (QoI) from primary data. Often, QoI are linked to specific features, regions, or time intervals, such that data can be adaptively reduced without compromising the integrity of QoI. For many spatiotemporal applications, these QoI are binary in nature and represent presence or absence of a physical phenomenon. We present a pipelined compression approach that first uses neural-network-based techniques to derive regions where QoI are highly likely to be present. Then, we employ a Guaranteed Autoencoder (GAE) to compress data with differential error bounds. GAE uses QoI information to apply low-error compression to only these regions. This results in overall high compression ratios while still achieving downstream goals of simulation or data collections. Experimental results are presented for climate data generated from the E3SM Simulation model for downstream quantities such as tropical cyclone and atmospheric river detection and tracking. These results show that our approach is superior to comparable methods in the literature.

Li, Xiao [University of Florida]↗
DOE OSTI2024

High-performance data management for whole slide image analysis in digital pathology

When dealing with giga-pixel digital pathology in whole-slide imaging, a notable proportion of data records holds relevance during each analysis operation. For instance, when deploying an image analysis algorithm on whole-slide images (WSI), the computational bottleneck often lies in the input-output (I/O) system. This is particularly notable as patch-level processing introduces a considerable I/O load onto the computer system. However, this data management process could be further paralleled, given the typical independence of patch-level image processes across different patches. This paper details our endeavors in tackling this data access challenge by implementing the Adaptable IO System version 2 (ADIOS2). Our focus has been constructing and releasing a digital pathology-centric pipeline using ADIOS2, which facilitates streamlined data management across WSIs. Additionally, we’ve developed strategies aimed at curtailing data retrieval times. The performance evaluation encompasses two key scenarios: (1) a pure CPU-based image analysis scenario (“CPU scenario”), and (2) a GPU-based deep learning framework scenario (“GPU scenario”). Our findings reveal noteworthy outcomes. Under the CPU scenario, ADIOS2 showcases an impressive two-fold speed-up compared to the brute-force approach. In the GPU scenario, its performance stands on par with the cutting-edge GPU I/O acceleration framework, NVIDIA Magnum IO GPU Direct Storage (GDS). From what we know, this appears to be among the initial instances, if any, of utilizing ADIOS2 within the field of digital pathology. The source code has been made publicly available at https://github.com/hrlblab/adios.

Wang, Xiao↗
DOE OSTI2025

HP-MDR: High-performance and Portable Data Refactoring and Progressive Retrieval with Advanced GPUs

Scientific applications produce vast amounts of data, posing grand challenges in the underlying data management and analytic tasks. Progressive compression is a promising way to address this problem, as it allows for on-demand data retrieval with significantly reduced data movement cost. However, most existing progressive methods are designed for CPUs, leaving a gap for them to unleash the power of today’s heterogeneous computing systems with GPUs.In this work, we propose HP-MDR, a high-performance and portable data refactoring and progressive retrieval framework for GPUs. Our contributions are four-fold: (1) We carefully optimize the bitplane encoding and lossless encoding, two key stages in progressive methods, to achieve high performance on GPUs; (2) We propose pipeline optimization and incorporate it with data refactoring and progressive retrieval workflows to further enhance the performance for large data process; (3) We leverage our framework to enable high-performance data retrieval with guaranteed error control for common Quantities of Interest; (4) We evaluate HP-MDR and compare it with state of the arts using five real-world datasets. Experimental results demonstrate that HP-MDR delivers an average 13.68 × and 6.31 × throughput in data refactoring and progressive retrieval tasks, respectively. It also leads to 11.22 × throughput for recomposing required data representations under Quantity-of-Interest error control and 6.04 × performance for the corresponding end-to-end data retrieval, when compared with state-of-the-art solutions.

Li, Yanliang [University of Oregon]↗
DOE OSTI2026

Automated Classification of Vehicle Movements at Signalized Intersections Using Vehicle Trajectories

Accurate vehicle movement classification through signalized intersections is of paramount importance to the analysis of intersection performance and the optimization of traffic control strategies. Conventional techniques for tracking vehicle turning movements depend on infrastructure-based strategies like human counts, loop detectors, and video analytics, all of which are costly, prone to errors, and spatially constrained. High-frequency trajectory data can be utilized to determine vehicle movement patterns in a scalable and infrastructure-independent method due to the adoption of connected vehicles (CVs). In recent years, several studies have utilized connected vehicle data to generate performance measures. Most of the trajectory-based performance measures approaches, however, require map matching-i.e., extracting geospatial references from maps to identify the movements that individual vehicles make at a signalized intersection. These approaches are often time-consuming and hinder scalability since geographic features need to be provided for an analysis to be conducted. Map matching methods are prone to errors as different map versions change these geographic features. This research presents a novel automatic classification pipeline that uses CV trajectory data to classify vehicle movements at signalized crossings, specifically pass-through left-turn and right-turn maneuvers. The process starts by filtering trips that cross a spatial bounding box that has been defined at the target intersection. Approach and departure headings for each trajectory crossing the boundary are computed and are clustered together to identify dominant movements. The proposed algorithm is used to classify the movement of vehicles at 10 intersections in the state of California, and the results indicate that the algorithm can classify movements at these intersections with varying traffic volumes and road network configurations, all in a map-less framework with no need for conflation of vehicle trajectories to a digital base map.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2022

A Scalable Pipeline for Gigapixel Whole Slide Imaging Analysis on Leadership Class HPC Systems

Whole Slide Imaging (WSI) captures microscopic details of a patient's histopathological features at multiple res-olutions organized across different levels. Images produced by WSI are gigapixel-sized, and saving a single image in memory requires a few gigabytes which is scarce since a complicated model occupies tens of gigabytes. Performing a simple met-ric operation on these large images is also expensive. High-performance computing (HPC) can help us quickly analyze such large images using distributed training of complex deep learning models. One popular approach in analyzing these images is to divide a WSI image into smaller tiles (patches) and then train a simpler model with these reduced-sized but large numbers of patches. However, we need to solve three pre-processing challenges efficiently for pursuing this patch-based approach. 1) Creating small patches from a high-resolution image can result in a high number (hundreds of thousands per image) of patches. Storing and processing these images can be challenging due to a large number of I/O and arithmetic operations. To reduce I/Oand memory accesses, an optimal balance between the size and number of patches must exist to reduce I/O and memory accesses. 2) WSI images may have tiny annotated regions for cancer tissue and a significant portion with normal and fatty tissues; correct patch sampling should avoid dataset imbalance. 3) storing and retrieving many patches to and from disk storage might incur I/O latency while training a deep learning model. An efficient distributed data loader should reduce I/O latency during the training and inference steps. This paper explores these three challenges and provides empirical and algorithmic solutions deployed on the Summit supercomputer hosted at the Oak Ridge Leadership Computing Facility.

Dash, Sajal↗
DOE OSTI2022

Update on Covariance Data Testing Strategy at LANL [Slides]

LANL is working towards an ENDF/B-VIII.0-based Covariance Library, with several key goals and work processes outlined. This includes processing through NJOY’s ERRORR module, identifying and correcting mathematical and physical deficiencies, communicating across pipeline from evaluator to end user, understanding use cases and interpreting results, and releasing to customers. Their testing approach includes interaction, processing, checks (mathematical properties, constraints, and physical bounds), and error propagation.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Models and Processes to Extract Drug-like Molecules From Natural Language Text

Researchers worldwide are seeking to repurpose existing drugs or discover new drugs to counter the disease caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). A promising source of candidates for such studies is molecules that have been reported in the scientific literature to be drug-like in the context of viral research. However, this literature is too large for human review and features unusual vocabularies for which existing named entity recognition (NER) models are ineffective. We report here on a project that leverages both human and artificial intelligence to detect references to such molecules in free text. We present 1) a iterative model-in-the-loop method that makes judicious use of scarce human expertise in generating training data for a NER model, and 2) the application and evaluation of this method to the problem of identifying drug-like molecules in the COVID-19 Open Research Dataset Challenge (CORD-19) corpus of 198,875 papers. We show that by repeatedly presenting human labelers only with samples for which an evolving NER model is uncertain, our human-machine hybrid pipeline requires only modest amounts of non-expert human labeling time (tens of hours to label 1778 samples) to generate an NER model with an F-1 score of 80.5%—on par with that of non-expert humans—and when applied to CORD’19, identifies 10,912 putative drug-like molecules. This enriched the computational screening team’s targets by 3,591 molecules, of which 18 ranked in the top 0.1% of all 6.6 million molecules screened for docking against the 3CLPro protein.

60 APPLIED LIFE SCIENCES↗