Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “pipelining”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

PhytoOracle: Scalable, modular phenomics data processing pipelines

As phenomics data volume and dimensionality increase due to advancements in sensor technology, there is an urgent need to develop and implement scalable data processing pipelines. Current phenomics data processing pipelines lack modularity, extensibility, and processing distribution across sensor modalities and phenotyping platforms. To address these challenges, we developed PhytoOracle (PO), a suite of modular, scalable pipelines for processing large volumes of field phenomics RGB, thermal, PSII chlorophyll fluorescence 2D images, and 3D point clouds. PhytoOracle aims to ( i ) improve data processing efficiency; ( ii ) provide an extensible, reproducible computing framework; and ( iii ) enable data fusion of multi-modal phenomics data. PhytoOracle integrates open-source distributed computing frameworks for parallel processing on high-performance computing, cloud, and local computing environments. Each pipeline component is available as a standalone container, providing transferability, extensibility, and reproducibility. The PO pipeline extracts and associates individual plant traits across sensor modalities and collection time points, representing a unique multi-system approach to addressing the genotype-phenotype gap. To date, PO supports lettuce and sorghum phenotypic trait extraction, with a goal of widening the range of supported species in the future. At the maximum number of cores tested in this study (1,024 cores), PO processing times were: 235 minutes for 9,270 RGB images (140.7 GB), 235 minutes for 9,270 thermal images (5.4 GB), and 13 minutes for 39,678 PSII images (86.2 GB). These processing times represent end-to-end processing, from raw data to fully processed numerical phenotypic trait data. Repeatability values of 0.39-0.95 (bounding area), 0.81-0.95 (axis-aligned bounding volume), 0.79-0.94 (oriented bounding volume), 0.83-0.95 (plant height), and 0.81-0.95 (number of points) were observed in Field Scanalyzer data. We also show the ability of PO to process drone data with a repeatability of 0.55-0.95 (bounding area).

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

DRAGraces: A Pipeline for the GRACES High-resolution Spectrograph at Gemini

This paper describes the software Data Reduction and Analysis for GRACES (DRAGraces), which is a pipeline reducing spectra from Gemini Remote Access to the CFHT ESPaDOnS Spectrograph (GRACES) at the Gemini North Telescope. The code is written in the IDL language. It is designed to find all the GRACES frames in a given directory, automatically determine the list of bias, flat, arc, and science frames, and perform the whole reduction and extraction within a few minutes. We compare the output from DRAGraces with that of the Open source Pipeline for ESPaDOnS Reduction and Analysis (OPERA), a pipeline developed at the Canada–France–Hawaii Telescope (CFHT) that also can extract GRACES spectra. Both pipelines were developed completely independently, yet they give very similar extracted spectra. They both have their advantages and disadvantages. For instance, DRAGraces is more straightforward and easy to use and is less likely to be derailed by a parameter that needs to be tweaked, while OPERA offers a more careful extraction that can be significantly superior when the highest resolution is required and when the signal-to-noise ratio is low. One should compare both before deciding which one to use for their science. Yet, both pipelines deliver a fairly comparable resolution power (R ~ 52.8k and 36.6k for DRAGraces and R ~ 58k and 40k for OPERA in high- and low-resolution spectral modes, respectively), wavelength solution, and signal-to-noise ratio per resolution element.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
NASA NTRS1980

Assessing the impact of pipeline construction on coniferous wetlands in central Michigan with aerial photography

The Remote Sensing Project at Michigan State University is using repetitive aerial photography to assess the impact of pipeline construction on coniferous wetlands in central Michigan. Preliminary results indicate that ponding, dieback, windthrow, and vegetation changes are readily detectable on medium-scale aerial photography. It is found that the major effect of the pipeline construction is the alteration of the water level, either by flooding or dessication. The most serious damage generally occurs when pipelines cross seepage and spiring wetland types; specific damage is related to the impoundment of the natural water flow, producing flooding on the upflow side of the pipeline and dessication of these wetlands below the pipeline rights-of-way.

Kittleson, K. M.↗
NASA NTRS2008

Orchestrator Telemetry Processing Pipeline

Orchestrator is a software application infrastructure for telemetry monitoring, logging, processing, and distribution. The architecture has been applied to support operations of a variety of planetary rovers. Built in Java with the Eclipse Rich Client Platform, Orchestrator can run on most commonly used operating systems. The pipeline supports configurable parallel processing that can significantly reduce the time needed to process a large volume of data products. Processors in the pipeline implement a simple Java interface and declare their required input from upstream processors. Orchestrator is programmatically constructed by specifying a list of Java processor classes that are initiated at runtime to form the pipeline. Input dependencies are checked at runtime. Fault tolerance can be configured to attempt continuation of processing in the event of an error or failed input dependency if possible, or to abort further processing when an error is detected. This innovation also provides support for Java Message Service broadcasts of telemetry objects to clients and provides a file system and relational database logging of telemetry. Orchestrator supports remote monitoring and control of the pipeline using browser-based JMX controls and provides several integration paths for pre-compiled legacy data processors. At the time of this reporting, the Orchestrator architecture has been used by four NASA customers to build telemetry pipelines to support field operations. Example applications include high-volume stereo image capture and processing, simultaneous data monitoring and logging from multiple vehicles. Example telemetry processors used in field test operations support include vehicle position, attitude, articulation, GPS location, power, and stereo images.

Powell, Mark↗
NASA NTRS2010

Kepler Science Operations Center Pipeline Framework

The Kepler mission is designed to continuously monitor up to 170,000 stars at a 30 minute cadence for 3.5 years searching for Earth-size planets. The data are processed at the Science Operations Center (SOC) at NASA Ames Research Center. Because of the large volume of data and the memory and CPU-intensive nature of the analysis, significant computing hardware is required. We have developed generic pipeline framework software that is used to distribute and synchronize the processing across a cluster of CPUs and to manage the resulting products. The framework is written in Java and is therefore platform-independent, and scales from a single, standalone workstation (for development and research on small data sets) to a full cluster of homogeneous or heterogeneous hardware with minimal configuration changes. A plug-in architecture provides customized control of the unit of work without the need to modify the framework itself. Distributed transaction services provide for atomic storage of pipeline products for a unit of work across a relational database and the custom Kepler DB. Generic parameter management and data accountability services are provided to record the parameter values, software versions, and other meta-data used for each pipeline execution. A graphical console allows for the configuration, execution, and monitoring of pipelines. An alert and metrics subsystem is used to monitor the health and performance of the pipeline. The framework was developed for the Kepler project based on Kepler requirements, but the framework itself is generic and could be used for a variety of applications where these features are needed.

Klaus, Todd C.↗
NASA NTRS2011

Time-Distance Helioseismology Data-Analysis Pipeline for Helioseismic and Magnetic Imager Onboard Solar Dynamics Observatory (SDO-HMI) and Its Initial Results

The Helioseismic and Magnetic Imager onboard the Solar Dynamics Observatory (SDO/HMI) provides continuous full-disk observations of solar oscillations. We develop a data-analysis pipeline based on the time-distance helioseismology method to measure acoustic travel times using HMI Doppler-shift observations, and infer solar interior properties by inverting these measurements. The pipeline is used for routine production of near-real-time full-disk maps of subsurface wave-speed perturbations and horizontal flow velocities for depths ranging from 0 to 20 Mm, every eight hours. In addition, Carrington synoptic maps for the subsurface properties are made from these full-disk maps. The pipeline can also be used for selected target areas and time periods. We explain details of the pipeline organization and procedures, including processing of the HMI Doppler observations, measurements of the travel times, inversions, and constructions of the full-disk and synoptic maps. Some initial results from the pipeline, including full-disk flow maps, sunspot subsurface flow fields, and the interior rotation and meridional flow speeds, are presented.

Sun: helioseismology↗
NASA NTRS2017

Kepler Planet Detection Metrics: Pixel-Level Transit Injection Tests of Pipeline Detection Efficiency for Data Release 25

This document describes the results of the fourth pixel-level transit injection experiment, which was designed to measure the detection efficiency of both the Kepler pipeline (Jenkins 2002, 2010; Jenkins et al. 2017) and the Robovetter (Coughlin 2017). Previous transit injection experiments are described in Christiansen et al. (2013, 2015a,b, 2016).In order to calculate planet occurrence rates using a given Kepler planet catalogue, produced with a given version of the Kepler pipeline, we need to know the detection efficiency of that pipeline. This can be empirically determined by injecting a suite of simulated transit signals into the Kepler data, processing the data through the pipeline, and examining the distribution of successfully recovered transits. This document describes the results for the pixel-level transit injection experiment performed to accompany the final Q1-Q17 Data Release 25 (DR25) catalogue (Thompson et al. 2017)of the Kepler Objects of Interest. The catalogue was generated using the SOC pipeline version 9.3 and the DR25 Robovetter acting on the uniformly processed Q1-Q17 DR25 light curves (Thompson et al. 2016a) and assuming the Q1-Q17 DR25 Kepler stellar properties (Mathur et al. 2017).

Pixel-Level Transit Injection↗
NASA NTRSDate not supplied

Updates in Developing a Prototype Science Pipeline and Full-Volume, Global Hyperspectral Synthetic Data Sets for NASA’s Earth System Observatory’s Upcoming Surface, Biology and Geology Mission

The Surface Biology and Geology (SBG) mission recently passed mission confirmation review and has entered phase A – design and development. SBG will acquire high resolution solar-reflected spectroscopy and thermal infrared observations at a data rate of ~2.5 TB/day and generate products at ~40 TB/day. Given that the per-day volume is greater than NASA’s total extant airborne hyperspectral data collection, collecting, processing, disseminating, and exploiting the SBG data present new challenges. To meet these challenges, we have developed a prototype science pipeline and a full-volume global hyperspectral synthetic data set to help prepare for SBG’s flight (see poster GC42D-0730). Our science pipeline is based on the science processing technology developed for NASA’s Kepler and TESS planet-hunting missions. The pipeline infrastructure, Ziggy, provides a scalable architecture for robust, repeatable, and replicable science and application products that can be run on a range of systems from a laptop to the cloud or a supercomputer. Ziggy is compliant with NASA Procedural Requirement (NPR) 7150.2C, is at a technical readiness level (TRL) of 7 and has been released to github.com/nasa/ziggy. We integrated Ziggy with EO-1/Hyperion workflows to build a prototype pipeline and ingested the 17-year mission archive that provides globally sampled visible through shortwave infrared spectra that are representative of SBG data types and volumes. We fully implemented the first stage and processed the entire 55 TB Hyperion data set from the raw data (Level 0) to top-of-the-atmosphere radiance (Level 1R). We are currently evaluating the ISOFIT atmospheric correction module to convert the L1R data to surface reflectance (Level 2) before reprocessing the full data set to L2. Crosschecks are being performed with RadCalNet as well as with coincident observations by AVIRIS. We are also investigating modern methods for georectifying the Hyperion scenes. Finally, we describe an analysis of the cost to conduct forward processing and reprocessing campaigns for SBG on HECC with dedicated compute and storage resources using the resurrected Hyperion pipeline as a proxy for full-volume SBG data. The analysis demonstrates that SBG L0 data can be processed to L2 on HECC with full reprocessing campaigns every two years for ~$2.6M over a 7-year lifespan. Moreover, 69% of the system capacity would be available for other activities, possibly enabling future open-source science activities, including algorithm development, L3+ processing, .etc.

ESD↗
NASA NTRSDate not supplied

Towards A Better Measurement of eta-Earth and Beyond Via Modernizing the Kepler Pipeline: An Update

The measurement of the occurrence of rocky habitable-zone planets orbiting Sun-like stars (eta-Earth), is a fundamental quantity for guiding our search for habitable exoplanets. Despite being launched 15 years ago, NASA’s Kepler mission remains responsible for finding the majority of all known exoplanet candidates relevant to eta-Earth, ushering in a new era of exoplanet demographics studies and continuing to drive planet occurrence rate calculations. However, the paucity of detections of likely rocky planets in the habitable zones of their host stars remains a limiting factor for estimating eta-Earth. We describe our five-year project for modernizing the Kepler planet detection and vetting pipeline in order to produce a more complete and reliable exoplanet catalog, which will lead to more accurate and precise measurements of eta-Earth. First, we are currently porting the original Kepler pipeline code from MATLAB to Python. We will then describe new stellar catalogs based on Gaia and ground-based imaging data, and ways to improve the pipeline detection and vetting algorithms. We will provide an update on the current state of this work. When completed, we will use this new pipeline and catalog to calculate updated estimates of eta-Earth. The full, updated pipeline code in Python, as well as all our inputs and results, will be made available to the public for detailed exoplanet occurrence-rate and demographics studies.

kepler↗
DOE OSTI2022

Reconstruction of Large Radius Tracks with the Exa.TrkX pipeline

Particle tracking is a challenging pattern recognition task at the Large Hadron Collider (LHC) and the High Luminosity-LHC. Conventional algorithms, such as those based on the Kalman Filter, achieve excellent performance in reconstructing the prompt tracks from the collision points. However, they require dedicated configuration and additional computing time to efficiently reconstruct the large radius tracks created away from the collision points. We developed an end-to-end machine learning-based track finding algorithm for the HL-LHC, the Exa.TrkX pipeline. The pipeline is designed so as to be agnostic about global track positions. In this work, we study the performance of the Exa.TrkX pipeline for finding large radius tracks. Trained with all tracks in the event, the pipeline simultaneously reconstructs prompt tracks and large radius tracks with high efficiencies. This new capability offered by the Exa.TrkX pipeline may enable us to search for new physics in real time.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2023

Investigation of the Hydrogen Embrittlement of API 5L Natural Gas Pipeline Steels

Hydrogen ions produced during corrosion or through hydrogen blending into natural gas pipelines can lead to the degradation of ductility of metals used in these transmission pipelines. The effect of hydrogen on the mechanical properties of pipeline steels was studied for three grades of API 5L steels: X56, X65, and X100. These steels are either commonly used in or considered for use in natural gas transmission pipelines which are being considered for use in hydrogen blending. These steels were subjected to constant strain rate tensile testing after charging with electrochemically generated hydrogen. This presentation reports on work to extend the life of the natural gas pipeline network via understanding the effect of hydrogen embrittlement induced by corrosion processes.

Teeter, Lucas↗
DOE OSTI2020

Data analysis and modeling pipelines for controlled networked social science experiments

There is large interest in networked social science experiments for understanding human behavior at-scale. Significant effort is required to perform data analytics on experimental outputs and for computational modeling of custom experiments. Moreover, experiments and modeling are often performed in a cycle, enabling iterative experimental refinement and data modeling to uncover interesting insights and to generate/refute hypotheses about social behaviors. The current practice for social analysts is to develop tailor-made computer programs and analytical scripts for experiments and modeling. This often leads to inefficiencies and duplication of effort. In this work, we propose a pipeline framework to take a significant step towards overcoming these challenges. Our contribution is to describe the design and implementation of a software system to automate many of the steps involved in analyzing social science experimental data, building models to capture the behavior of human subjects, and providing data to test hypotheses. The proposed pipeline framework consists of formal models, formal algorithms, and theoretical models as the basis for the design and implementation. We propose a formal data model, such that if an experiment can be described in terms of this model, then our pipeline software can be used to analyze data efficiently. The merits of the proposed pipeline framework is elaborated by several case studies of networked social science experiments.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Pilot-Scale Validation of Distributed Optical Fiber Sensors for Underground Pipeline Monitoring

Distributed fiber optic sensing is a cutting-edge technology that has found extensive applications in the monitoring of Ensuring the safety, integrity, and operational efficiency of underground product pipelines is vital for maintaining the nation’s critical infrastructure. Monitoring parameters such as hoop strain, pressure, and acoustic vibrations is key to detecting potential leaks, intrusions, or structural issues. Distributed optical fiber sensor (DOFS) systems provide a compelling solution for continuous, real-time monitoring over long distances. This paper details the development and pilot-scale implementation of DOFS systems for underground pipeline monitoring, evolving from a proof-of-concept stage. Multiple custom-designed DOFS interrogator units—such as optical frequency-domain reflectometry (OFDR), Brillouin optical time-domain analysis (BOTDA), and multimodal interferometer-based fiber acoustic sensors—were employed to measure key parameters like hoop strain, pressure, and acoustic vibrations. The underground product pipeline's outer diameter is 30 inches, the wall thickness is 1.28 inches, and the 3-foot depth. The fiber deployment strategies, and sensing data acquisition methods for these systems are discussed. The results demonstrate the effectiveness of DOFS in detecting hoop strain, temperature changes, and acoustic vibrations, showcasing their potential for real-time monitoring and enhancing pipeline safety.

distributed fiber sensing↗
DOE OSTI2025

Automated Image Segmentation and Processing Pipeline Applied to X–Ray Computed Tomography Studies of Pitting Corrosion in Aluminum Wires

Understanding pitting corrosion is critical, yet its kinetics and morphology remain challenging to study from X-ray computed tomography (XCT) due to manual segmentation barriers. To address this, an automated pipeline leveraging deep learning for efficient large-scale XCT analysis is developed, revealing new corrosion insights. The pipeline enables pit segmentation, 3D reconstruction, statistical characterization, and a topological transformation for visualization. Here, the pipeline is applied to 87 648 XCT images capturing commercial purity aluminum (1100 Al) wire exposed to sodium chloride (NaCl) salt particles over a period of 122 h. The pipeline achieves complete feature extraction and statistical quantification across the entire XCT dataset, leveraging distributed computing environment for high efficiency. Global growth kinetics such as high-level stepwise sigmoidal volume loss patterns and granular individual pit developments are both captured for 36 detected pits. By combining automation, computer vision, and extensive XCT datasets, this research accelerates precise corrosion assessment to enable materials science discoveries at scale.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Alaska Liquid Natural Gas Pipeline Front-End Engineering & Design (Final Technical Report)

The Alaska Gasline Development Corporation (AGDC) is Alaska’s natural gas infrastructure development corporation established in 2013. AGDC’s mission is to maximize the benefit of Alaska’s vast North Slope natural gas resources for Alaskans through the development of infrastructure necessary to move the gas into local and international markets. AGDC was identified for a Congressionally Directed Spending (CDS) project for funding in the Energy and Water Development and Related Agencies Appropriations Act, 2023 under the heading: “Congressionally Directed Energy Efficiency and Renewable Energy Projects.” The CDS included $\$$4,000,000 of direct funding, with required match funds, to move the project forward. Alaska’s North Slope holds America’s largest proven and conventional natural gas supply. The integrated Alaska LNG Project will deliver 3.5 billion cubic feet of natural gas per day from Alaska’s North Slope gas fields to Alaskans as well as to a marine terminal located at tidewater in Cook Inlet. Alaska LNG is an integrated gas infrastructure project with three major components: a gas treatment plant (GTP) located at Prudhoe Bay, an 807-mile (1,287 km) gas pipeline (Mainline Pipeline) to Southcentral Alaska with interconnections for in-state gas use, and a natural gas liquefaction facility (LNG Facility) in Nikiski, Alaska. The integrated Alaska LNG Project has several strategic advantages including proven gas resources, existing upstream infrastructure, an advantageous arctic climate for LNG production, proximity to LNG markets, a track record of reliability from a state that first began exporting LNG to Japan in 1969, and broad support from Alaskans. North Slope natural gas is a conventional resource and can be produced with minimal drilling at a fraction of the carbon dioxide emissions of shale gas from the Lower 48 states. Through the development of the Alaska LNG Project, Alaska can provide energy security to Alaskans and a stable source of LNG to the Asia-Pacific region for generations. The Alaska LNG Project has been progressed through Pre-Front-End Engineering Design (Pre-FEED) and has obtained all major federal and State of Alaska permits and authorizations to construct the project, including the Federal Energy Regulatory Commission (FERC) Order Granting Authorization Under Section 3 of the Natural Gas Act. On September 5, 2024, the U.S. Department of Energy (DOE), National Energy Technology Laboratory (NETL) awarded Project No. DE-FE0032307 to AGDC with the objective to progress the project to Front-End Engineering Design (FEED) entry for the Alaska LNG Project Phase 1 Pipeline. The award Start Date was made effective July 1, 2023, with a Period of Performance through June 30, 2025. On March 27, 2025, AGDC announced the execution of definitive commercial agreements with Glenfarne Alaska LNG, LLC, an affiliate of Glenfarne Group, LLC, (together as “Glenfarne”), to lead the development of the Alaska LNG Project and enter FEED for the Phase 1 Pipeline. Project activities are now funded and directed by this private sector partner who holds a 75% interest in 8 Star Alaska, LLC (8 Star). 8 Star holds the assets of the Alaska LNG Project. As planned, AGDC continues to hold 25% minority interest in 8 Star and will play a governance role moving forward with Alaska LNG. This definitive commercial agreement milestone led to the successful completion of AGDC’s Statement of Project Objectives (SOPO) for FEED entry and led to the completion of DOE Project No. DE-FE0032307. At conclusion of the SOPO, AGDC also reached the award’s maximum federal cost share of $\$$4,000,000. AGDC is, therefore, providing Final Technical Report to close out DOE Project No. DE-FE0032307.

02 PETROLEUM↗
DOE OSTI2023

Quasi-Distributed Fiber Sensor-Based Approach for Pipeline Health Monitoring: Generating and Analyzing Physics-Based Simulation Datasets for Classification

This study presents a framework for detecting mechanical damage in pipelines, focusing on generating simulated data and sampling to emulate distributed acoustic sensing (DAS) system responses. The workflow transforms simulated ultrasonic guided wave (UGW) responses into DAS or quasi-DAS system responses to create a physically robust dataset for pipeline event classification, including welds, clips, and corrosion defects. This investigation examines the effects of sensing systems and noise on classification performance, emphasizing the importance of selecting the appropriate sensing system for a specific application. The framework shows the robustness of different sensor number deployments to experimentally relevant noise levels, demonstrating its applicability in real-world scenarios where noise is present. Overall, this study contributes to the development of a more reliable and effective method for detecting mechanical damage to pipelines by emphasizing the generation and utilization of simulated DAS system responses for pipeline classification efforts. The results on the effects of sensing systems and noise on classification performance further enhance the robustness and reliability of the framework.

36 MATERIALS SCIENCE↗
NASA NTRS2012

The HEASARC Swift Gamma-Ray Burst Archive: The Pipeline and the Catalog

Since its launch in late 2004, the Swift satellite triggered or observed an average of one gamma-ray burst (GRB) every 3 days, for a total of 771 GRBs by 2012 January. Here, we report the development of a pipeline that semi automatically performs the data-reduction and data-analysis processes for the three instruments on board Swift (BAT, XRT, UVOT). The pipeline is written in Perl, and it uses only HEAsoft tools and can be used to perform the analysis of a majority of the point-like objects (e.g., GRBs, active galactic nuclei, pulsars) observed by Swift. We run the pipeline on the GRBs, and we present a database containing the screened data, the output products, and the results of our ongoing analysis. Furthermore, we created a catalog summarizing some GRB information, collected either by running the pipeline or from the literature. The Perl script, the database, and the catalog are available for downloading and querying at the HEASARC Web site.

BURST ARCHIVE↗
NASA NTRS2016

The Kepler Science Data Processing Pipeline Source Code Road Map

We give an overview of the operational concepts and architecture of the Kepler Science Processing Pipeline. Designed, developed, operated, and maintained by the Kepler Science Operations Center (SOC) at NASA Ames Research Center, the Science Processing Pipeline is a central element of the Kepler Ground Data System. The SOC consists of an office at Ames Research Center, software development and operations departments, and a data center which hosts the computers required to perform data analysis. The SOC's charter is to analyze stellar photometric data from the Kepler spacecraft and report results to the Kepler Science Office for further analysis. We describe how this is accomplished via the Kepler Science Processing Pipeline, including, the software algorithms. We present the high-performance, parallel computing software modules of the pipeline that perform transit photometry, pixel-level calibration, systematic error correction, attitude determination, stellar target management, and instrument characterization.

Kepler pipeline software↗