SEARCH · Engineering Papers

Results for “PIPELINES”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

All sources NASA NTRS DOE OSTI

DOE OSTI2025

A machine-learning-driven data labeling pipeline for scientific analysis in MLExchange

This study introduces a novel labeling pipeline to accelerate the labeling process of scientific data sets by using artificial intelligence (AI)-guided tagging techniques. This pipeline includes a set of interconnected web-based graphical user interfaces (GUIs), where Data Clinic and MLCoach enable the preparation of machine learning (ML) models for data reduction and classification, respectively, while Label Maker is used for label assignment. Throughout this pipeline, data can be accessed through a direct connection to a file system or through Tiled for access through Hypertext Transfer Protocol (HTTP). Our experimental results present three use cases where this labeling pipeline has been instrumental for the study of large X-ray scattering data sets in the area of pattern recognition, the remote analysis of resonant soft X-ray scattering data and the fine-tuning process of foundation models. These use cases highlight the labeling capabilities of this pipeline, including the ability to label large data sets in a short period of time, to perform remote data analysis while minimizing data movement and to enhance the fine-tuning process of complex ML models with human involvement.

Chavez, Tanny (ORCID:0000000193172896)↗

DOE OSTI2024

Pipeline Hydrogen Decarbonization and Repurposing Analyzers (P-HyDRAs)

The Pipeline Hydrogen Decarbonization and Repurposing Analyzers (P-HyDRAs) are a set of prototype computational tools for simulating and optimizing midstream natural gas pipeline system operations subject to location and time-dependent hydrogen blending. The models can accurately resolve dynamic gas flows through large-scale pipeline networks using non-ideal gas equations of state. The codes can be used as decision support for planning and design decisions involving intra-day energy flow schedules as well as spatiotemporal economic values of natural gas, hydrogen, and net energy delivered to consumers while ensuring that pipeline hydraulic limitations, gas compressor station constraints, operational factors, and pre-existing shipping contracts are satisfied. The inputs to the codes are a model of the pipeline system as well as time-series data that specify boundary conditions on the network. For optimization, the code module requires price and quantity offers for natural gas and hydrogen and price and quantity bids for energy, which are used as time-dependent constraints in an optimal control problem. The outputs are time-series data that provide a predictive simulation of gas flows, mass fractions, and pressures, or with additional degrees of freedom give an approximately optimal solution for gas injections/withdrawals, compressor settings, and sensitivities to the objective function that provide locational values of energy.

Zlotnik, Anatoly↗

DOE OSTI2024

A Data Processing Pipeline To Extract A Knowledge Graph From Sec Documents For Socio-technical Analysis Of Critical Infrastructure Influence

Weaver, GabrielA.↗

DOE OSTI2025

A Data Processing Pipeline To Extract A Knowledge Graph From Heterogeneous Data For Socio-technical Analysis Of Critical Infrastructure Influence

The code is written in Python and consists of the following pipeline that is implemented in Apache Airflow. This pipeline intends to understand the companies that are directly or indirectly involved with a type of critical infrastructure system at some point in that system's lifecycle. The pipeline takes a configuration file that specifies a list of initial companies to consider, a geographic region of interest, and a set of SEC form types as well as other data sources (e.g. CrunchBase) from which to extract entities and relations. There are four main components to this pipeline as currently implemented: Entity Extraction, Network Construction, Analysis, and Visualization. First, Entity Extraction, is implemented as the `topear-extract_organizations` Apache Airflow workflow. Given an initial query that specifies a geographic region of interest and a time interval, the software will extract CI facilities of interest and organizations that have a direct influence relationship to those facilities (e.g. ownership). During the course of the LDRD, we focused on Electric Vehicle charging stations and this information is available via the Department of Energy (DOE) database on fueling stations maintained by NREL. Within the context of the DOE CESER project, we have focused on Battery Energy Storage Systems (BESS). Second, the Network Extraction component will iteratively construct a social network graph given the set of organizations and people extracted in the previous step. Organizations (and eventually People if desired) are then fed as a query to the `topgear-construct_social_network` Apache Airflow workflow which given a set of initial companies and data sets (e.g. SEC EDGAR form types, OpenCorporates, Crunchbase). This Airflow workflow will iteratively query such data sources to discover relationships with new organizations and people. For example, this module can iteratively query SEC EDGAR for metadata that documents the number of each type of form for the given set of companies and their location. This forms metadata represents a catalog of data sources from SEC EDGAR for the extracted social network knowledge graph. The pipeline then downloads these forms from the website and saves them in a build directory for further processing. These documents are then parsed for entities and relations. Again, we note that in additional to SEC data sources, this step can also pull in information on organizations via API services such as CrunchBase and OpenCorporates or bulk data sources. At the end of this step, the resultant social network, the Critical Infrastructure network, and the edges that encode relationships between organizations and CI facilities, form the Adversarial Socio-Technical Network (ASTN) that informs the analysis. Third, the Analysis component processes these generated ASTN. Previously, that has included the ability to compare prevalence of different vendors for a given infrastructure component type across different regions as well as identify common public and private investors across those vendors. This was demonstrated for EV Charging Stations across several different metropolitan areas within an IEEE PES GridEdge publication. More recently, we have looked at ways to identify infrastructure owners and operators of BESS with the most nameplate capacity across different states as well as other indictors of risk resulting from changes in ownership over time. Finally, the Visualization component consists of an HTML/CSS/JS framework by which users can interact geospatial, operational, and organizational relationships across a given portfolio of Critical Infrastructure facilities. The objective is to provide a library of UI/UX modules that can be repurposed for stakeholder-specific dashboards. All of the modules are related via a common event model that enables UI actions in one view to percolate across the other views.

Weaver, Gabriel [Idaho National Laboratory (INL), ↗

DOE OSTI2020

Automated pipeline framework for processing of large-scale building energy time series data

Commercial buildings account for one third of the total electricity consumption in the United States and a significant amount of this energy is wasted. Therefore, there is a need for “virtual” energy audits, to identify energy inefficiencies and their associated savings opportunities using methods that can be non-intrusive and automated for application to large populations of buildings. Here we demonstrate virtual energy audits applied to large populations of buildings’ time-series smart-meter data using a systematic approach and a fully automated Building Energy Analytics (BEA) Pipeline that unifies, cleans, stores and analyzes building energy datasets in a non-relational data warehouse for efficient insights and results. This BEA pipeline is based on a custom compute job scheduler for a high performance computing cluster to enable parallel processing of Slurm jobs. Within the analytics pipeline, we introduced a data qualification tool that enhances data quality by fixing common errors, while also detecting abnormalities in a building’s daily operation using hierarchical clustering. We analyze the HVAC scheduling of a population of 816 buildings, using this analytics pipeline, as part of a cross-sectional study. With our approach, this sample of 816 buildings is improved in data quality and is efficiently analyzed in 34 minutes, which is 85 times faster than the time taken by a sequential processing. The analytical results for the HVAC operational hours of these buildings show that among 10 building use types, food sales buildings with 17.75 hours of daily HVAC cooling operation are decent targets for HVAC savings. Overall, this analytics pipeline enables the identification of statistically significant results from population based studies of large numbers of building energy time-series datasets with robust results. These types of BEA studies can explore numerous factors impacting building energy efficiency and virtual building energy audits. This approach enables a new generation of data-driven buildings energy analysis at scale.

36 MATERIALS SCIENCE↗

DOE OSTI2011

Development of Protective Coatings for Co-Sequestration Processes and Pipelines

The program, entitled Development of Protective Coatings for Co-Sequestration Processes and Pipelines, examined the sensitivity of existing coating systems to supercritical carbon dioxide (SCCO2) exposure and developed new coating system to protect pipelines from their corrosion under SCCO2 exposure. A literature review was also conducted regarding pipeline corrosion sensors to monitor pipes used in handling co-sequestration fluids. Research was to ensure safety and reliability for a pipeline involving transport of SCCO2 from the power plant to the sequestration site to mitigate the greenhouse gas effect. Results showed that one commercial coating and one designed formulation can both be supplied as potential candidates for internal pipeline coating to transport SCCO2.

36 MATERIALS SCIENCE↗

DOE OSTI2014

Electricity Transmission, Pipelines, and National Trails: An Analysis of Current and Potential Intersections on Federal Lands in the Eastern United States, Alaska, and Hawaii

As has been noted in many reports and publications, acquiring new or expanded rights-of-way for transmission is a challenging process, because numerous land use and land ownership constraints must be overcome to develop pathways suitable for energy transmission infrastructure. In the eastern U.S., more than twenty federally protected national trails (some of which are thousands of miles long, and cross many states) pose a potential obstacle to the development of new or expanded electricity transmission capacity. However, the scope of this potential problem is not well-documented, and there is no baseline information available that could allow all stakeholders to study routing scenarios that could mitigate impacts on national trails. This report, Electricity Transmission, Pipelines, and National Trails: An Analysis of Current and Potential Intersections on Federal Lands in the Eastern United States, was prepared by the Environmental Science Division of Argonne National Laboratory (Argonne). Argonne was tasked by DOE to analyze the “footprint” of the current network of National Historic and Scenic Trails and the electricity transmission system in the 37 eastern contiguous states, Alaska, and Hawaii; assess the extent to which national trails are affected by electrical transmission; and investigate the extent to which national trails and other sensitive land use types may be affected in the near future by planned transmission lines. Pipelines are secondary to transmission lines for analysis, but are also within the analysis scope in connection with the overall directives of Section 368 of the Energy Policy Act of 2005, and because of the potential for electrical transmission lines being collocated with pipelines. Based on Platts electrical transmission line data, a total of 101 existing intersections with national trails on federal land were found, and 20 proposed intersections. Transmission lines and pipelines are proposed in Alaska; however there are no locations that intersect national trails. Source data did not indicate any planned transmission lines or pipelines in Hawaii. A map atlas provides more detailed mapping of the topics investigated in this study, and the accompanying GIS database provides the baseline information for further investigating locations of interest. In many cases the locations of proposed transmission lines are not accurately mapped (or a specific route may not yet be determined), and accordingly the specific crossing locations are speculative. However since both national trails and electrical transmission lines are long linear systems, the characteristics of the crossings reported in this study are expected to be similar to both observed characteristics of the existing infrastructure provided in this report, and of the new infrastructure if these proposed projects are built. More focused study of these siting challenges is expected to mitigate some of potential impacts by choosing routes that minimize or eliminate them. The current study primarily addresses a set of screening-level characterizations that provide insights into how the National Trail System may influence the siting of energy transport facilities in the states identified under Section 368(b) of the Energy Policy Act of 2005. As such, it initializes gathering and beginning analysis of the primary environmental and energy data, and maps the contextual relationships between an important national environmental asset and how this asset intersects with energy planning activities. Thus the current study sets the stage for more in-depth analyses and data development activities that begin to solve key transmission siting constraints. Our recommendations for future work incorporate two major areas: (1) database development and analytics and (2) modeling and scenario analysis for energy planning. These recommendations provide a path forward to address key issues originally developed under the Energy Policy Act of 2005 that are now being carried forward under the President’s Climate Action Plan.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

DOE OSTI2016

FutureGen 2.0 Pipeline and Regional Carbon Capture Storage Project - Final Report

The U.S. Department of Energy’s (DOE) FutureGen 2.0 Program involves two projects: (1) the Oxy-Combustion Power Plant Project and (2) the CO2 Pipeline and Storage Project. This Final Technical Report is focused on the CO2 Pipeline and Storage Project. The FutureGen 2.0 CO2 Pipeline and Storage Project evolved from an initial siting and project definition effort in Phase I, into the Phase II activity consisting permitting, design development, the acquisition of land rights, facility design, and licensing and regulatory approvals. Phase II also progressed into construction packaging, construction procurement, and targeted early preparatory activities in the field. The CO2 Pipeline and Storage Project accomplishments were significant, and in some cases unprecedented. The engineering, permitting, legal, stakeholder, and commercial learnings substantially advance the nation’s understanding of commercial-scale CO2 storage in deep saline aquifers. Voluminous and significant information was obtained from the drilling and the testing program of the subsurface, and sophisticated modeling was performed that held up to a wide range of scrutiny. All designs progressed to the point of securing construction contracts or comfort letters attesting to successful negotiation of all contract terms and willing execution at the appropriate time all major project elements – pipeline, surface facilities, and subsurface – as well as operations. While the physical installation of the planned facilities did not proceed in part due to insufficient time to complete the project prior to the expiration of federal funding, the project met significant objectives prior to DOE’s closeout decision. Had additional time been available, there were no known, insurmountable obstacles that would have precluded successful construction and operation of the project. Due to the suspension of the project, site restoration activities were developed and the work was accomplished. The site restoration efforts are also documented in this report. All permit applications had been submitted to all agencies for those permits or approvals required prior to the start of project construction. Most of the requisite permits were received during Phase II. This report includes information on each permitting effort. Successes and lessons learned are included in this report that will add value to the next generation of carbon storage efforts.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2018

Exploring Flexible Communications for Streamlining DNN Ensemble Training Pipelines

Parallel training of a Deep Neural Network (DNN) ensemble on a cluster of nodes is a common practice to train multiple models in order to construct a model with a higher prediction accuracy. Existing ensemble training pipelines can perform a great deal of redundant operations, resulting in unnecessary CPU usage, or even poor pipeline performance. In order to remove these redundancies, we need pipelines with more communication flexibility than existing DNN frameworks can provide. This project investigates a series of designs to improve pipeline flexibility and adaptivity, while also increasing performance. We implement our designs using Tensorflow with Horovod, and test it using several large DNNs in a large scale GPU cluster, Titan supercomputer at Oak Ridge National Lab. Our results show that the CPU time spent during training is reduced by 2-11X. Furthermore, our implementation can achieve up to 10X speedups when CPU core limits are imposed. Our best pipeline also reduces the average power draw of the ensemble training process by 5-16% when compared to the baseline.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2019

Estimating Effects of Various Factors on Corrosion of Pipeline Grade Carbon Steel Under Disbonded Coating (Status Update)

This report provides description of the activities, associated data and results for a work focused on understanding corrosion of pipeline grade carbon steel under coating disbondment. Specifically, the work reported here was for a pipeline that was installed in 1968 with the following characteristics. The line was coated with polyethylene tape coat (brand Polyken), and the line joints were electric-flash welded. The line joints are 34-inch outer diameter, and made of API 5L X52 grade carbon steel. The line was initially operated at 55 °F, but subsequently the line temperature increased to 105 °F due to higher fluid flow rate inside the pipe. Due to age and type of the pipeline coating, coating disbondment has occurred at several locations. The increase in operating temperature, coupled with age of the coating, is likely to have resulted in migration in corrosive species from the soil underneath the coating. As a result, the pipeline is experiencing enhanced corrosion. The objective of this study was to evaluate whether the change in operating temperature has affected the corrosion rate of the pipeline external surfaces under areas of disbonded coating.

36 MATERIALS SCIENCE↗

DOE OSTI2020

Multi-Functional Distributed Fiber Sensors for Pipeline Monitoring and Methane Detections. Final Report

As an abundant and cheap fossil energy source, natural gas has become a significant energy supply to support the United States’ economy. However, the large-scale extraction and utilization of natural gas also impose significant challenges on methane leakage. This problem is exacerbated by aging gas utility delivery systems, including interstate high-pressure pipelines, storage, and transmission facilities. This project aims to develop a cost-effective fiber optical sensing method that can perform multi-parameter real-time measurements of natural gas pipelines across long interrogation distances up to 100 km with 1-meter spatial resolution. This sensing tool can evaluate overall pipeline efficiency and reduce methane emissions for mid-stream methane infrastructures. To accomplish this objective, research and development efforts funded by this project have resulted in the following accomplishments: This project successfully has developed new functional sensory polymer materials using Metal-Organic Frameworks (MOFs) that can be coated on optical fiber through the reel-to-reel coating process. Functional polymer-coated optical fibers can perform sensitive methane detection through evanescence wave interaction and strain-based measurements to achieve 1% detection sensitivities. The new sensors fibers support both distributed measurements and multiplexed fiber sensors array for multi-point measurements. This project developed and optimized a new multi-core optical fiber that supports simultaneous and distributed measurements of strain and temperatures with 1-meter spatial resolutions across up to 100-km interrogation distance. This new fiber, combined with sensory polymercoated fiber, could perform both distributed temperature and methane detections. This project developed a new artificial intelligence big data algorithm approach that can effectively analyze high-resolution data harnessed by distributed fiber sensors to protect natural gas pipelines against external threats and detect internal defects induced by corrosion. Working with our industry partner, this project developed new optical fibers that support fiber sensor fabrications through polymer coating after the fibers are drawn. These new fibers eliminate the need for direct sensor fabrication when the fiber is fabricated on a fiber draw tower, which drastically expands fiber sensors' applicability. This research project has significantly advanced the distributed fiber sensing technology. It will dramatically increase the applicability and adaptability of distributed fiber sensors for a wide array of applications in energy, sustainability, and environmental science, including structural health monitoring of natural gas pipelines, oil infrastructures, hydrogen facilities, and environmental monitoring of carbon storage sites, water supply systems, and others.

03 NATURAL GAS↗

DOE OSTI2023

Abstract for CRADA between National Energy Technology Laboratory and Colonial Pipeline Company

The National Energy Technology Laboratory (NETL) and Colonial Pipeline Company (Participant) will collaborate in the field demonstration of optical fiber sensor systems developed at NETL on Participant’s fuel pipeline. The optical fiber sensor technologies are capable of distributed temperature and strain sensing, distributed acoustic sensing, and ultrasensitive acoustic sensing. Real-time monitoring of these parameters enables pipeline integrity monitoring, security monitoring, flow rate monitoring, etc. Successful demonstration on a real fuel pipeline at Participant’s facilities will validate the sensor technologies and installation methods at a real scale. This effort aligns with NETL’s mission in reliable and sustainable energy and reducing environmental effects due to pipeline failures.

42 ENGINEERING↗

DOE OSTI2023

Onshore U.S. Carbon Pipeline Deployment: Siting, Safety, and Regulation

Carbon capture, utilization, and storage technology has significant potential to reduce greenhouse gas emissions and mitigate the impact of climate change, particularly in hard-to-decarbonize industrial and commercial sectors. Unless utilization or storage occurs at the same location as carbon capture, which is rare, carbon must be transported from a point source to a utilization or storage site. Pipelines offer significant advantages for large-scale transportation of carbon dioxide over other methods. While studies show that reaching net-zero carbon emissions in the United States by 2050 will require between 29,000 and 66,000 miles of carbon pipelines, the U.S. had deployed fewer than 6,000 miles of carbon pipelines by 2022. Although closing this gap is important to achieving low-carbon goals, carbon pipelines operate in a complex and uncertain local, state and federal regulatory landscape and face public concerns about safety and siting. This report covers numerous regulatory issues surrounding carbon pipeline development, including the current narrow federal definition of carbon dioxide and the considerable variation in state and local governments’ laws and regulations. The report serves as a primer for regulators and stakeholders who seek to better understand the regulatory challenges and opportunities facing this critical infrastructure

01 COAL, LIGNITE, AND PEAT↗

DOE OSTI2022

Transport Affordable Clean Hydrogen Energy via Existing Pipeline Infrastructure (CRADA 573)

Our Nation’s vast network of oil pipelines span over 230,000 miles connecting remote energy producing regions to distant markets. The COVID-19 pandemic’s influence on oil prices intensified technical and economic vulnerabilities afflicting the Trans-Alaska Pipeline System (TAPS, operating below 25% capacity) that may soon resurface as energy demand shifts to low carbon sources. Through the opportunity afforded by the Arctic Advanced Manufacturing Program, I’ve toured the great State of Alaska learning about its culture, unique energy challenges, business ecosystem, research institutions, and regulatory agencies. With encouragement and support from dedicated Alaskans and my professional mentors, I’ve founded Mighty Pipeline for the purpose of developing and commercializing proprietary technology and hardware to convert oil pipelines into clean hydrogen energy transmission systems. Leveraging the advanced technical capabilities of Pacific Northwest National Laboratory, we’ve demonstrated proof-of-concept (TRL = 3) and are seeking pre-pilot development project opportunities to conduct hardware and system level tests. If we successfully achieve technical and regulatory milestones, Mighty Pipeline’s technology could be available to help facilitate bulk clean hydrogen energy export this decade.

08 HYDROGEN↗

DOE OSTI2026

Statistical and Machine Learning Approaches to Analyzing Pipeline Incidents in the United States (2010–2024)

This study applies machine learning methods to analyze natural gas pipeline incidents in the United States using the Pipeline and Hazardous Materials Safety Administration (PHMSA) Gas Distribution Incident Dataset (2010–2024). The dataset includes over 600 variables describing incident characteristics, infrastructure attributes, and contributing factors associated with unintentional gas releases. The objective is to assess whether these features can reliably predict the underlying cause of pipeline failures. Multinomial logistic regression and Random Forest models were developed to classify incident causes, including excavation damage, corrosion, equipment failure, and natural forces. Results show that excavation damage is both the most frequent and most predictable cause, with models achieving strong performance for this category. However, when excavation damage is excluded, model accuracy declines significantly, with some models performing near random levels. Across all approaches, severe class imbalance and limited variability in key predictors constrain predictive performance. Pipeline age and diameter emerge as the most influential variables, but they provide insufficient discriminatory power to distinguish among less frequent failure types. These findings indicate that non-excavation-related incidents are rare, heterogeneous, and weakly represented in the dataset, limiting the effectiveness of machine learning classification. Overall, this study highlights the structural limitations of the PHMSA dataset for predictive modeling and underscores the need for improved data balance and feature enrichment. The results reinforce excavation damage prevention as the most impactful strategy for reducing pipeline incidents.

03 NATURAL GAS↗

DOE OSTI2026

Adaptive Cybersecurity for Distributed Energy Resources (AdCyDER): Online Reinforcement Learning with Stackelberg-Optimized Defenses — Pipeline Architecture, Evaluation Methodology, and Findings from a Synthetic-Data Evaluation

This report documents the design and evaluation of an integrated online-learning pipeline developed within the AdCyDER project for Distributed Energy Resource (DER) cybersecurity. The pipeline couples a Reinforcement Learning (RL) attack classifier — which produces an attack-type probability distribution — with a Stackelberg game-theoretic (GT) defense selector that consumes those distributions alongside SME-encoded priors over (defense, attack) effectiveness pairings and perdefense costs to choose grid-health-preserving defenses. The objective is not attack classification per se but production of distributions that drive effective defense selection through the Stackelberg layer, learned from delayed grid-health feedback rather than labeled attack data. AdCyDER as a whole is broader than the work presented here; this report covers the specific RL/GT loop integration and its evaluation. We present the integrated pipeline (SCADA telemetry with Fronius inverter physics, Suricata IDS, time-windowed aggregation, per-facility LSTM classifier, Stackelberg optimizer, OpenC2 actuators), an experimental campaign of 28 eight-hour iterations across three baseline modes, and a pipeline-ordered diagnostic protocol. The protocol identifies two distinct failure modes within the loop: paired supervised ceilings on the same features establish that the deployed online RL classifier (macro F1 ≈ 0.07) sits at least 4.7× below a same-architecture supervised LSTM (≈ 0.34) and 10–11× below a linear feature-signal ceiling (≈ 0.70–0.79 depending on per-facility isolation), localizing the dominant failure to the training procedure; and the reward signal driving online updates carries weak directional coupling with classifier correctness in the methodology-expected direction (multi-lens convergent: top-decile P(true) records produce more frequent state changes and slightly larger improvements, top-vs-bot Cohen’s 𝑑 ≈ −0.19), but at effect magnitudes too small to drive gradient-based learning at the campaign sample size. The original learning hypothesis is not supported by the data. The primary contributions are the diagnostic methodology — proposed as a transferable falsification protocol for online RL/GT defense pipelines learning from delayed environmental reward — and the open, reproducible experimental infrastructure. We outline reward reformulation as the highest-priority aspirational next step given the underpowered-but-aligned Q6 reading, with hardware-in-the-loop evaluation as the broadest scope-expansion option.

Blakely, Benjamin [Argonne National Laboratory (AN↗

DOE OSTI2023

PhytoOracle: Scalable, modular phenomics data processing pipelines

As phenomics data volume and dimensionality increase due to advancements in sensor technology, there is an urgent need to develop and implement scalable data processing pipelines. Current phenomics data processing pipelines lack modularity, extensibility, and processing distribution across sensor modalities and phenotyping platforms. To address these challenges, we developed PhytoOracle (PO), a suite of modular, scalable pipelines for processing large volumes of field phenomics RGB, thermal, PSII chlorophyll fluorescence 2D images, and 3D point clouds. PhytoOracle aims to ( i ) improve data processing efficiency; ( ii ) provide an extensible, reproducible computing framework; and ( iii ) enable data fusion of multi-modal phenomics data. PhytoOracle integrates open-source distributed computing frameworks for parallel processing on high-performance computing, cloud, and local computing environments. Each pipeline component is available as a standalone container, providing transferability, extensibility, and reproducibility. The PO pipeline extracts and associates individual plant traits across sensor modalities and collection time points, representing a unique multi-system approach to addressing the genotype-phenotype gap. To date, PO supports lettuce and sorghum phenotypic trait extraction, with a goal of widening the range of supported species in the future. At the maximum number of cores tested in this study (1,024 cores), PO processing times were: 235 minutes for 9,270 RGB images (140.7 GB), 235 minutes for 9,270 thermal images (5.4 GB), and 13 minutes for 39,678 PSII images (86.2 GB). These processing times represent end-to-end processing, from raw data to fully processed numerical phenotypic trait data. Repeatability values of 0.39-0.95 (bounding area), 0.81-0.95 (axis-aligned bounding volume), 0.79-0.94 (oriented bounding volume), 0.83-0.95 (plant height), and 0.81-0.95 (number of points) were observed in Field Scanalyzer data. We also show the ability of PO to process drone data with a repeatability of 0.55-0.95 (bounding area).

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2021

DRAGraces: A Pipeline for the GRACES High-resolution Spectrograph at Gemini

This paper describes the software Data Reduction and Analysis for GRACES (DRAGraces), which is a pipeline reducing spectra from Gemini Remote Access to the CFHT ESPaDOnS Spectrograph (GRACES) at the Gemini North Telescope. The code is written in the IDL language. It is designed to find all the GRACES frames in a given directory, automatically determine the list of bias, flat, arc, and science frames, and perform the whole reduction and extraction within a few minutes. We compare the output from DRAGraces with that of the Open source Pipeline for ESPaDOnS Reduction and Analysis (OPERA), a pipeline developed at the Canada–France–Hawaii Telescope (CFHT) that also can extract GRACES spectra. Both pipelines were developed completely independently, yet they give very similar extracted spectra. They both have their advantages and disadvantages. For instance, DRAGraces is more straightforward and easy to use and is less likely to be derailed by a parameter that needs to be tweaked, while OPERA offers a more careful extraction that can be significantly superior when the highest resolution is required and when the signal-to-noise ratio is low. One should compare both before deciding which one to use for their science. Yet, both pipelines deliver a fairly comparable resolution power (R ~ 52.8k and 36.6k for DRAGraces and R ~ 58k and 40k for OPERA in high- and low-resolution spectral modes, respectively), wavelength solution, and signal-to-noise ratio per resolution element.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗