Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “PIPELINES”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 523 records · Page 29

MetFish: a Metabolomics Pipeline for Studying Microbial Communities in Chemically Extreme Environments

Metabolites have essential roles in microbial communities, including as mediators of nutrient and energy exchange, cell-to-cell communication, and antibiosis. However, detecting and quantifying metabolites and other chemicals in samples having extremes in salt or mineral content using liquid chromatography-mass spectrometry (LC-MS)-based methods remains a significant challenge. Here, we report a facile method based on in situ chemical derivatization followed by extraction for analysis of metabolites and other chemicals in hypersaline samples, enabling for the first time direct LC-MS-based exometabolomics analysis in sample matrices containing up to 2 M total dissolved salts. The method, MetFish, is applicable to molecules containing amine, carboxylic acid, carbonyl, or hydroxyl functional groups, and it can be integrated into either targeted or untargeted analysis pipelines. In targeted analyses, MetFish provided limits of quantification as low as 1 nM, broad linear dynamic ranges (up to 5 to 6 orders of magnitude) with excellent linearity, and low median interday reproducibility (e.g., 2.6%). MetFish was successfully applied in targeted and untargeted exometabolomics analyses of microbial consortia, quantifying amino acid dynamics in the exometabolome during community succession; in situ in a native prairie soil, whose exometabolome was isolated using a hypersaline extraction; and in input and produced fluids from a hydraulically fractured well, identifying dramatic changes in the exometabolome over time in the well.

59 BASIC BIOLOGICAL SCIENCES↗

A Length Adaptive Algorithm-Hardware Co-design of Transformer on FPGA Through Sparse Attention and Dynamic Pipelining

Transformers are considered one of the most important deep learning models since 2018, in part because it establishes state-of-the-art (SOTA) records and could potentially replace existing Deep Neural Networks (DNNs). Despite the remarkable triumphs, the prolonged turnaround time of Transformer models is a widely recognized roadblock. The variety of sequence lengths imposes additional computing overhead where inputs need to be zero-padded to the maximum sentence length in the batch to accommodate the parallel computing platforms. This paper targets the field-programmable gate array (FPGA) and proposes a coherent sequence length adaptive algorithm–hardware co-design for Transformer acceleration. Particularly, we develop a hardware-friendly sparse attention operator and a length-aware hardware resource scheduling algorithm. The proposed sparse attention operator brings the complexity of attention-based models down to linear complexity and alleviates the off-chip memory traffic. The proposed length-aware resource hardware scheduling algorithm dynamically allocates the hardware resources to fill up the pipeline slots and eliminates bubbles for NLP tasks. Experiments show that our design has very small accuracy loss and has 80.2 × and 2.6 × speedup compared to CPU and GPU implementation, and 4 × higher energy efficiency than state-of-the-art GPU accelerator optimized via CUBLAS GEMM.

Peng, Hongwu↗

Integrase-On-Demand-Pipeline Data Set

Files needed to run the Integrase-On-Demand-Pipeline, a program designed to provide users with a list of putative attachment site and integrase pairs for a prokaryotic genome of interest. isles.pkl: Serialized python-object file, containing a dictionary of attachment site sequences and reference genomic island information extracted from the Genomic island database ints.gff: Gene format file containing annotations for all integrases referenced in isles.pkl. The source genome, gene coordinates, integrase name, protein IDs and amino acid sequence included. reps.msh: Binary file containing 1000 128-bit MurmurHash3 hashes for >80,000 genomes

McClain, Hannah Marie [Sandia National Laboratorie↗

Rapid Threat Organism Recognition Pipeline

The RAPTOR computational pipeline identifies microbial nucleic acid sequences present in sequence data from clinical samples. It takes as input raw short-read genomic sequence data (in particular, the type generated by the Illumina sequencing platforms) and outputs taxonomic evaluation of detected microbes in various human-readable formats. This software was designed to assist in the diagnosis or characterization of infectious disease, by detecting pathogen sequences in nucleic acid sequence data from clinical samples. It has also been applied in the detection of algal pathogens, when algal biofuel ponds became unproductive. RAPTOR first trims and filters genomic sequence reads based on quality and related considerations, then performs a quick alignment to the human (or other host) genome to filter out host sequences, then performs a deeper search against microbial genomes. Alignment to a protein sequence database is optional. Alignment results are summarized and placed in a taxonomic framework using the Lowest Common Ancestor algorithm.

Solberg, OwenD.↗

Drug Target and Chemotype Identification Pipeline

The TargetID pipeline enables rapid identification and characterization of binding sites in SARS-CoV-2 proteins as well as the core chemical components with which these sites interact.

Sandholtz, SarahH↗

GAPs (Geospatial Analysis Pipelines)

Geospatial Analysis Pipelines (GAPs) is a framework designed to scale single-location geospatial models to a High-Performance Computing (HPC) environment.

Pinchuk, Pavlo (Paul)↗

Machine Learning Automation Pipeline

Machine Learning Automation Pipeline (MLAP) is a package to perform machine learning (ML) analysis in a step by step manner, starting with data extraction until analysis and prediction. The scripts provide the users option to chose an action such as "Extract", "Prep", and "Train" and numerous cases can be launched with just a single command. The inputs for each case are provided using a JSON file. The simulation results of several cases can be assessed using an automated process and analyzed for various metrics pertinent to ML analysis.

Jha, Pankaj↗

Disclosure of the XRD pipeline software

The XRD pipeline software is a program to reduce 2D X-ray diffraction data from area detectors with advanced algorithms on automatic masking and image segmentation, which help characterization of multiple phases recorded in the data and facilitate data analysis.SF-25-114

XU, WENQIAN [Argonne National Laboratory (ANL), Ar↗

PIPES (Pipeline for Integrated Projects in Energy Systems) [SWR-24-89]

The Pipeline for Integrated Projects in Energy Systems (PIPES) is a comprehensive project, data, and workflow management tool designed for integrated modeling teams. PIPES facilitates the management of data requirements, tasks, and progress tracking, serving as a higher-level integration layer that works across various data and modeling software. This tool integrates models, data, and tools to perform large-scale, integrated analysis work at scale. PIPES is designed to streamline integrated modeling projects, enhance collaboration, and ensure the quality and efficiency of data management and workflow processes. https://github.com/nrel-pipes/pipes-api https://github.com/nrel-pipes/pipes-web https://github.com/nrel-pipes/nrel-pipes

Gu, Jianli↗

POWER DATA PIPELINE

SF-25-081 Utility software for creating high-performance data pipelines to extract, load, and transform raw electric power systems measurements. For use with anomaly detection models training workflows. The software supports the project: Adaptive Cybersecurity for DER: A Game-Theoretic and Machine Learning approach for Real-Time Threat Detection and Mitigation

Plathottam, Silby Jose [Argonne National Laborator↗

NLR Data Processing Pipeline for MADIS [SWR-26-050]

The NLR Data Processing Pipeline for MADIS software package is for downloading, processing, and performing QA/QC on MADIS data. Designed to handle the following steps: 1) Download all MADIS data as compressed netcdf files for a given time period. 2) Unpack netcdf files into timeseries csvs for each coordinate within the given bounding box. 3) Process the csvs to filter according to quality control checks and convert variables to correct units. 4) Write processed csvs to a single nc file.

Benton, Brandon [National Laboratory of the Rockie↗

HiFiAdapterFilt, a memory efficient read processing pipeline, prevents occurrence of adapter sequence in PacBio HiFi reads and their negative impacts on genome assembly

Abstract Background Pacific Biosciences HiFi read technology is currently the industry standard for high accuracy long-read sequencing that has been widely adopted by large sequencing and assembly initiatives for generation of de novo assemblies in non-model organisms. Though adapter contamination filtering is routine in traditional short-read analysis pipelines, it has not been widely adopted for HiFi workflows. Results Analysis of 55 publicly available HiFi datasets revealed that a read-sanitation step to remove sequence artifacts derived from PacBio library preparation from read pools is necessary as adapter sequences can be erroneously integrated into assemblies. Conclusions Here we describe the nature of adapter contaminated reads, their consequences in assembly, and present HiFiAdapterFilt, a simple and memory efficient solution for removing adapter contaminated reads prior to assembly.

59 BASIC BIOLOGICAL SCIENCES↗

COMPILE: a GWAS computational pipeline for gene discovery in complex genomes

Abstract Background Genome-Wide Association Studies (GWAS) are used to identify genes and alleles that contribute to quantitative traits in large and genetically diverse populations. However, traits with complex genetic architectures create an enormous computational load for discovery of candidate genes with acceptable statistical certainty. We developed a streamlined computational pipeline for GWAS (COMPILE) to accelerate identification and annotation of candidate maize genes associated with a quantitative trait, and then matches maize genes to their closest rice and Arabidopsis homologs by sequence similarity. Results COMPILE executed GWAS using a Mixed Linear Model that incorporated, without compression, recent advancements in population structure control, then linked significant Quantitative Trait Loci (QTL) to candidate genes and RNA regulatory elements contained in any genome. COMPILE was validated using published data to identify QTL associated with the traits of α-tocopherol biosynthesis and flowering time, and identified published candidate genes as well as additional genes and non-coding RNAs. We then applied COMPILE to 274 genotypes of the maize Goodman Association Panel to identify candidate loci contributing to resistance of maize stems to penetration by larvae of the European Corn Borer ( Ostrinia nubilalis ). Candidate genes included those that encode a gene of unknown function, WRKY and MYB-like transcriptional factors, receptor-kinase signaling, riboflavin synthesis, nucleotide-sugar interconversion, and prolyl hydroxylation. Expression of the gene of unknown function has been associated with pathogen stress in maize and in rice homologs closest in sequence identity. Conclusions The relative speed of data analysis using COMPILE allowed comparison of population size and compression. Limitations in population size and diversity are major constraints for a trait and are not overcome by increasing marker density. COMPILE is customizable and is readily adaptable for application to species with robust genomic and proteome databases.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

A high solids field-to-fuel research pipeline to identify interactions between feedstocks and biofuel production

Abstract Background Environmental factors, such as weather extremes, have the potential to cause adverse effects on plant biomass quality and quantity. Beyond adversely affecting feedstock yield and composition, which have been extensively studied, environmental factors can have detrimental effects on saccharification and fermentation processes in biofuel production. Only a few studies have evaluated the effect of these factors on biomass deconstruction into biofuel and resulting fuel yields. This field-to-fuel evaluation of various feedstocks requires rigorous coordination of pretreatment, enzymatic hydrolysis, and fermentation experiments. A large number of biomass samples, often in limited quantity, are needed to thoroughly understand the effect of environmental conditions on biofuel production. This requires greater processing and analytical throughput of industrially relevant, high solids loading hydrolysates for fermentation, and led to the need for a laboratory-scale high solids experimentation platform. Results A field-to-fuel platform was developed to provide sufficient volumes of high solids loading enzymatic hydrolysate for fermentation. AFEX pretreatment was conducted in custom pretreatment reactors, followed by high solids enzymatic hydrolysis. To accommodate enzymatic hydrolysis of multiple samples, roller bottles were used to overcome the bottlenecks of mixing and reduced sugar yields at high solids loading, while allowing greater sample throughput than possible in bioreactors. The roller bottle method provided 42–47% greater liquefaction compared to the batch shake flask method for the same solids loading. In fermentation experiments, hydrolysates from roller bottles were fermented more rapidly, with greater xylose consumption, but lower final ethanol yields and CO 2 production than hydrolysates generated with shake flasks. The entire platform was tested and was able to replicate patterns of fermentation inhibition previously observed for experiments conducted in larger-scale reactors and bioreactors, showing divergent fermentation patterns for drought and normal year switchgrass hydrolysates. Conclusion A pipeline of small-scale AFEX pretreatment and roller bottle enzymatic hydrolysis was able to provide adequate quantities of hydrolysate for respirometer fermentation experiments and was able to overcome hydrolysis bottlenecks at high solids loading by obtaining greater liquefaction compared to batch shake flask hydrolysis. Thus, the roller bottle method can be effectively utilized to compare divergent feedstocks and diverse process conditions.

09 BIOMASS FUELS↗

MetaPop: a pipeline for macro- and microdiversity analyses and visualization of microbial and viral metagenome-derived populations

Abstract Background Microbes and their viruses are hidden engines driving Earth’s ecosystems from the oceans and soils to humans and bioreactors. Though gene marker approaches can now be complemented by genome-resolved studies of inter-(macrodiversity) and intra-(microdiversity) population variation, analytical tools to do so remain scattered or under-developed. Results Here, we introduce MetaPop, an open-source bioinformatic pipeline that provides a single interface to analyze and visualize microbial and viral community metagenomes at both the macro - and microdiversity levels. Macrodiversity estimates include population abundances and α- and β-diversity. Microdiversity calculations include identification of single nucleotide polymorphisms, novel codon-constrained linkage of SNPs, nucleotide diversity ( π and θ ), and selective pressures (pN/pS and Tajima’s D ) within and fixation indices ( F ST ) between populations. MetaPop will also identify genes with distinct codon usage. Following rigorous validation, we applied MetaPop to the gut viromes of autistic children that underwent fecal microbiota transfers and their neurotypical peers. The macrodiversity results confirmed our prior findings for viral populations (microbial shotgun metagenomes were not available) that diversity did not significantly differ between autistic and neurotypical children. However, by also quantifying microdiversity, MetaPop revealed lower average viral nucleotide diversity ( π ) in autistic children. Analysis of the percentage of genomes detected under positive selection was also lower among autistic children, suggesting that higher viral π in neurotypical children may be beneficial because it allows populations to better “bet hedge” in changing environments. Further, comparisons of microdiversity pre- and post-FMT in autistic children revealed that the delivery FMT method (oral versus rectal) may influence viral activity and engraftment of microdiverse viral populations, with children who received their FMT rectally having higher microdiversity post-FMT. Overall, these results show that analyses at the macro level alone can miss important biological differences. Conclusions These findings suggest that standardized population and genetic variation analyses will be invaluable for maximizing biological inference, and MetaPop provides a convenient tool package to explore the dual impact of macro - and microdiversity across microbial communities.

59 BASIC BIOLOGICAL SCIENCES↗

The standard operating procedure of the DOE-JGI Microbial Genome Annotation Pipeline (MGAP v.4)

Abstract The DOE-JGI Microbial Genome Annotation Pipeline performs structural and functional annotation of microbial genomes that are further included into the Integrated Microbial Genome comparative analysis system. MGAP is applied to assembled nucleotide sequence datasets that are provided via the IMG submission site. Dataset submission for annotation first requires project and associated metadata description in GOLD. The MGAP sequence data processing consists of feature prediction including identification of protein-coding genes, non-coding RNAs and regulatory RNA features, as well as CRISPR elements. Structural annotation is followed by assignment of protein product names and functions.

59 BASIC BIOLOGICAL SCIENCES↗

From Escherichia coli mutant 13 C labeling data to a core kinetic model: A kinetic model parameterization pipeline

Kinetic models of metabolic networks offer the promise of quantitative phenotype prediction. The mechanistic characterization of enzyme catalyzed reactions allows for tracing the effect of perturbations in metabolite concentrations and reaction fluxes in response to genetic and environmental perturbation that are beyond the scope of stoichiometric models. In this study, we develop a two-step computational pipeline for the rapid parameterization of kinetic models of metabolic networks using a curated metabolic model and available 13 C-labeling distributions under multiple genetic and environmental perturbations. The first step involves the elucidation of all intracellular fluxes in a core model of E. coli containing 74 reactions and 61 metabolites using 13 C-Metabolic Flux Analysis ( 13 C-MFA). Here, fluxes corresponding to the mid-exponential growth phase are elucidated for seven single gene deletion mutants from upper glycolysis, pentose phosphate pathway and the Entner-Doudoroff pathway. The computed flux ranges are then used to parameterize the same (i.e., k-ecoli74) core kinetic model for E. coli with 55 substrate-level regulations using the newly developed K-FIT parameterization algorithm. The K-FIT algorithm employs a combination of equation decomposition and iterative solution techniques to evaluate steady-state fluxes in response to genetic perturbations. k-ecoli74 predicted 86% of flux values for strains used during fitting within a single standard deviation of 13 C-MFA estimated values. By performing both tasks using the same network, errors associated with lack of congruity between the two networks are avoided, allowing for seamless integration of data with model building. Product yield predictions and comparison with previously developed kinetic models indicate shifts in flux ranges and the presence or absence of mutant strains delivering flux towards pathways of interest from training data significantly impact predictive capabilities. Using this workflow, the impact of completeness of fluxomic datasets and the importance of specific genetic perturbations on uncertainties in kinetic parameter estimation are evaluated.

59 BASIC BIOLOGICAL SCIENCES↗