Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “pipeline”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 811 records · Page 45

Combined tracer analysis for DESI 2024 BAO

This paper demonstrates how the Dark Energy Spectroscopic Instrument (DESI) Data Release 1 (DR1) and future baryon acoustic oscillations (BAO) analyses can optimally combine overlapping tracers (galaxies of distinct types) in the same redshift range. We make a unified catalog of Luminous Red Galaxies (LRGs) and Emission Line Galaxies (ELGs) in the redshift range 0.8 < z < 1.1 and investigate the impact on the BAO constraints. DESI DR1 contains ∼ 30% of the final DESI LRG sample and less than 25% of the final ELG sample, and the combination of LRGs and ELGs increases the number density and reduces the shot noise. We developed a pipeline to merge the overlapping tracers using galaxy bias as an approximately optimal weight and tested the pipeline on a suite of Abacus simulations, calibrated on the final version of the DESI Early Data Release. When applying our pipeline to the DESI DR1 catalog, we find an improvement in the BAO constraints of 11% for α iso and ∼ 7.0% for α AP consistent with our findings in mock catalogs. Our analysis was integrated into the DESI DR1 BAO analysis to produce theLRG+ELG result in the 0.8 < z < 1.1 redshift bin, which provided the most precise BAO measurement from DESI DR1 with a 0.86% constraint on the BAO distance scale and a 9.1σ detection of the isotropic BAO feature.

baryon acoustic oscillations↗

Galaxy clustering in harmonic space from the dark energy survey year 1 data: compatibility with real-space results

ABSTRACT We perform an analysis in harmonic space of the Dark Energy Survey Year 1 Data (DES-Y1) galaxy clustering photometric data using products obtained for the real-space analysis. We test our pipeline with a suite of lognormal simulations, which are used to validate scale cuts in harmonic space as well as to provide a covariance matrix that takes into account the DES-Y1 mask. We then apply this pipeline to DES-Y1 data taking into account survey property maps derived for the real-space analysis. We compare with real-space DES-Y1 results obtained from a similar pipeline. We show that the harmonic space analysis we develop yields results that are compatible with the real-space analysis for the bias parameters. This verification paves the way to performing a harmonic space analysis for the upcoming DES-Y3 data.

79 ASTRONOMY AND ASTROPHYSICS↗

Anomaly detection in the Zwicky Transient Facility DR3

We present results from applying the SNAD anomaly detection pipeline to the third public data release of the Zwicky Transient Facility (ZTF DR3). The pipeline is composed of three stages: feature extraction, search of outliers with machine learning algorithms, and anomaly identification with followup by human experts. Our analysis concentrates in three ZTF fields, comprising more than 2.25 million objects. A set of four automatic learning algorithms was used to identify 277 outliers, which were subsequently scrutinized by an expert. From these, 188 (68 per cent) were found to be bogus light curves – including effects from the image subtraction pipeline as well as overlapping between a star and a known asteroid, 66 (24 per cent) were previously reported sources whereas 23 (8 per cent) correspond to non-catalogued objects, with the two latter cases of potential scientific interest (e.g. one spectroscopically confirmed RS Canum Venaticorum star, four supernovae candidates, one red dwarf flare). Moreover, using results from the expert analysis, we were able to identify a simple bi-dimensional relation that can be used to aid filtering potentially bogus light curves in future studies. We provide a complete list of objects with potential scientific application so they can be further scrutinised by the community. These results confirm the importance of combining automatic machine learning algorithms with domain knowledge in the construction of recommendation systems for astronomy. Our code is publicly available.

79 ASTRONOMY AND ASTROPHYSICS↗

On the inconsistency of [C/Fe] abundances and the fractions of carbon-enhanced metal-poor stars among various stellar surveys

Carbon-enhanced metal-poor (CEMP) stars are a unique resource for Galactic archaeology because they probe the properties of the First Stars, early chemical evolution, and binary interactions at very low metallicity. Comparing the fractions and properties of CEMP stars in different Galactic environments can provide us with unique insights into the formation and evolution of the Milky Way halo and its building blocks. In this work, we investigate whether directly comparing fractions of CEMP stars from different literature samples of very metal-poor [Fe/H] < -2.0 stars is valid. We compiled published CEMP fractions and samples of Galactic halo stars from the past 25 years, and find that they are not all consistent with each other. Focusing on giant stars, we find significant differences between various surveys when comparing their trends of [Fe/H] versus [C/Fe] and their distributions of CEMP stars. To test the role of the analysis pipelines for low-resolution spectroscopic samples, we re-analysed giant stars from various surveys with the sspp and ferre pipelines. We found systematic differences in [C/Fe] of ~0.1-0.4 dex, partly independent of degeneracies with the stellar atmospheric parameters. These systematics are likely due to the different pipeline approaches, different assumptions in the employed synthetic grids, and/or the comparison of different evolutionary phases. We conclude that current biases in (the analysis of) very metal-poor samples limit the conclusions one can draw from comparing different surveys. We provide some recommendations and suggestions that will hopefully aid the community to unlock the full potential of CEMP stars for Galactic archaeology.

79 ASTRONOMY AND ASTROPHYSICS↗

STRIDES: automated uniform models for 30 quadruply imaged quasars

ABSTRACT Gravitational time delays provide a powerful one-step measurement of H0, independent of all other probes. One key ingredient in time-delay cosmography are high-accuracy lens models. Those are currently expensive to obtain, both, in terms of computing and investigator time (105–106 CPU hours and ∼0.5–1 yr, respectively). Major improvements in modelling speed are therefore necessary to exploit the large number of lenses that are forecast to be discovered over the current decade. In order to bypass this roadblock, we develop an automated modelling pipeline and apply it to a sample of 31 lens systems, observed by the Hubble Space Telescope in multiple bands. Our automated pipeline can derive models for 30/31 lenses with few hours of human time and <100 CPU hours of computing time for a typical system. For each lens, we provide measurements of key parameters and predictions of magnification as well as time delays for the multiple images. We characterize the cosmography-readiness of our models using the stability of differences in the Fermat potential (proportional to time delay) with respect to modelling choices. We find that for 10/30 lenses, our models are cosmography or nearly cosmography grade (<3 per cent and 3–5 per cent variations). For 6/30 lenses, the models are close to cosmography grade (5–10 per cent). These results utilize informative priors and will need to be confirmed by further analysis. However, they are also likely to improve by extending the pipeline modelling sequence and options. In conclusion, we show that uniform cosmography grade modelling of large strong lens samples is within reach.

79 ASTRONOMY AND ASTROPHYSICS↗

CasCollect: targeted assembly of CRISPR-associated operons from high-throughput sequencing data

Abstract CRISPR arrays and CRISPR-associated (Cas) proteins comprise a widespread adaptive immune system in bacteria and archaea. These systems function as a defense against exogenous parasitic mobile genetic elements that include bacteriophages, plasmids and foreign nucleic acids. With the continuous spread of antibiotic resistance, knowledge of pathogen susceptibility to bacteriophage therapy is becoming more critical. Additionally, gene-editing applications would benefit from the discovery of new cas genes with favorable properties. While next-generation sequencing has produced staggering quantities of data, transitioning from raw sequencing reads to the identification of CRISPR/Cas systems has remained challenging. This is especially true for metagenomic data, which has the highest potential for identifying novel cas genes. We report a comprehensive computational pipeline, CasCollect, for the targeted assembly and annotation of cas genes and CRISPR arrays—even isolated arrays—from raw sequencing reads. Benchmarking our targeted assembly pipeline demonstrates significantly improved timing by almost two orders of magnitude compared with conventional assembly and annotation, while retaining the ability to detect CRISPR arrays and cas genes. CasCollect is a highly versatile pipeline and can be used for targeted assembly of any specialty gene set, reconfigurable for user provided Hidden Markov Models and/or reference nucleotide sequences.

Podlevsky, Joshua D.↗

Benefits and Limits of Phasing Alleles for Network Inference of Allopolyploid Complexes

Abstract Accurately reconstructing the reticulate histories of polyploids remains a central challenge for understanding plant evolution. Although phylogenetic networks can provide insights into relationships among polyploid lineages, inferring networks may be hindered by the complexities of homology determination in polyploid taxa. We use simulations to show that phasing alleles from allopolyploid individuals can improve phylogenetic network inference under the multispecies coalescent by obtaining the true network with fewer loci compared with haplotype consensus sequences or sequences with heterozygous bases represented as ambiguity codes. Phased allelic data can also improve divergence time estimates for networks, which is helpful for evaluating allopolyploid speciation hypotheses and proposing mechanisms of speciation. To achieve these outcomes in empirical data, we present a novel pipeline that leverages a recently developed phasing algorithm to reliably phase alleles from polyploids. This pipeline is especially appropriate for target enrichment data, where the depth of coverage is typically high enough to phase entire loci. We provide an empirical example in the North American Dryopteris fern complex that demonstrates insights from phased data as well as the challenges of network inference. We establish that our pipeline (PATÉ: Phased Alleles from Target Enrichment data) is capable of recovering a high proportion of phased loci from both diploids and polyploids. These data may improve network estimates compared with using haplotype consensus assemblies by accurately inferring the direction of gene flow, but statistical nonidentifiability of phylogenetic networks poses a barrier to inferring the evolutionary history of reticulate complexes.

Evolutionary Biology↗

The Simons Observatory: Combining cross-spectral foreground cleaning with multitracer B -mode delensing for improved constraints on inflation

The Simons Observatory (SO), due to start full science operations in early 2025, aims to set tight constraints on inflationary physics by inferring the tensor-to-scalar ratio r from measurements of cosmic microwave background (CMB) polarization B-modes. Its nominal design including three small-aperture telescopes (SATs) targets a precision σ⁡(r = 0) ≤ 0.003 without delensing. Achieving this goal and further reducing uncertainties requires a thorough understanding and mitigation of other large-scale B-mode sources such as Galactic foregrounds and weak gravitational lensing. We present an analysis pipeline aiming to estimate r by including delensing within a cross-spectral likelihood, and demonstrate it for the first time on SO-like simulations accounting for various levels of foreground complexity, inhomogeneous noise and partial sky coverage. As introduced in an earlier SO delensing paper, lensing B-modes are synthesized using internal CMB lensing reconstructions as well as Planck-like cosmic infrared background maps and LSST-like galaxy density maps. We then extend SO’s power-spectrum-based foreground-cleaning algorithm to include all auto- and cross-spectra between the lensing template and the SAT B-modes in the likelihood function. This allows us to constrain r and the parameters of our foreground model simultaneously. Within this framework, we demonstrate the equivalence of map-based and cross-spectral delensing and use it to motivate an optimized pixel-weighting scheme for power spectrum estimation. We start by validating our pipeline in the simplistic case of uniform foreground spectral energy distributions. In the absence of primordial B-modes, we find that the 1⁢σ statistical uncertainty on r, σ⁡(r), decreases by 37% as a result of delensing. Tensor modes at the level of r = 0.01 are successfully detected by our pipeline. Even when using more realistic foreground models including spatial variations in the dust and synchrotron spectral properties, we obtain unbiased estimates of r both with and without delensing by employing the moment-expansion method. In this case, uncertainties are increased due to the higher number of model parameters, and delensing-related improvements range between 27% and 31%. These results constitute the first realistic assessment of the delensing performance at SO’s nominal sensitivity level.

79 ASTRONOMY AND ASTROPHYSICS↗

Characterizing Sub-Cohorts via Data Normalization and Representation Learning

The process of identifying a cohort of interest is a very challenging task. It requires manually inspecting many patient records of complex structure that might include medical coding errors and missing data. This paper presents a computational pipeline for refining the process of cohort selection based on medical concepts recorded in the electronic health records (EHRs). The pipeline extracts EHR data for a given cohort and normalizes this data using standard vocabularies. Then a stacked denoising autoencoder is used to embed the normalized patient vectors in a low dimensional space, where the patients are subsequently clustered into sub-cohorts. The goal is to represent the cohort in a standard format and abstract variants of sub-populations. As a use-case, we applied the pipeline to 1.8 million Veterans diagnosed with major depressive disorder (MDD), and identified four meaningful sub-cohorts using the features learned by the autoencoder. Then, each sub-cohort was explored using a set of keywords for interpretation.

Rush III, Everett↗

Power Profile Monitoring and Tracking Evolution of System-Wide HPC Workloads

The power & energy demands of HPC machines have grown significantly. Modern exascale HPC systems require tens of megawatts of combined power for computing resources and cooling facilities at full capacity. The current energy trend is not sustainable for future HPC systems, and there is a need to work toward the energy efficiency aspect of HPC performance. Energy awareness of the HPC applications at the job level is essential for running an efficient HPC system. This work aims to develop a pipeline to provide a production-level system-wide overview of the HPC workloads' power profile while handling evolving workloads exhibiting new power trends. We developed an open-set classification model for HPC jobs based on the properties of power profiles to continuously provide a system-wide holistic view of recently completed jobs. The pipeline helps continuously monitor the job-level power usage pattern of HPC and enables us to capture the new trends in applications' power behavior. We employed a comprehensive set of techniques to generate job-level data, custom-designed feature extraction methods to extract critical features from jobs' power profiles, clustering techniques powered by generative modeling, and open-set classification for identifying job profiles into known classes or an unknown set. With extensive evaluations, we demonstrate the effectiveness of each component in our pipeline. We provide an analysis of the resulting clusters that characterize the power profile landscape of the Summit supercomputer from more than 60K jobs executed in a year. The open-set classification classifies the known data sets into known classes with high accuracy and identifies unknown data noints with over 85% accuracy.

Karimi, Ahmad Maroof↗

A Science Gateway for the Repeatable Analysis of Machine Learning Predicted Gravity Anomalies

In recent years, deep learning has become an increasingly popular alternative for modeling in geoscience applications due to its scalability and efficiency. However, the interpretability, compute, data volume, and hyperparameter tuning requirements of deep learning models make development and monitoring difficult. Furthermore, model explainability and communicating results obtained by these models to users or domain experts is a challenge, as domain experts in geoscience also need to have a deep understanding of how those models function in order to support their scientific works. Here, we describe a science gateway and machine learning pipeline for predicting gravity anomalies from geophysical data. The gateway, built on open-source technologies, provides a holistic view of the pipeline through interactive visualizations aimed at enabling efficient exploratory data analysis. The repeatability, reproducibility, and monitoring capabilities of this overall system allow us to iterate and analyze at scale. Using this pipeline and gateway, we can repeatedly produce accurate high-resolution gravity anomaly datasets. By describing the underlying technologies, implementation, and results, here we provide a foundation for the broader adoption of science gateways into cross-cutting geoscience and machine learning research projects as a means to improve the scientific discovery and collaboration in the geophysics and computational sciences community.

58 GEOSCIENCES↗

Eukaryotic genomes from a global metagenomic data set illuminate trophic modes and biogeography of ocean plankton

ABSTRACT Metagenomics is a powerful method for interpreting the ecological roles and physiological capabilities of mixed microbial communities. Yet, many tools for processing metagenomic data are neither designed to consider eukaryotes nor are they built for an increasing amount of sequence data. EukHeist is an automated pipeline to retrieve eukaryotic and prokaryotic metagenome-assembled genomes (MAGs) from large-scale metagenomic sequence data sets. We developed the EukHeist workflow to specifically process large amounts of both metagenomic and/or metatranscriptomic sequence data in an automated and reproducible fashion. Here, we applied EukHeist to the large-size fraction data (0.8–2,000 µm) from Tara Oceans to recover both eukaryotic and prokaryotic MAGs, which we refer to as TOPAZ (Tara Oceans Particle-Associated MAGs). The TOPAZ MAGs consisted of >900 environmentally relevant eukaryotic MAGs and >4,000 bacterial and archaeal MAGs. The bacterial and archaeal TOPAZ MAGs expand upon the phylogenetic diversity of likely particle- and host-associated taxa. We use these MAGs to demonstrate an approach to infer the putative trophic mode of the recovered eukaryotic MAGs. We also identify ecological cohorts of co-occurring MAGs, which are driven by specific environmental factors and putative host-microbe associations. These data together add to a number of growing resources of environmentally relevant eukaryotic genomic information. Complementary and expanded databases of MAGs, such as those provided through scalable pipelines like EukHeist, stand to advance our understanding of eukaryotic diversity through increased coverage of genomic representatives across the tree of life. IMPORTANCE Single-celled eukaryotes play ecologically significant roles in the marine environment, yet fundamental questions about their biodiversity, ecological function, and interactions remain. Environmental sequencing enables researchers to document naturally occurring protistan communities, without culturing bias, yet metagenomic and metatranscriptomic sequencing approaches cannot separate individual species from communities. To more completely capture the genomic content of mixed protistan populations, we can create bins of sequences that represent the same organism (metagenome-assembled genomes [MAGs]). We developed the EukHeist pipeline, which automates the binning of population-level eukaryotic and prokaryotic genomes from metagenomic reads. We show exciting insight into what protistan communities are present and their trophic roles in the ocean. Scalable computational tools, like EukHeist, may accelerate the identification of meaningful genetic signatures from large data sets and complement researchers’ efforts to leverage MAG databases for addressing ecological questions, resolving evolutionary relationships, and discovering potentially novel biodiversity.

59 BASIC BIOLOGICAL SCIENCES↗

Uncertainty Quantification of Metal Additive Manufacturing Processing Conditions Through the use of Exascale Computing

Metal additive manufacturing (AM) is a disruptive manufacturing technology that opens the design space for parts outside those possible from traditional manufacturing methods. In order to accelerate industry and R&D needs to certify AM parts, the Exascale Additive Manufacturing project (ExaAM) has developed a suite of exascale-ready computational tools to model the process-to-structure-to-properties (PSP) relationship for additively manufactured metal components. One such tool is an uncertainty quantification (UQ) pipeline to quantify the effect that uncertainty in processing conditions has on local mechanical responses. We present an overview of this pipeline and its required simulation and workflow codes. Using the Oak Ridge National Laboratory’s (ORNL) exascale computer, Frontier, we utilize this pipeline to cross multiple length and time scales to predict the local mechanical response of a location within a complex AM bridge part, AMB2018-01 produced by the National Institute of Standards and Technology (NIST) as part of their 2018 AM-Bench test series. Our results are then compared to experimental mechanical tests of parts from the NIST build to quantify the error in the ExaAM UQ workflow.

Carson, Robert↗

Panel-Segmentation [SWR-21-18]

Panel-Segmentation contains the scripts for automated metadata extraction of solar PV installations, using satellite imagery coupled with computer vision techniques. In this package, the user can perform the following actions: *Automatically generate a satellite image using a set of lat-long coordinates, and a Google Maps API key. Users would need to set up a Google Cloud account and get a Maps Static API key. Please refer to Setting Up Google Maps Static API Key section for this process. *Perform image segmentation on the satellite image, to locate the solar array(s) in the image on a pixel-by-pixel basis, using an image segmentation model (panel_detection_model.pth). Get classification of the installation (rooftop, ground mounted fixed-tilt or tracking, carport, etc). *Perform azimuth estimation on each solar array cluster in the masked image. *Detect solar panels and get its latitude, longitude, and address within a geographic bounding box through the SOL-Searcher Pipeline. *Detect and calculate hurricane damage on solar installations given pre-hurricane and post-hurricane satellite imagery through the Hurricane Detection Pipeline. *Detect and calculate hail damage on solar installations given satellite imagery through the Hail Detection pipeline. *Convert NOAA MESH (Maximum Estimated Size of Hail) grib2 files into kml or geojson files. *Estimate tilt and azimuth of a solar array by processing USGS LiDAR data for the array’s location.

Edun, Ayobami↗

SynBio QC-Library QC (LibQC) v2.0.0

This software is a pipeline used to identify the distribution of user-specified sequences in oligo pools that have been sequenced on the MiSeq platform for quality control purposes. The software counts instances of user-specified sequences (inserts or spacers) identified in sequencing data, returning the distribution of sequences in each pool and the length distribution of all the sequences. Among other types similar types of projects, this can be used to assess the quality of a guideRNA library once it has been inserted into the vector of choice to ensure that there is an even distribution of all the individual protospacers in the pool. This pipeline uses many freely available open source libraries, the value added is that in our application the steps of the pipeline are defined in Workflow Description Language (WDL) and run through the Cromwell workflow engine in Docker containers, for easy distribution and set up.

Simirenko, Lisa↗

BioSiting Tool (BioSiting) v2

The BioSiting Tool provides a geospatial interface for analyzing bioeconomy resources and infrastructure across the continental U.S. The tool integrates empirical and modeled data from a broad range of sources. Bioeconomy resources mapped in the tool include agricultural residues, forest residues, municipal solid waste streams, food waste, manure, fats, oils and greases and potential yields of energy crops. Infrastructure mapped in the tool includes biorefineries, material recovery facilities, anaerobic digesters, wastewater treatment plants, combustion plants, district energy systems, crude oil pipelines, petroleum pipelines, natural gas pipelines, railways and freight terminals. Additional data layers include environmental justice indicators at the census tract level and carbon dioxide geologic storage potential. Users can select a location on the map, define a buffer radius in kilometers and generate an inventory of all bioecomony resources within the buffer zone. Data from the tool can be downloaded from individual buffer zones, or at the state or national level.

Huntington, Tyler↗

ADEPT: a domain independent sequence alignment strategy for gpu architectures

Bioinformatic workflows frequently make use of automated genome assembly and protein clustering tools. At the core of most of these tools, a significant portion of execution time is spent in determining optimal local alignment between two sequences. This task is performed with the Smith-Waterman algorithm, which is a dynamic programming based method. With the advent of modern sequencing technologies and increasing size of both genome and protein databases, a need for faster Smith-Waterman implementations has emerged. Multiple SIMD strategies for the Smith-Waterman algorithm are available for CPUs. However, with the move of HPC facilities towards accelerator based architectures, a need for an efficient GPU accelerated strategy has emerged. Existing GPU based strategies have either been optimized for a specific type of characters (Nucleotides or Amino Acids) or for only a handful of application use-cases. In this paper, we present ADEPT, a new sequence alignment strategy for GPU architectures that is domain independent, supporting alignment of sequences from both genomes and proteins. Our proposed strategy uses GPU specific optimizations that do not rely on the nature of sequence. We demonstrate the feasibility of this strategy by implementing the Smith-Waterman algorithm and comparing it to similar CPU strategies as well as the fastest known GPU methods for each domain. ADEPT’s driver enables it to scale across multiple GPUs and allows easy integration into software pipelines which utilize large scale computational systems. We have shown that the ADEPT based Smith-Waterman algorithm demonstrates a peak performance of 360 GCUPS and 497 GCUPs for protein based and DNA based datasets respectively on a single GPU node (8 GPUs) of the Cori Supercomputer. Overall ADEPT shows 10x faster performance in a node-to-node comparison against a corresponding SIMD CPU implementation. ADEPT demonstrates a performance that is either comparable or better than existing GPU strategies. We demonstrated the efficacy of ADEPT in supporting existing bionformatics software pipelines by integrating ADEPT in MetaHipMer a high-performance denovo metagenome assembler and PASTIS a high-performance protein similarity graph construction pipeline. Our results show 10% and 30% boost of performance in MetaHipMer and PASTIS respectively.

59 BASIC BIOLOGICAL SCIENCES↗

Scalable production of recombinant three-finger proteins: from inclusion bodies to high quality molecular probes

The three-finger proteins are a collection of disulfide bond rich proteins of great biomedical interests. Scalable recombinant expression and purification of bioactive three-finger proteins is quite difficult. We introduce a working pipeline for expression, purification and validation of disulfide-bond rich three-finger proteins using E. coli as the expression host. With this pipeline, we have successfully obtained highly purified and bioactive recombinant α-Βungarotoxin, k-Bungarotoxin, Hannalgesin, Mambalgin-1, α-Cobratoxin, MTα, Slurp1, Pate B etc. Milligrams to hundreds of milligrams of recombinant three finger proteins were obtained within weeks in the lab. The recombinant proteins showed specificity in binding assay and six of them were crystallized and structurally validated using X-ray diffraction protein crystallography. Our pipeline allows refolding and purifying recombinant three finger proteins under optimized conditions and can be scaled up for massive production of three finger proteins. As many three finger proteins have attractive therapeutic or research interests and due to the extremely high quality of the recombinant three finger proteins we obtained, our method provides a competitive alternative to either their native counterparts or chemically synthetic ones and should facilitate related research and applications.

59 BASIC BIOLOGICAL SCIENCES↗