Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data platform”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,783 records · Page 99

spammR: an R package designed for analysis and integration of spatial multi-omic measurements

Spatial omics is a young and evolving field and as such shows rapid development of novel technologies and analysis methods to measure transcripts, proteins, metabolites, and post-translational modifications at high spatial resolution. These advances in technology have enabled the simultaneous generation of abundance profiles for multiple different omics types and associated microscopy imaging data, as well as their analysis in a spatial context. However, most analytical tools are designed for spatial transcriptomics platforms and are challenging to use in other contexts such as mass spectrometry-based measurements or metagenomics. To this end we present spammR (spatial analysis of multi-omics measurements in R), an R package that enables end-to-end analysis with a specific focus on mass-spectrometry derived spatial omics datasets with (1) smaller sample sizes and spatial sparsity of samples, (2) considerable missingness, and (3) no a-priori knowledge about proteins or genes of interest, relying on a fully data-driven approach.

spammR↗

Metagenome-assembled genome extraction and analysis from microbiomes using KBase

Uncultivated Bacteria and Archaea account for the vast majority of species on Earth, but obtaining their genomes directly from the environment, using shotgun sequencing, has only become possible recently. In order to realize the hope of capturing Earth’s microbial genetic complement and to facilitate the investigation of the functional roles of specific lineages in a given ecosystem, technologies that accelerate the recovery of high-quality genomes are necessary. We present a series of analysis steps and data products for the extraction of high-quality metagenome-assembled genomes (MAGs) from microbiomes using the U.S. Department of Energy Systems Biology Knowledgebase (KBase) platform (http://www.kbase.us/). Overall, these steps take about a day to obtain extracted genomes when starting from smaller environmental shotgun read libraries, or up to about a week from larger libraries. In KBase, the process is end-to-end, allowing a user to go from the initial sequencing reads all the way through to MAGs, which can then be analyzed with other KBase capabilities such as phylogenetic placement, functional assignment, metabolic modeling, pangenome functional profiling, RNA-Seq and others. While portions of such capabilities are available individually from other resources, the combination of the intuitive usability, data interoperability and integration of tools in a freely available computational resource makes KBase a powerful platform for obtaining MAGs from microbiomes. While this workflow offers tools for each of the key steps in the genome extraction process, it also provides a scaffold that can be easily extended with additional MAG recovery and analysis tools, via the KBase software development kit (SDK).

59 BASIC BIOLOGICAL SCIENCES↗

Performance Analysis, Design Considerations, and Applications of Extreme-Scale In Situ Infrastructures

A key trend facing extreme-scale computational science is the widening gap between computational and I/O rates, and the challenge that follows is how to best gain insight from simulation data when it is increasingly impractical to save it to persistent storage for subsequent visual exploration and analysis. One approach to this challenge is centered around the idea of in situ processing, where visualization and analysis processing is performed while data is still resident in memory. This paper examines several key design and performance issues related to the idea of in situ processing at extreme scale on modern platforms: Scalability, overhead, performance measurement and analysis, comparison and contrast with a traditional post hoc approach, and interfacing with simulation codes. We illustrate these principles in practice with studies, conducted on large-scale HPC platforms, that include a miniapplication and multiple science application codes, one of which demonstrates in situ methods in use at greater than 1M-way concurrency.

Ayachit, Utkarsh↗

Planck 2015 results: XII. Full focal plane simulations

In this paper, we present the 8th full focal plane simulation set (FFP8), deployed in support of the Planck 2015 results. FFP8 consists of 10 fiducial mission realizations reduced to 18 144 maps, together with the most massive suite of Monte Carlo realizations of instrument noise and CMB ever generated, comprising 10 4 mission realizations reduced to about 10 6 maps. The resulting maps incorporate the dominant instrumental, scanning, and data analysis effects, and the remaining subdominant effects will be included in future updates. Finally, generated at a cost of some 25 million CPU-hours spread across multiple high-performance-computing (HPC) platforms, FFP8 is used to validate and verify analysis algorithms and their implementations, and to remove biases from and quantify uncertainties in the results of analyses of the real data.

79 ASTRONOMY AND ASTROPHYSICS↗

16s Amplicon Analysis of Soil Data for Interactive effects of depth and differential irrigation on soil microbiome composition and functioning

Genomic DNA was isolated from soil and rhizosphere samples using the Zymo Quick-DNA fecal/soil microbe miniprep kit (catalog no. D6010) according to the manufacturer’s instructions (Zymo Research; Irvine, CA) with the modification of eluting in 100 uL elution buffer. Sample concentrations were quantified using the Qubit dsDNA HS assay kit (Thermo Fisher). For rhizosphere samples only, DNA was subsequently purified using Zymo’s ZR-96 DNA Clean & Concentrator kit (catalog no. D4024) to account for low DNA concentrations of these samples. . In each replicate block, there were five drip irrigation treatments (T1 = 100% normal irrigation, T2 = 56.25%, T3 = 37.5%, T4 = 18.75% and T5=no irrigation. On July 20, the strength of the drought treatments was increased: T1 remained at 100%, whereas T2 changed from 75% to 56.25%, T2 changed from 50% to 37.5%, and T4 changed from 25% to 18.75%. Sequencing was performed as described previously (Naylor, Fansler, et al. 2020). Sequences were amplified on the MiSeq platform (Illumina, San Diego, CA) using 16S primers (515F and 806R) specific to the V4 region. Raw sequence data was processed with the pipeline Hundo for amplicon quality control and annotation. Downstream statistical analyses on 16S datasets were performed using the program R and the packages ‘phyloseq’ and ‘vegan’.

Soil microbiome, metatranscriptomics↗

Development of an Ion Mobility Spectrometry-Orbitrap Mass Spectrometer Platform

Complex samples benefit from multidimensional measurements where higher resolution enables more complete characterization of biological and environmental systems. To address this challenge, we developed a drift tube-based ion mobility spectrometry-Orbitrap mass spectrometer (IMS-Orbitrap MS) platform. To circumvent the time scale disparity between the fast IMS separation and the much slower Orbitrap MS acquisition, we utilized a dual gate and pseudorandom sequences to multiplexed injection of ions and allowing operation in signal averaging (SA), single multiplexing (SM) and double multiplexing (DM) IMS modes to optimize the signal-to-noise ratio of the measurements. For the SM measurements, a previously developed algorithm was used to reconstruct the IMS data. A new algorithm was developed for the DM analyses involving a two-step process that first recovers the SM data and then decodes the SM data. The algorithm also performs multiple refining procedures in order to minimize demultiplexing artifacts. The new IMS-Orbitrap MS platform was demonstrated by the analysis of proteomic and petroleum samples, where the integration of IMS and high mass resolution proved essential for accurate assignment of molecular formulae.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

A Co-design Framework for Online Data Analysis and Reduction

Science applications preparing for the exascale era are increasingly exploring in situ computations comprising of simulation-analysis-reduction pipelines coupled in-memory. Efficient composition and execution of such complex pipelines for a target platform is a codesign process that evaluates the impact and tradeoffs of various application- and system-specific parameters. In this article, we describe a toolset for automating performance studies of composed HPC applications that perform online data reduction and analysis. We describe Cheetah, a new framework for composing parametric studies on coupled applications, and Savanna, a runtime engine for orchestrating and executing campaigns of codesign experiments. Furthermore, this toolset facilitates understanding the impact of various factors such as process placement, synchronicity of algorithms, and storage versus compute requirements for online analysis of large data. Ultimately, we aim to create a catalog of performance results that can help scientists understand tradeoffs when designing next-generation simulations that make use of online processing techniques. We illustrate the design of Cheetah and Savanna, and present application examples that use this framework to conduct codesign studies on small clusters as well as leadership class supercomputers.

97 MATHEMATICS AND COMPUTING↗

KG-Hub—building and exchanging biological knowledge graphs

Knowledge graphs (KGs) are a powerful approach for integrating heterogeneous data and making inferences in biology and many other domains, but a coherent solution for constructing, exchanging, and facilitating the downstream use of KGs is lacking. Here we present KG-Hub, a platform that enables standardized construction, exchange, and reuse of KGs. Features include a simple, modular extract–transform–load pattern for producing graphs compliant with Biolink Model (a high-level data model for standardizing biological data), easy integration of any OBO (Open Biological and Biomedical Ontologies) ontology, cached downloads of upstream data sources, versioned and automatically updated builds with stable URLs, web-browsable storage of KG artifacts on cloud infrastructure, and easy reuse of transformed subgraphs across projects. Current KG-Hub projects span use cases including COVID-19 research, drug repurposing, microbial–environmental interactions, and rare disease research. KG-Hub is equipped with tooling to easily analyze and manipulate KGs. KG-Hub is also tightly integrated with graph machine learning (ML) tools which allow automated graph ML, including node embeddings and training of models for link prediction and node classification.

59 BASIC BIOLOGICAL SCIENCES↗

Build and Execute Environment

At exascale, the challenge becomes to develop applications that run at scale and use exascale platforms reliably, efficiently, and flexibly. Workflows become much more complex because they must seamlessly integrate simulation and data analytics. They must include down-sampling, post-processing, feature extraction, and visualization. Power and data transfer limitations require these analysis tasks to be run in-situ or in-transit. We expect successful workflows will comprise multiple linked simulations along with tens of analysis routines. Users will have limited development time at scale and, therefore, must have rich tools to develop, debug, test, and deploy applications. At this scale, successful workflows will compose linked computations from an assortment of reliable, well-defined computation elements, ones that can come and go as required, based on the needs of the workflow over time. We propose a novel framework that utilizes both virtual machines (VMs) and software containers to create a workflow system that establishes a uniform build and execution environment (BEE) beyond the capabilities of current systems. In this environment, applications will run reliably and repeatably across heterogeneous hardware and software. Containers, both commercial (Docker and Rocket) and open-source (LXC and LXD), define a runtime that isolates all software dependencies from the machine operating system. Workflows may contain multiple containers that run different operating systems, different software, and even different versions of the same software. We will run containers in open-source virtual machines (KVM) and emulators (QEMU) so that workflows run on any machine entirely in user-space. On this platform of containers and virtual machines, we will deliver workflow software that provides services, including repeatable execution, provenance, checkpointing, and future proofing. We will capture provenance about how containers were launched and how they interact to annotate workflows for repeatable and partial re-execution. We will coordinate the physical snapshots of virtual machines with parallel programming constructs, such as barriers, to automate checkpoint and restart. We will also integrate with HPC-specific container runtimes to gain access to accelerators and other specialized hardware to preserve native performance. Containers will link development to continuous integration. When application developers check code in, it will automatically be tested on a suite of different software and hardware architectures.

Guan, Qiang↗

Build and Execute Environment

At exascale, the challenge becomes to develop applications that run at scale and use exascale platforms reliably, efficiently, and flexibly. Workflows become much more complex because they must seamlessly integrate simulation and data analytics. They must include down-sampling, post-processing, feature extraction, and visualization. Power and data transfer limitations require these analysis tasks to be run in-situ or in-transit. We expect successful workflows will comprise multiple linked simulations along with tens of analysis routines. Users will have limited development time at scale and, therefore, must have rich tools to develop, debug, test, and deploy applications. At this scale, successful workflows will compose linked computations from an assortment of reliable, well-defined computation elements, ones that can come and go as required, based on the needs of the workflow over time. We propose a novel framework that utilizes both virtual machines (VMs) and software containers to create a workflow system that establishes a uniform build and execution environment (BEE) beyond the capabilities of current systems. In this environment, applications will run reliably and repeatably across heterogeneous hardware and software. Containers, both commercial (Docker and Rocket) and open-source (LXC and LXD), define a runtime that isolates all software dependencies from the machine operating system. Workflows may contain multiple containers that run different operating systems, different software, and even different versions of the same software. We will run containers in open-source virtual machines (KVM) and emulators (QEMU) so that workflows run on any machine entirely in user-space. On this platform of containers and virtual machines, we will deliver workflow software that provides services, including repeatable execution, provenance, checkpointing, and future proofing. We will capture provenance about how containers were launched and how they interact to annotate workflows for repeatable and partial re-execution. We will coordinate the physical snapshots of virtual machines with parallel programming constructs, such as barriers, to automate checkpoint and restart. We will also integrate with HPC-specific container runtimes to gain access to accelerators and other specialized hardware to preserve native performance. Containers will link development to continuous integration. When application developers check code in, it will automatically be tested on a suite of different software and hardware architectures.

Guan, Qiang↗

Vision 20/20: Automation and advanced computing in clinical radiation oncology

This Vision 20/20 paper considers what computational advances are likely to be implemented in clinical radiation oncology in the coming years and how the adoption of these changes might alter the practice of radiotherapy. Four main areas of likely advancement are explored: cloud computing, aggregate data analyses, parallel computation, and automation. As these developments promise both new opportunities and new risks to clinicians and patients alike, the potential benefits are weighed against the hazards associated with each advance, with special considerations regarding patient safety under new computational platforms and methodologies. While the concerns of patient safety are legitimate, the authors contend that progress toward next-generation clinical informatics systems will bring about extremely valuable developments in quality improvement initiatives, clinical efficiency, outcomes analyses, data sharing, and adaptive radiotherapy.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Measuring Atmospheric Gas Using Small Satellites (Final Report)

The research focused on demonstrating that a small satellite (< 10 kg) is able to provide useable data in support of stratospheric circulation and broader atmospheric measurement generally thought to required large, lengthy and high cost space programs. The research goals targeting enabling CubeSats to serve as science platforms including low-cost communication infrastructure, science enabling solar panels and a ruggedized small form factor science instrument were highly successful with demonstrated on-orbit results. The collection of science data on-orbit was not successful due to a high tumbling rate impacted to the spacecraft at time of release on orbit, preventing reliable solar charging and leading to premature ending of the mission. The successes acquired on orbit will act as a springboard for further research and space projects at LLNL. Several research groups (both private and government) have expressed interest in the area of this research that proved successful such as the solar panels and iridium crosslink as well as the LLNL developed CubeSat standard.

42 ENGINEERING↗

AMPX: A Modern cross section processing system for generating nuclear data libraries - 14619

Nuclear data libraries are essential for nuclear analysis code systems like SCALE. AMPX has been continuously developed since the early 1970's and is used to generate these data libraries for SCALE. Recently, Oak Ridge National Laboratory (ORNL) initiated a major modernization effort to position AMPX for continued development and deployment on modern computing platforms. An important step toward modernization has been to merge AMPX under the SCALE continuous integration (CI) development framework. As a result, AMPX is now developed under the SCALE software quality assurance plan, providing increased confidence in the quality of data libraries developed and deployed by AMPX. This merge allowed us to develop novel methods to generate self-shielding factors that combine SCALE's resonance self-shielding capabilities with the AMPX library-generation capabilities. AMPX processes Evaluated Nuclear Data Format (ENDF/B) files. The ENDF community is working to develop a more modern format that will replace the existing ENDF/B format. Due to strict space and memory requirements when AMPX was first developed, the original AMPX processing and reading of ENDF data were tightly integrated. On newer computer systems, this is no longer necessary. Now there is a separation between reading and processing which allows for simultaneous support of current and new ENDF formats. As a result, AMPX is positioned to adopt any new ENDF format when available. For end users, the modernization effort has resulted in improved processing capabilities for generating nuclear data libraries. The objective of the paper is to document the latest AMPX cross-section processing capabilities and provide the current status of the AMPX modernization effort. (authors)

42 ENGINEERING↗

Combining Hadoop with MPI to Solve Metagenomics Problems that are both Data- and Compute-intensive

Metagenomics, the study of all microbial species cohabitants in an environment, usually produces large amount of sequence data varying from several GBs to a few TBs. Analyzing metagenomics data includes both data-intensive and compute-intensive steps, making the entire process hard to scale. Here we aim to optimize a metagenomics application that partitions the shortgun metagenomics sequences based on their species of origin. Our solution combines MapReduce-based BioPig analytic toolkit with MPI to provide scalability in respective to both data and compute. We also made some improvements to the existing BioPig toolkit by using simplified data types and compressed k-mer storage. These optimizations leads up to 193× speedup for the computing-intensive step and 9.6× speedup over the entire pipeline. Our optimized application is also capable of processing datasets that are 16 times larger on the same hardware platform. These conclusions indicate integrating heterogeneous technologies such as Hadoop and MPI is quite efficient to solve large genomics problems that are both data-intensive and compute-intensive.

97 MATHEMATICS AND COMPUTING↗

Flexible and Accessible 4D Subsurface Visualization Using a Web-Based Platform

4D subsurface visualization using a web platform can provide improved communication, education and outreach to non-experts and stakeholders. It allows for the improved understanding of complex relationships and interactions that occur in inaccessible locations of which many experimental testbeds exist in. The use of a web-based visualization tool lessens the cognitive impact further by reducing the need of specialized software. Users can define their own exploration of a 3D or 4D scene, adding and removing data as needed and are able to make informed decisions based on accurate model visualizations, while providing a collaborative tool that can be accessed on any platform using a browser window. The tool is also able to point to real time streams to display up-to-the-second data as an experiment is ongoing. This level of latency can also provide operators with essential information that can direct an experiment’s progress. Additionally, this tool can leverage augmented and virtual reality (AR, VR) capabilities of certain mobile devices and head-mounted displays, providing further engaging visualization possibilities.

Pratt, Martin J.↗

Application of an energy-dependent instrument response function to analysis of nTOF data from cryogenic DT experiments

Neutron time-of-flight (nTOF) detectors are used to diagnose the conditions present in inertial confinement fusion (ICF) experiments and basic laboratory physics experiments performed on an ICF platform. The instrument response function (IRF) of these detectors is constructed by convolution of two components: an x-ray IRF and a neutron interaction response. The shape of the neutron interaction response varies with incident neutron energy, changing the shape of the total IRF. Analyses of nTOF data that span a broad range of energies must account for this energy-dependence in order to accurately infer plasma parameters and nuclear properties in ICF experiments. This work briefly reviews a matrix multiplication approach to convolution which allows for an energy-dependent change in the shape of the IRF. This method is applied to synthetic data resembling symmetric cryogenic DT implosions to examine the effect of the energy-dependent IRF on the inferred areal density. Here, results of forward fits that infer ion temperatures and areal densities from nTOF data collected during cryogenic DT experiments on OMEGA are also discussed.

47 OTHER INSTRUMENTATION↗

An OpenMP GPU-offload implementation of a non-equilibrium solidification cellular automata model for additive manufacturing

Here, in this paper, performance strategies on GPU-based HPC platforms of a cellular automata (CA) simulation code for non-equilibrium solidification, including nucleation, grain growth, solute partitioning and transport for the metal additive manufacturing (AM) process are investigated using OpenMP 4.5. To accurately report the speed-up for multicore CPUs and GPUs, a rigorous performance analysis employed optimizations appropriate for both CPU-only code (baseline) and GPU offload codes for an isothermal test problem. The performance results on Summit at the Oak Ridge Leadership Computing Facility indicate that using a precomputed list of interface cells significantly decreased the wall-clock time on GPUs. The speedup due to GPU acceleration was evaluated for a full Summit node and measured to be 1.8X when comparing a 6 MPI tasks run with 6 GPUs versus 36 MPI tasks on the CPU only. That speed-up was found to be 7.9X when comparing 6 MPI tasks with 6 GPUs versus the 6 MPI tasks running on the CPU only. Performance measurements showed that system total time is almost constant for runs with more than 96 MPI tasks (or GPUs), indicating that the GPU-accelerated code showed an excellent weak scaling performance. Finally, a rapid directional solidification problem was considered to demonstrate the CA code capability on Summit. It was found that a mesh size of at least 0.05 μm is recommended for the AM-like simulations in order to obtain accurate elongated grain microstructure and elongated subgrain features, which are in qualitative good agreement with experimental data. The results presented in this study indicate that the performance strategies on GPU-based HPC platforms for the CA code are appropriate for novel HPC exascale platforms.

36 MATERIALS SCIENCE↗

Evaluation of a scientific data search infrastructure

The ability to search over large scientific datasets has become crucial to next-generation scientific discoveries as data generated from scientific facilities grow dramatically. In previous work, we developed and deployed ScienceSearch, a search infrastructure for scientific data which uses machine learning to automate metadata creation. Our current deployment is deployed atop a container based platform at a HPC center. In this article, we present an evaluation and discuss our experiences with the ScienceSearch infrastructure. Specifically, we present a performance evaluation of ScienceSearch's infrastructure focusing on scalability trends. The obtained results show that ScienceSearch is able to serve up to 130 queries/min with latency under 3 s. We discuss our infrastructure setup and evaluation results to provide our experiences and a perspective on opportunities and challenges of our search infrastructure.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗