Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Science Pipeline”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 595 records · Page 33

Interactive 3D Mars Visualization

The Interactive 3D Mars Visualization system provides high-performance, immersive visualization of satellite and surface vehicle imagery of Mars. The software can be used in mission operations to provide the most accurate position information for the Mars rovers to date. When integrated into the mission data pipeline, this system allows mission planners to view the location of the rover on Mars to 0.01-meter accuracy with respect to satellite imagery, with dynamic updates to incorporate the latest position information. Given this information so early in the planning process, rover drivers are able to plan more accurate drive activities for the rover than ever before, increasing the execution of science activities significantly. Scientifically, this 3D mapping information puts all of the science analyses to date into geologic context on a daily basis instead of weeks or months, as was the norm prior to this contribution. This allows the science planners to judge the efficacy of their previously executed science observations much more efficiently, and achieve greater science return as a result. The Interactive 3D Mars surface view is a Mars terrain browsing software interface that encompasses the entire region of exploration for a Mars surface exploration mission. The view is interactive, allowing the user to pan in any direction by clicking and dragging, or to zoom in or out by scrolling the mouse or touchpad. This set currently includes tools for selecting a point of interest, and a ruler tool for displaying the distance between and positions of two points of interest. The mapping information can be harvested and shared through ubiquitous online mapping tools like Google Mars, NASA WorldWind, and Worldwide Telescope.

Powell, Mark W.↗

Sequence length scaling in vision transformers for scientific images on frontier

Vision Transformers (ViTs) are pivotal for foundational models in scientific imagery, including Earth science applications, due to their capability to process large sequence lengths. While transformers for text have inspired scaling sequence lengths in ViTs, adapting these for ViTs introduces unique challenges. We develop distributed sequence parallelism for ViTs, enabling them to handle up to 1M tokens. Our approach, leveraging DeepSpeed-Ulysses and Long-Sequence-Segmentation with model sharding, is the first to apply sequence parallelism in ViT training, achieving a 94% batch scaling efficiency on 2,048 AMD-MI250X GPUs. Evaluating sequence parallelism in ViTs, particularly in models up to 10B parameters, highlighted substantial bottlenecks. We countered these with hybrid sequence, pipeline, and flash attention strategies, to scale beyond single GPU memory limits. Our method significantly enhances climate modeling accuracy by 20% in temperature predictions, marking the first training of a vision transformer model to convergence with a sequence length of 188K tokens, using full self-attention.

Tsaris, Aristeidis (aris) [ORNL] (ORCID:0000000277↗

The SEED and the Rapid Annotation of microbial genomes using Subsystems Technology (RAST)

In 2004, the SEED (http://pubseed.theseed.org/) was created to provide consistent and accurate genome annotations across thousands of genomes and as a platform for discovering and developing de novo annotations. The SEED is a constantly updated integration of genomic data with a genome database, web front end, API and server scripts. It is used by many scientists for predicting gene functions and discovering new pathways. In addition to being a powerful database for bioinformatics research, the SEED also houses subsystems (collections of functionally related protein families) and their derived FIGfams (protein families), which represent the core of the RAST annotation engine (http://rast. nmpdr.org/). When a new genome is submitted to RAST, genes are called and their annotations are made by comparison to the FIGfam collection. If the genome is made public, it is then housed within the SEED and its proteins populate the FIGfam collection. This annotation cycle has proven to be a robust and scalable solution to the problem of annotating the exponentially increasing number of genomes. To date, >12 000 users worldwide have annotated >60 000 distinct genomes using RAST. Here we describe the interconnectedness of the SEED database and RAST, the RAST annotation pipeline and updates to both resources.

59 BASIC BIOLOGICAL SCIENCES↗

Microbial Community Analysis with Ribosomal Gene Fragments from Shotgun Metagenomes

Shotgun metagenomic sequencing does not depend on gene-targeted primers or PCR amplification; thus, it is not affected by primer bias or chimeras. However, searching rRNA genes from large shotgun Illumina data sets is computationally expensive, and no approach exists for unsupervised community analysis of small-subunit (SSU) rRNA gene fragments retrieved from shotgun data. We present a pipeline, SSUsearch, to achieve the faster identification of short-subunit rRNA gene fragments and enabled unsupervised community analysis with shotgun data. It also includes classification and copy number correction, and the output can be used by traditional amplicon analysis platforms. Shotgun metagenome data using this pipeline yielded higher diversity estimates than amplicon data but retained the grouping of samples in ordination analyses. We applied this pipeline to soil samples with paired shotgun and amplicon data and confirmed bias against Verrucomicrobia in a commonly used V6-V8 primer set, as well as discovering likely bias against Actinobacteria and for Verrucomicrobia in a commonly used V4 primer set. In conclusion, this pipeline can utilize all variable regions in SSU rRNA and also can be applied to large-subunit (LSU) rRNA genes for confirmation of community structure.

59 BASIC BIOLOGICAL SCIENCES↗

Integrated System and Application Continuous Performance Monitoring and Analysis Capability

Scientific applications run on high-performance computing (HPC) systems are critical for many national security missions within Sandia and the NNSA complex. However, these applications often face performance degradation and even failures that are challenging to diagnose. To provide unprecedented insight into these issues, the HPC Development, HPC Systems, Computational Science, and Plasma Theory & Simulation departments at Sandia crafted and completed their FY21 ASC Level 2 milestone entitled "Integrated System and Application Continuous Performance Monitoring and Analysis Capability." The milestone created a novel integrated HPC system and application monitoring and analysis capability by extending Sandia's Kokkos application portability framework, Lightweight Distributed Metric Service (LDMS) monitoring tool, and scalable storage, analysis, and visualization pipeline. The extensions to Kokkos and LDMS enable collection and storage of application data during run time, as it is generated, with negligible overhead. This data is combined with HPC system data within the extended analysis pipeline to present relevant visualizations of derived system and application metrics that can be viewed at run time or post run. This new capability was evaluated using several week-long, 290-node runs of Sandia's ElectroMagnetic Plasma In Realistic Environments ( EMPIRE ) modeling and design tool and resulted in 1TB of application data and 50TB of system data. EMPIRE developers remarked this capability was incredibly helpful for quickly assessing application health and performance alongside system state. In short, this milestone work built the foundation for expansive HPC system and application data collection, storage, analysis, visualization, and feedback framework that will increase total scientific output of Sandia's HPC users.

97 MATHEMATICS AND COMPUTING↗

Integrated System and Application Continuous Performance Monitoring and Analysis Capability (Final)

Scientific applications run on high-performance computing (HPC) systems are critical for many national security missions within Sandia and the NNSA complex. However, these applications often face performance degradation and even failures that are challenging to diagnose. To provide unprecedented insight into these issues, the HPC Development, HPC Systems, Computational Science, and Plasma Theory & Simulation departments at Sandia crafted and completed their FY21 ASC Level 2 milestone entitled "Integrated System and Application Continuous Performance Monitoring and Analysis Capability." The milestone created a novel integrated HPC system and application monitoring and analysis capability by extending Sandia’s Kokkos application portability framework, Lightweight Distributed Metric Service (LDMS) monitoring tool, and scalable storage, analysis, and visualization pipeline. The extensions to Kokkos and LDMS enable collection and storage of application data during run time, as it is generated, with negligible overhead. This data is combined with HPC system data within the extended analysis pipeline to present relevant visualizations of derived system and application metrics that can be viewed at run time or post run. This new capability was evaluated using several week-long, 290-node runs of Sandia’s ElectroMagnetic Plasma In Realistic Environments (EMPIRE) modeling and design tool and resulted in 1TB of application data and 50TB of system data. EMPIRE developers remarked this capability was incredibly helpful for quickly assessing application health and performance alongside system state. In short, this milestone work built the foundation for expansive HPC system and application data collection, storage, analysis, visualization, and feedback framework that will increase total scientific output of Sandia’s HPC users.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Blast from the Past: ASDC Curation for NASA Suborbital Legacy Missions to Promote Data Discovery and Accessibility

NASA has an extensive history of conducting suborbital field campaigns to further advances in atmospheric sciences. Beginning with the Chemical Instrument Test and Evaluation (CITE) conducted in 1983-1984, NASA has completed many suborbital campaigns over the past three decades. Since the early 2010s, suborbital missions are typically assigned to a NASA Distributed Active Archive Center (DAAC) prior to the mission for long-term archival and distribution. Efforts are being made by NASA’s Earth Science Data and Information System (ESDIS) Project and the Airborne Data Management Group (ADMG) to assign legacy missions to DAACs for permanent archival and distribution, so that these valuable datasets remain to be available to the scientific community. NASA’s Atmospheric Science Data Center (ASDC) has been named the assigned DAAC for nearly 20 atmospheric composition legacy missions, including missions conducted as part of the Global Tropospheric Experiment (GTE) and expects to be named the assigned DAAC for more of these missions over the next few years. The primary goal of the ASDC is to provide access to the datasets as they are currently formatted to the broad user community and enhance their findability and accessibility. However, data reporting standards have evolved significantly since 1983 and the datasets span a wide variety of file formats, including text, Ames, GTE, and ICARTT (International Consortium for Atmospheric Research on Transport and Transformation), and the amount of metadata and relevant information included in the files also varies greatly and can not be readily extracted without subject matter knowledge. This has caused challenges for the ASDC’s suborbital metadata extraction pipeline in ensuring that accurate and necessary metadata is being provided for the missions by all the ASDC’s existing search mechanisms. To make the data more findable and accessible, the ASDC has begun researching ways to further enhance the datasets, including distributing value-added products (i.e. consistent file format such as ICARTT or netCDF), adding standard names from the ESDIS Standards Coordination Office (ESCO)-approved Atmospheric Composition Variable Standard Names Convention (ACVSNC), and creating outreach materials such as ArcGIS StoryMaps, User Guides, and Micro Articles, providing overviews of the missions and what type of data was collected during the missions. These efforts also help support NASA’s Open-Source Science by enhancing the FAIRness of the legacy data products. This presentation will review the ASDC’s ongoing efforts, progress made, and future plans for legacy missions.

Megan Buzanowicz↗

ThunderSecure: deploying real-time intrusion detection for 100G research networks by leveraging stream-based features and one-class classification network

Nowadays, data generated by large-scale scientific experiments are on the scale of petabytes per month. These data are transferred through dedicated high-bandwidth networks (40/100G) across distributed sites for processing, storage, and analysis. Like general purpose networks, research networks experience intrusions. However, monitoring anomalies in such high-speed network traffics is challenging given current cyber-infrastructure. Moreover, traditional network intrusion detection systems (NIDS) are signature based. However, anomaly patterns are difficult to define and that rulesets are often not updated frequently enough to reflect the changes of attack behaviors. We present ThunderSecure, a high-throughput, unsupervised learning-based intrusions detection system for 100G research networks. ThunderSecure implements an efficient packet processing and detection pipeline using multi-cores and GPUs. It extracts statistical and temporal features from real-time network data streams and feeds them to a one-class anomaly detection network. A baseline of normal distribution will be created based on the training observation. Testing traffic deviated from the learned profile will be marked as anomalies. We trained ThunderSecure on hundreds of billions of science data packets mirrored from two 100G network connections at Fermi National Accelerator Laboratory. The detection performance was evaluated on traffic captured from the same research network days and weeks after the training with different types of attack flows injected. Results show that ThunderSecure can recognize science data traffic captured long after the training and made nearly certain detection on the segment of the streams where anomalous flows were injected.

100G research network↗

A streamlined approach to characterize microalgae strains for biomass productivity under dynamic climate simulation conditions

Screening microalgae strains under static light and temperature flask conditions cannot directly quantify the biomass productivities of algae growing in the dynamically fluctuating light and temperature conditions of outdoor ponds. In this effort, we describe a testing pipeline that screens for productivity under climate simulated conditions. A validated, miniaturized Laboratory Environmental Algae Pond Simulator (mini-LEAPS) photobioreactor was used to determine optimal medium salinities and biomass productivities of cold-tolerant microalgae collected from Arctic, subarctic, Antarctic, and subalpine habitats. Strains characterized in this effort include: Chlorella antarctica UTEX1959, Chlorella sp. UTEXSNO69, Chloromonas rosae UTEXSNO11, Phaeodactylum tricornutum UTEX646, and Stichococcus minutus CCALA727. Observed productivities of the characterized strains ranged from 1.74 ± 0.25 to 8.18 ± 0.81 g m –2 day –1 (as ash-free dry weight). For each strain, a temperature tolerance profile was generated to identify the most appropriate season (s) for cultivation. The two most promising strains, Chlorella sp. UTEX SNO69 and P. tricornutum UTEX646, were tested alongside the DISCOVR winter benchmark strain Monoraphidium minutum 26B-AM in outdoor open ponds at the PNNL Algae Testbed (PAT) in Arizona. Further, under cold-season outdoor pond conditions, Chlorella sp. UTEXSNO69 achieved average areal biomass productivities of 8.21 ± 0.50 g m –2 day –1 , not significantly different from that of the benchmark strain (8.52 ± 0.43 g m –2 day –1 , p > 0.05). Under spring outdoor pond conditions, P. tricornutum UTEX646 achieved average areal biomass productivities 10.34 ± 0.27 g m –2 day –1 , not significantly different from that of the benchmark (10.07 ± 0.50 g m –2 day –1 ) under the tested conditions (p > 0.05). The streamlined pipeline was thus demonstrated to successfully characterize productive microalgae strains for outdoor deployment by weeding out algae strains that were not productive under the tested dynamic light and temperature conditions.

59 BASIC BIOLOGICAL SCIENCES↗

A survey of the sorghum transcriptome using single-molecule long reads

Alternative splicing and alternative polyadenylation (APA) of pre-mRNAs greatly contribute to transcriptome diversity, coding capacity of a genome and gene regulatory mechanisms in eukaryotes. Second-generation sequencing technologies have been extensively used to analyse transcriptomes. However, a major limitation of short-read data is that it is difficult to accurately predict full-length splice isoforms. Here we sequenced the sorghum transcriptome using Pacific Biosciences single-molecule real-time long-read isoform sequencing and developed a pipeline called TAPIS (Transcriptome Analysis Pipeline for Isoform Sequencing) to identify full-length splice isoforms and APA sites. Our analysis reveals transcriptome-wide full-length isoforms at an unprecedented scale with over 11,000 novel splice isoforms. Additionally, we uncover APA ofB11,000 expressed genes and more than 2,100 novel genes. Lastly, these results greatly enhance sorghum gene annotations and aid in studying gene regulation in this important bioenergy crop. The TAPIS pipeline will serve as a useful tool to analyse Iso-Seq data from any organism.

60 APPLIED LIFE SCIENCES↗

Mathematical model of testing of pipeline integrity by thermal fields

Thermal fields testing at the ground surface above a pipeline are considered. One method to obtain and investigate an ideal thermal field in different environments is a direct numerical simulation of heat transfer processes taking into account the most important physical factors. In the paper a mathematical model of heat propagation from an underground source is described with accounting of physical factors such as filtration of water in soil and solar radiation. Thermal processes are considered in 3D origin where the heat source is a pipeline with constant temperature and non-uniform isolated shell (with 'damages'). This problem leads to solution of heat diffusivity equation with nonlinear boundary conditions. Approaches to analysis of thermal fields are considered to detect damages.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Global transcriptional regulatory network for Escherichia coli robustly connects gene expression to transcription factor activities

Significance While the transcriptional regulatory network (TRN) of Escherichia coli has expanded considerably in recent years through new chromatin immunoprecipitation (ChIP) data, an open question remains: Does the global TRN, reconstructed by combining ChIP data for individual transcriptions factors, consistently explain observed differential gene expression? We have reconstructed a high-confidence TRN, determined its consistency with transcriptomics and predictive capabilities across multiple conditions, extracted 10 functional regulatory modules, and characterized this network at the sequence and structural levels. Our multiomics algorithmic pipeline is expected to facilitate rigorous validation and prioritization of experiments to elucidate TRNs in other bacteria.

59 BASIC BIOLOGICAL SCIENCES↗

Rational design and testing of abiotic stress‐inducible synthetic promoters from poplar cis ‐regulatory elements

Summary Abiotic stress resistance traits may be especially crucial for sustainable production of bioenergy tree crops. Here, we show the performance of a set of rationally designed osmotic‐related and salt stress‐inducible synthetic promoters for use in hybrid poplar. De novo motif‐detecting algorithms yielded 30 water‐deficit (SD) and 34 salt stress (SS) candidate DNA motifs from relevant poplar transcriptomes. We selected three conserved water‐deficit stress motifs (SD18, SD13 and SD9) found in 16 co‐expressed gene promoters, and we discovered a well‐conserved motif for salt response (SS16). We characterized several native poplar stress‐inducible promoters to enable comparisons with our synthetic promoters. Fifteen synthetic promoters were designed using various SD and SS subdomains, in which heptameric repeats of five‐to‐eight subdomain bases were fused to a common core promoter downstream, which, in turn, drove a green fluorescent protein (GFP) gene for reporter assays. These 15 synthetic promoters were screened by transient expression assays in poplar leaf mesophyll protoplasts and agroinfiltrated Nicotiana benthamiana leaves under osmotic stress conditions. Twelve synthetic promoters were induced in transient expression assays with a GFP readout. Of these, five promoters (SD18‐1, SD9‐2, SS16‐1, SS16‐2 and SS16‐3) endowed higher inducibility under osmotic stress conditions than native promoters. These five synthetic promoters were stably transformed into Arabidopsis thaliana to study inducibility in whole plants. Herein, SD18‐1 and SD9‐2 were induced by water‐deficit stress, whereas SS16‐1, SS16‐2 and SS16‐3 were induced by salt stress. The synthetic biology design pipeline resulted in five synthetic promoters that outperformed endogenous promoters in transgenic plants.

09 BIOMASS FUELS↗

Nebraska Integrated Carbon Capture and Storage Pre-Feasibility Study

In collaboration with the Nebraska Public Power District (NPPD), the Energy & Environmental Research Center (EERC) has conducted a pre-feasibility study for a commercial-scale carbon dioxide (CO 2 ) geologic storage complex in western Nebraska integrated with potential CO 2 capture at Gerald Gentleman Station (GGS). GGS is the largest coal-fired electricity-generating station in Nebraska, emitting 8.5 million metric tons (Mt) of CO 2 annually, and is located near the town of Sutherland. This pre-feasibility (Phase 1) project has been executed as part of the U.S. Department of Energy (DOE) CarbonSAFE program, a multiphase initiative to support the deployment of large-scale carbon capture and storage (CCS) projects. Each CarbonSAFE project is required to demonstrate the potential to capture and store at least 50 million tonnes (Mt) of CO 2 over a 25-year operational period. The EERC and NPPD established a coordination team to identify challenges to a potential Nebraska CCS project, comprising local stakeholder organizations, which met twice in Lincoln and via several Webinars, providing feedback and guidance throughout the pre-feasibility study. The EERC also secured the technical support of Schlumberger Carbon Services and Computer Modelling Group Ltd. (CMG). The project comprised four technical themes, all using published information sources. Regional and stakeholder analysis, including identification of sensitive environmental areas, potential resource conflicts, and strategies for public outreach. A review of geographic and socioeconomic characteristics, in combination with geologic characterization in the subbasinal analysis described below, identified an area to the southwest and within a 75-mile radius of GGS as the most prospective for development of a storage site. This area largely avoided lands with protected status such as wetlands. A public outreach plan has also been developed for implementation in any further phases of CCS assessment in western Nebraska, for example a CarbonSAFE Phase 2 feasibility study. Scenario analysis, addressing economic and regulatory factors. GGS is the only single major source of CO 2 emissions capable of satisfying the CarbonSAFE 50-Mt scale requirement within the study region. Chemical absorption using amines was identified as the most viable technology for postcombustion CO 2 capture at GGS. The total cost of a CCS project at GGS was estimated to be between $67/tonne CO 2 for capture and auxiliary boiler to minimize parasitic load and $70/tonne CO 2 avoided cost, using the Carnegie-Mellon University Integrated Environmental Control Model (IECM). The total avoided cost included the capture facility and parasitic load, a flue gas desulfurization plant required for the use of amine solvent technology, transport via pipeline, and dedicated storage infrastructure. Nebraska has no legislation in place to address typical CCS-specific issues, for example pore space ownership for storage. Long-term liability, therefore, falls under the U.S. Environmental Protection Agency Underground Injection Control (UIC) Class 6 program regulations. Subbasinal analysis, addressing the potential for a dedicated subsurface “container” to store the required 50-Mt quantity of CO 2 . Modeling and simulation studies identified an area to the southwest of GGS with the potential for storage of 50 Mt CO 2 in the Cloverly Formation, comprising sandstones with interbedded and intermingled shales. The area of review (AOR) that would be required for monitoring under a Class VI operating permit was estimated to be as high as 400–700 square miles, due to uncertain pressure effects. The viability of this storage option is subject to significant uncertainty due to the relatively limited amount of existing characterization data available to the pre-feasibility study; for example, dynamic simulation indicated that the proposed storage rate might require as little as two or as many as 14 injection wells. A key uncertainty is the relative proportion and distribution of sandstone and shale within the Cloverly Formation. A preliminary, semiquantitative risk assessment also suggested uncertainty over storage capacity and injectivity constitute the most significant project risks at this pre-feasibility stage. No assessed risks were considered to rule out the possibility of a project moving to deployment. National Risk Assessment Partnership (NRAP) validation, using software tools developed by the National Energy Technology Laboratory (NETL) to assess risks associated with the potential 50-Mt CO 2 storage complex. NRAP tools were used to assess hypothetical leakage scenarios. Results broadly supported the conclusion of the semiquantitative risk analysis – for example, even worst-case analysis of theoretical leakage scenarios found limited migration rates and impacts. In summary, the work undertaken in this Phase 1 pre-feasibility study has shown that western Nebraska has potential to host a commercial-scale CCS project, including a dedicated storage “container” for 50 Mt of CO 2 . However, the following key challenges would need to be overcome: The business case for deploying CCS projects is uncertain; recently announced federal tax credits and sales of CO 2 for enhanced oil recovery may not cover the full costs of a CCS project at GGS, as estimated by this pre-feasibility study. The potential 50-Mt CO 2 dedicated storage container defined in this pre-feasibility study should be regarded as having a relatively low level of readiness to support a CCS project. Public outreach would be a vital element in western Nebraska, where sensitivities around such environmental issues as water resource protection and pipeline construction would need to be carefully addressed.

54 ENVIRONMENTAL SCIENCES↗

Completed UIC Permit Document (SECARB Work Product 3.2.4.b)

The Southeast Regional Carbon Sequestration Partnership’s (SECARB’s) Anthropogenic Test is a demonstration of the deployment of CO 2 capture, transport, geologic storage and monitoring technologies. The capture component of the test takes place at the James M. Barry Electric Generating Plant in Bucks, Alabama. The capture facility utilizes post-combustion amine capture technology licensed from Mitsubishi Heavy Industries America to process flue gas from a coal-fired power unit at Plant Barry. The amount of CO 2 captured using this process is equivalent to the emissions produced when generating 25 megawatts (MW) of electricity. CO 2 captured at the plant is transported by pipeline for underground storage in a saline geologic formation within the Citronelle Dome. This project is one of the first and the largest fully-integrated commercial prototype coal-fired carbon capture and storage projects in the USA.

54 ENVIRONMENTAL SCIENCES↗

The Effect of Flow on CO2 Corrosion of Self-Healing Metallic Coatings

Internal corrosion is an issue that affects natural gas pipelines, a significant part of the United States energy infrastructure. Over time, this corrosion has worn away longstanding pipelines due to the original construction materials used for the lines and impurities in the gas and liquid streams flowing through them. The main impurities in natural gas are H2O, CO2, H2S, and O2. One of the leading causes of corrosion is the CO2 dissolved in water, giving rise to carbonic acid formation, which can further dissolve the steel pipe. The National Energy Technology Laboratory (NETL) has been studying different solutions to this problem. One potential answer is a self-healing sacrificial metallic coating applied by a novel cold spray technique. Previous studies at NETL have shown that the zinc-rich self-healing coating can withstand corrosion through passivation and create a barrier to the diffusion of corrosive species. This study explored the impact of flow on corrosion with this coating and carbon steel when exposed to a saturated CO2 environment by simulating the pipeline flow profile in a small-scale lab setting. The samples were evaluated using multiple electrochemical techniques that found corrosion rates and were backed up with surface analysis to conclude the behavior of the corrosion mechanisms.

CO2 corrosion↗

Towards Interactive, Reproducible Analytics at Scale on HPC Systems

The growth in scientific data volumes has resulted in a need to scale up processing and analysis pipelines using High Performance Computing (HPC) systems. These workflows need interactive, reproducible analytics at scale. The Jupyter platform provides core capabilities for interactivity but was not designed for HPC systems. In this paper, we outline our efforts that bring together core technologies based on the Jupyter Platform to create interactive, reproducible analytics at scale on HPC systems. Our work is grounded in a real world science use case-applying geophysical simulations and inversions for imaging the subsurface. Our core platform addresses three key areas of the scientific analysis workflow-reproducibility, scalability, and interactivity. We describe our implemention of a system, using Binder, Science Capsule, and Dask software. We demonstrate the use of this software to run our use case and interactively visualize real-Time streams of HDF5 data.

containers↗

Opportunities in Hydropower and Pumped Storage Hydropower

Hydropower and pumped storage hydropower (PSH) are established technologies with a long-standing presence in the United States, but the industry remains dynamic with an active relicensing pipeline and growing interest in PSH to provide large-scale energy storage and grid services. NLR has developed a range of tools, data, and analysis that supports the evaluation of hydropower and PSH opportunities, complementing other DOE-supported efforts in this space. This presentation initiates a conversation around the future of hydropower and PSH and how it can inform the National Academy of Sciences investigation of possible DOE-supported regional energy-water technology pilots.

13 HYDRO ENERGY↗