Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Scientific Workflows”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 469 records · Page 26
All sourcesNASA NTRSDOE OSTI
NASA NTRS2008

Automated Production of Movies on a Cluster of Computers

A method of accelerating and facilitating production of video and film motion-picture products, and software and generic designs of computer hardware to implement the method, are undergoing development. The method provides for automation of most of the tedious and repetitive tasks involved in editing and otherwise processing raw digitized imagery into final motion-picture products. The method was conceived to satisfy requirements, in industrial and scientific testing, for rapid processing of multiple streams of simultaneously captured raw video imagery into documentation in the form of edited video imagery and video derived data products for technical review and analysis. In the production of such video technical documentation, unlike in production of motion-picture products for entertainment, (1) it is often necessary to produce multiple video derived data products, (2) there are usually no second chances to repeat acquisition of raw imagery, (3) it is often desired to produce final products within minutes rather than hours, days, or months, and (4) consistency and quality, rather than aesthetics, are the primary criteria for judging the products. In the present method, the workflow has both serial and parallel aspects: processing can begin before all the raw imagery has been acquired, each video stream can be subjected to different stages of processing simultaneously on different computers that may be grouped into one or more cluster(s), and the final product may consist of multiple video streams. Results of processing on different computers are shared, so that workers can collaborate effectively.

Nail, Jasper↗
NASA NTRS2012

Social Networking Adapted for Distributed Scientific Collaboration

Share is a social networking site with novel, specially designed feature sets to enable simultaneous remote collaboration and sharing of large data sets among scientists. The site will include not only the standard features found on popular consumer-oriented social networking sites such as Facebook and Myspace, but also a number of powerful tools to extend its functionality to a science collaboration site. A Virtual Observatory is a promising technology for making data accessible from various missions and instruments through a Web browser. Sci-Share augments services provided by Virtual Observatories by enabling distributed collaboration and sharing of downloaded and/or processed data among scientists. This will, in turn, increase science returns from NASA missions. Sci-Share also enables better utilization of NASA s high-performance computing resources by providing an easy and central mechanism to access and share large files on users space or those saved on mass storage. The most common means of remote scientific collaboration today remains the trio of e-mail for electronic communication, FTP for file sharing, and personalized Web sites for dissemination of papers and research results. Each of these tools has well-known limitations. Sci-Share transforms the social networking paradigm into a scientific collaboration environment by offering powerful tools for cooperative discourse and digital content sharing. Sci-Share differentiates itself by serving as an online repository for users digital content with the following unique features: a) Sharing of any file type, any size, from anywhere; b) Creation of projects and groups for controlled sharing; c) Module for sharing files on HPC (High Performance Computing) sites; d) Universal accessibility of staged files as embedded links on other sites (e.g. Facebook) and tools (e.g. e-mail); e) Drag-and-drop transfer of large files, replacing awkward e-mail attachments (and file size limitations); f) Enterprise-level data and messaging encryption; and g) Easy-to-use intuitive workflow.

Karimabadi, Homa↗
DOE OSTI2026

Navigating the Path to Autonomy: Real-World Lessons from an Air-Free Self-Driving Laboratory

While autonomous experimentation has promise to accelerate discovery in physcial sciences, the real-world integration of predictive models and experimentation is non-trivial. Here we describe the genesis of a self-driving laboratory (SDL) for air-sensitive chemistry at Argonne National Laboratory and demonstrate the experimental design considerations needed for high-throughput experiments before predictive models can lead to scientific discovery. Our SDL was designed to explore battery electrolyte stability. Our final SDL utilized plate readers in a glovebox with a nitrogen atmosphere to perform kinetic assays and screen hundreds of battery-relevant solvents. However, the roadmap to autonomy and airfree-friendly experimentation required the complex evaluation of several spectroscopic and chromatographic methods. The greatest experimental challenges were (a) developing long-term sampling methods that remained air-free; (b) accelerating kinetics to advance reactivity projections; and (c) ensuring labware compatibility with nonaqueous solvents used in battery chemistry. Our experiences highlight the practical gap between closed-loop aspirations and the realities of chemical discovery, offering lessons on the challenges of transferring every day laboratory workflows to autonomy. These results suggest a more realistic blueprint for autonomy in chemistry—one that balances thoughtful and realistic experimental formulation.

Robertson, Lily A.↗
NASA NTRS2018

NASA Earth eXchange (NEX) App Store

NASA Earth Exchange (NEX), and her public cloud version OpenNEX, have become platforms supporting scientific collaboration, knowledge sharing and research for the entire Earth science community. To date, a number of custom tools and capabilities have been integrated into the platforms. However, such integration has to undergo a case-by-case manual process thus lacks scalability. This timely project builds an App Store onto OpenNEX as a building block. Climate data analytics tools/programs can be easily uploaded, shared, organized, searched, and recommended like photos and videos on the YouTube. The foundation of our App Store is a provenance server, which not only records metadata but also execution history of climate data analytics apps including the input data and parameters, output data and products, who runs the app for which purpose, and how apps may be chained into workflows. Researchers can thus understand, reproduce, and repurpose existing apps and workflows. Machine learning approaches are applied to mine provenance to provide recommend-as-you-go services for Earth scientists, such as to recommend suitable apps and workflow snippets. A browser-based workflow tool is also provided for researchers to explore the provenance server and design value-added workflows. Scalability, sustainability, extensibility, usability, adaptability, security and privacy are considered in the App Store.

eXchange↗
DOE OSTI2026

Enabling the Broader Use of MOOSE for Nuclear Energy and Other Simulation

This Final Scientific and Technical Report summarizes work performed under the Phase IIA SBIR project “Enabling the Broader Use of MOOSE for Nuclear Energy and Other Simulation” (DE-SC0020906) from August 2023 through August 2025. The objective of the Phase IIA effort was to mature and harden capabilities developed during Phase II, with the goal of enabling practical interoperability between Coreform’s isogeometric analysis (IGA) technologies and the Multiphysics Object-Oriented Simulation Environment (MOOSE), while improving robustness, performance, and scalability for complex, nuclear-relevant geometries. Over the course of Phase IIA, the project established and validated an extraction-based interoperability pathway between Coreform tools and MOOSE. A combined mesh and matrix format was defined collaboratively with MOOSE developers and integrated into the solver, enabling standard MOOSE workflows to operate on data exported from Coreform’s IGA and Flex Representation Method (FRM) pipelines. Early demonstrations validated architectural compatibility using linear solid mechanics problems, while later efforts focused on benchmark testing and external use. By the end of the project period, engineers at BWXT were able to independently set up and execute a simulation using the Coreform–MOOSE workflow and provide direct feedback that informed further refinement. In parallel, substantial effort was devoted to improving the robustness of trimmed U-spline construction for complex CAD geometries. A growing test suite of nuclear-relevant models was compiled through collaboration with multiple stakeholders and used to drive extensive bug fixing and reliability improvements. These efforts resulted in improved robustness and performance, including the addition of fallback capabilities that enhance reliability when the underlying commercial CAD kernel fails. Performance-oriented work progressed later in the project, with the development and demonstration of methods to decompose complex geometries into structured subregions and updated data representations to support more efficient solver processing. Additionally, extensive enhancements to threadsafe parallel data structures and trimming operations established a foundation for scalable processing of large assemblies. Collaboration with Sandia National Laboratories on the SGM geometric modeling kernel advanced to a functioning interface test case, positioning the workflow for future kernel integration. Overall, the Phase IIA effort successfully transitioned the project from architectural proof-of-concept to externally exercised, solver-integrated capability, while clarifying remaining technical challenges related to standardization, performance optimization, and kernel integration.

42 ENGINEERING↗
DOE OSTI2026

Bayesian Optimization of Catalysis with In-Context Learning

Large language models (LLMs) can perform accurate classification with zero or few examples through in-context learning (ICL), allowing the model to observe query-relevant examples at inference time and eliminating the need for additional weight updates to generalize beyond its original training data. We extend this capability to regression with uncertainty estimation using frozen LLMs (e.g., GPT-4o, Gemini), enabling Bayesian optimization (BO) in natural language without explicit model training or feature engineering. We apply this to materials discovery by representing materials as synthesis and testing procedures for use in natural language prompts. This Bayesian, design-first approach prioritizes optimization toward target material properties before detailed characterization, in contrast to conventional experimental workflows that often emphasize characterization of suboptimal materials. On benchmarks like aqueous solubility and oxidative coupling of methane (OCM), BO-ICL matches or outperforms Gaussian processes. In live experiments on the reverse water–gas shift (RWGS) reaction, BO-ICL identifies multimetallic catalysts that approach equilibrium CO yield within 6 and 10 iterations from a pool of 3,700 and 360,000 candidates, respectively. Our method redefines materials representation and accelerates discovery, with broad applications across catalysis, materials science, and AI.

Calibration↗
DOE OSTI2025

HP-MDR: High-performance and Portable Data Refactoring and Progressive Retrieval with Advanced GPUs

Scientific applications produce vast amounts of data, posing grand challenges in the underlying data management and analytic tasks. Progressive compression is a promising way to address this problem, as it allows for on-demand data retrieval with significantly reduced data movement cost. However, most existing progressive methods are designed for CPUs, leaving a gap for them to unleash the power of today’s heterogeneous computing systems with GPUs.In this work, we propose HP-MDR, a high-performance and portable data refactoring and progressive retrieval framework for GPUs. Our contributions are four-fold: (1) We carefully optimize the bitplane encoding and lossless encoding, two key stages in progressive methods, to achieve high performance on GPUs; (2) We propose pipeline optimization and incorporate it with data refactoring and progressive retrieval workflows to further enhance the performance for large data process; (3) We leverage our framework to enable high-performance data retrieval with guaranteed error control for common Quantities of Interest; (4) We evaluate HP-MDR and compare it with state of the arts using five real-world datasets. Experimental results demonstrate that HP-MDR delivers an average 13.68 × and 6.31 × throughput in data refactoring and progressive retrieval tasks, respectively. It also leads to 11.22 × throughput for recomposing required data representations under Quantity-of-Interest error control and 6.04 × performance for the corresponding end-to-end data retrieval, when compared with state-of-the-art solutions.

Li, Yanliang [University of Oregon]↗
DOE OSTI2025

FY 2025 Multidimensional Data Correlation Platform: Unified Software Architecture for Advanced Materials and Manufacturing Technologies Data Management and Processing

The Advanced Materials and Manufacturing Technologies (AMMT) program continues to advance a data-driven approach to demonstrate the utility of additive manufacturing for fabricating components for nuclear applications. A key scientific goal is to leverage data to better understand manufacturing outcomes and thereby improve the performance, reliability, and lifespan of nuclear components. Ultimately, this effort supports the development of standards for certification and qualification of additively manufactured components, enabling broader industry adoption. In support of this objective, the AMMT program is building and deploying a data management platform to record, index, analyze, and make available the manufacturing data generated across the AMMT program. In FY 2023, the team conceptualized the architecture of the platform and, in FY 2024, deployed the first functional version at the Oak Ridge National Laboratory (ORNL) Manufacturing Demonstration Facility (MDF). In FY 2025, the platform was officially opened to all AMMT members. To enable this expansion, core modifications and enhancements were developed, including improvements to the user interface and workflows for data entry and retrieval. Most notably, robust security and access control mechanisms were implemented to protect data and manage information sharing. This effort featured a logging system, protected views, and controlled access mechanisms. This report documents these enhancements and the transition of the platform into program-wide use.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Advancing AI-Driven Analysis in X-ray Absorption Spectroscopy: Spectral Domain Mapping and Universal Models

In recent years, rapid progress has been made in developing artificial intelligence (AI) and machine learning (ML) methods for X-ray absorption spectroscopy (XAS) analysis. Compared to traditional XAS analysis methods, AI/ML approaches offer dramatic improvements in efficiency and help eliminate human bias. To advance this field, we advocate an AI-driven XAS analysis pipeline that features several interconnected key building blocks: benchmarks, workflows, databases, and AI/ML models. Specifically, we present two case studies for XAS ML. In the first study, we demonstrate the importance of reconciling the discrepancies between simulation and experiment using spectral domain mapping (SDM). Our ML model, which is trained solely on simulated spectra, predicts an incorrect oxidation state trend for Ti atoms in a combinatorial zinc titanate film. After transforming the experimental spectra into a simulation-like representation using SDM, the same model successfully recovers the correct oxidation state trend. In the second study, we explore the development of universal XAS ML models that are trained on the entire periodic table, which enables them to leverage common trends across elements. Looking ahead, we envision that an AI-driven pipeline can unlock the potential of real-time XAS analysis to accelerate scientific discovery.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Engineering Computational Practices (Rev. 1)

This manual will attempt to motivate the use of an automated build system for the purposes of computational science and engineering. As part of this motivation, the surrounding computational practices of version control, documen tation, compute environment management, and regression testing will also be addressed as applied to the practice of computational engineering. Specifically, this manual intends to motivate the adoption of these traditional software engineering practices for use in research and production engineering simulation projects. This manual is not the first such effort in the greater scientific computing community. In fact, the authors relied heavily on the lesson plans of the Software Carpentry, established to teach computing skills to researchers in 1998. As the intention for this manual is to lay out fundamental practices of engineering computing, it will not attempt to fully teach the underlying concepts and will instead reference the well designed lesson plans of the Software Carpentry. Where possible, this manual will explain to general computing practices and concepts and limit discussion of specific software implementations to examples or vehicles for practice in concrete application. The specific software taught by the Software Carpentry curriculum is an excellent starting point to learn the core concepts of computational engi neering. However, the authors have found that applications to engineering simulation and analysis require translation of these software development concepts into the language and workflows of computational engineers. Adopting these computational tools may require engineers to re-imagine their workflows in some combination of traditional engineer ing and software concepts. It has also been necessary to extend existing software build systems for engineering practices beyond the simple wrap ping of engineering software execution. Where necessary, examples of specific software and their method of extension to engineering simulations will be given, with reference to the User Manual for recommended practical use. Where this manual relies on specific implementation examples, it should be understood that the practicing engineer may find that different software is more amenable to their specific work. It is always the overall collection of computational practices is more important than any specific software implementation. The ability to recognize which concepts are implemented by a software package will make a practicing engineer agile to changing project needs, computing resources, numeric solvers, programming languages, and even available funding.

42 ENGINEERING↗
NASA NTRSDate not supplied

MBSE Applications for the MSR SRC Mars Ascent Vehicle

The objective of the NASA Mars Sample Return (MSR) Campaign is to collect samples from the surface of Mars and return them to Earth for scientific research. The Mars Ascent Vehicle (MAV) will be integrated into a larger Mars Sample Retrieval Lander (SRL) for transit to and storage on Mars. After all Martian samples have been collected and loaded into the MAV payload assembly, MAV will deliver the samples from the Martian surface to Mars orbit. A separate spacecraft, the Earth Return Orbiter (ERO) will retrieve the samples from Mars orbit and return them to Earth. To address common systems engineering challenges associated with using traditional systems engineering practices on complex projects, the MAV systems engineering team has explored implementation of Model-Based Systems Engineering (MBSE) tools and languages. This paper describes the current state of implementation and development of the MAV MBSE model with the Systems Modeling Language (SysML) within the scope of the MAV Systems Requirement Cycle (SRC) systems engineering workflow. The MAV MBSE model has been developed within Magic Draw – a SysML editor commonly used to implement MBSE. The MAV MBSE model has been used to develop mission phase functional flow diagrams for the Concept of Operations, decompose mission to vehicle subsystem functions, develop a functional decomposition, derive functional requirements, trace requirements up to customer-imposed requirements, trace requirements within MAV requirement space, identify requirements trace gaps, define and map the physical design space architecture, allocate requirements to subsystems, develop validation items, define assembly, integration, and test (AI&T) operations, and trace these items across driving goals to develop an integrated digital thread of systems engineering information used to drive design specifications, decision making, and ultimately design verification and validation. Findings and results associated with implementing MBSE in these ways, alongside traditional methods will be discussed.

MBSE↗
NASA NTRSDate not supplied

MBSE Applications for the MSR SRC Mars Ascent Vehicle

The objective of the NASA Mars Sample Return (MSR) Campaign is to collect samples from the surface of Mars and return them to Earth for scientific research. The Mars Ascent Vehicle (MAV) will be integrated into a larger Mars Sample Retrieval Lander (SRL) for transit to and storage on Mars. After all Martian samples have been collected and loaded into the MAV payload assembly, MAV will deliver the samples from the Martian surface to Mars orbit. A separate spacecraft, the Earth Return Orbiter (ERO) will retrieve the samples from Mars orbit and return them to Earth. To address common systems engineering challenges associated with using traditional systems engineering practices on complex projects, the MAV systems engineering team has explored implementation of Model-Based Systems Engineering (MBSE) tools and languages. This paper describes the current state of implementation and development of the MAV MBSE model with the Systems Modeling Language (SysML) within the scope of the MAV Systems Requirement Cycle (SRC) systems engineering workflow. The MAV MBSE model has been developed within Magic Draw – a SysML editor commonly used to implement MBSE. The MAV MBSE model has been used to develop mission phase functional flow diagrams for the Concept of Operations, decompose mission to vehicle subsystem functions, develop a functional decomposition, derive functional requirements, trace requirements up to customer-imposed requirements, trace requirements within MAV requirement space, identify requirements trace gaps, define and map the physical design space architecture, allocate requirements to subsystems, develop validation items, define assembly, integration, and test (AI&T) operations, and trace these items across driving goals to develop an integrated digital thread of systems engineering information used to drive design specifications, decision making, and ultimately design verification and validation. Findings and results associated with implementing MBSE in these ways, alongside traditional methods will be discussed.

MBSE↗
DOE OSTI2022

Celeritas: GPU-accelerated particle transport for detector simulation in High Energy Physics experiments

Within the next decade, experimental High Energy Physics (HEP) will enter a new era of scientific discovery through a set of targeted programs recommended by the Particle Physics Project Prioritization Panel (P5), including the upcoming High Luminosity Large Hadron Collider (LHC) HL-LHC upgrade and the Deep Underground Neutrino Experiment (DUNE). These efforts in the Energy and Intensity Frontiers will require an unprecedented amount of computational capacity on many fronts including Monte Carlo (MC) detector simulation. In order to alleviate this impending computational bottleneck, the Celeritas MC particle transport code is designed to leverage the new generation of heterogeneous computer architectures, including the exascale computing power of U.S. Department of Energy (DOE) Leadership Computing Facilities (LCFs), to model targeted HEP detector problems at the full fidelity of Geant4. This paper presents the planned roadmap for Celeritas, including its proposed code architecture, physics capabilities, and strategies for integrating it with existing and future experimental HEP computing workflows.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2026

Optimization of well design and CO 2 injection strategy for risk reduction in Class VI geological carbon sequestration wells

The safety and durability of Class VI wells are critical for geological carbon sequestration (GCS). However, current GCS operations face unique challenges: unlike traditional Class II wells, Class VI CO 2 injection wells operate at rates up to 100 times higher, dramatically increasing the risk of wellbore leakage and structural compromise due to severe temperature drops and associated mechanical stresses. Despite existing guidelines on material selection, there remains a substantial gap in understanding how rapid CO 2 injection rates, low surface temperatures, and variable reservoir conditions interact to threaten long-term well integrity. This study presents a comprehensive, original workflow integrating advanced analytical and numerical models for both well flow and well integrity analysis. By systematically simulating a wide range of field-relevant scenarios—including variations in injection rate, CO 2 temperature, and reservoir pressure—this work provides the first cross-validated assessment of cooling effects on wellbore. The results reveal that extreme temperature drops, up to 60 °C, can occur under high injection rates, particularly in depleted reservoirs, significantly increasing the risk of cement failure. Building on these insights, the study proposes innovative, practical well design and operational strategies, including ductile cement formulations, pre-stressing techniques, advanced insulation coatings, and proactive management of injection rates. The safety of Class VI well extends beyond simply using CO 2 resistant materials. Cement materials should possess optimal thermo-hydraulic-mechanical-chemical properties for effective performance. This work provides a scientific basis for optimizing Class VI well designs, with direct benefits for minimizing environmental risk, lowering operational costs, and enhancing the long-term reliability of GCS.

25 ENERGY STORAGE↗
DOE OSTI2024

Large-Scale Visualization of 3D Unstructured Groundwater Model Using Cave Automated Virtual Environment

The immersive three-dimensional (3D) virtual reality (VR) visualization of groundwater models allows us to deepen our understanding of aquifer systems and provide better solutions to present groundwater-related problems, such as groundwater recharge, water quality, and sustainability. Visualization assists in accurately developing groundwater models and revealing important subsurface features, including faulting, folding, and unconformity. However, assessing model accuracy poses challenges due to the complexity of geology and groundwater systems. This research demonstrates a workflow to visualize and analyze raw 3D unstructured groundwater model data using an immersive Cave Automated Virtual Environment (CAVE). To visualize the unstructured groundwater model data, the raw dataset is converted into interactive CAVE-compatible formats utilizing a set of tools: ParaView, Blender, and Unity. This enables researchers to immerse themselves in the data, identifying influential patterns and relationships. e resulting insights can inform the development of sophisticated machine-learning models for groundwater level prediction. The CAVE’s immersive capabilities allow intuitive exploration from various perspectives, providing a more holistic understanding of the factors affecting groundwater levels. These insights are crucial to improve predictive models. The CAVE results also facilitate collaborative analysis and have potential applications in training and education. is research demonstrates the value of immersive VR tools such as the CAVE for unraveling intricacies within high-dimensional scientific data to drive real-world forecasting and modeling applications.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

Integration of the Biot–Gassmann Fluid Substitution Method and Machine Learning-Based Velocity–Stress Relationship for Estimating In Situ Stresses

Recent advancements have shown that in situ stresses can be reliably estimated through an integrated machine/deep learning (ML/DL)-based framework, which relies on models trained and validated using true triaxial ultrasonic velocity (TUV) experimental data that involve measurements of ultrasonic velocity in saturated rocks under varying stress configurations. However, when the goal is to interpret lower frequency measurements, it may be more appropriate to run experiments on dry rocks and then obtain Biot–Gassmann-derived equivalent saturated velocities (low-frequency approximation) and employ these quantities for training ML/DL models to predict in situ stress. Whether the dispersion effect of frequency on the velocity–stress relationship substantially impacts in situ stress prediction is an important and unresolved question. This work presents an enhancement of ML/DL-based workflow by training and implementing ML/DL models using equivalent saturated acoustic velocities (low-frequency) obtained by applying Biot–Gassmann fluid substitution on the ultrasonic velocities of dry cores. The models were trained on TUV data sets derived from three subsurface cores extracted from the geothermal well 16B(78)-32 at the Utah FORGE site. Each core was subjected to 75 unique stress configurations for velocity measurement in the dry state. The ML/DL trained on the TUV data set with equivalent saturated velocities demonstrated promising performance to predict in situ stress in subsurface geological rocks using velocity–stress relationships with R 2 of 0.86, 0.971, and 0.975 and root mean squared error (RMSE) of 2.59, 1.92, and 1.80 for validation/testing phases of vertical, minimum horizontal, and maximum horizontal stress models, respectively. Additionally, interpretation and explanation by Shapley additive explanations (SHAP) analysis further improved scientific validation and model reliability for estimating in situ stresses.

colloids↗
DOE OSTI2023

Conquering Data Chaos: Research Data Management with Kubernetes

Managing massive volumes of data and effectively making it accessible to researchers poses significant challenges and is a barrier to scientific discovery. In many cases, critical data is locked up in unwieldy file formats or one-off databases and is too large to effectively process on a single machine. This talk explores the role of Kubernetes, an open-source container orchestration platform, in addressing research data management challenges. I will discuss how we are using a set of publicly available open-source and home-grown tools in the National Renewable Energy Lab (NREL) Data, Analysis, and Visualization (DAV) group to help researchers overcome data-related bottlenecks. The talk will begin by providing an overview of the data challenges faced in research data management, including data storage, processing, and analysis. I will highlight Kubernetes' ability to handle large-scale data by leveraging containerization and distributed computing, including distributed storage. Kubernetes allows researchers to encapsulate data processing infrastructure and workflows into portable containers, enabling reproducibility and ease of deployment. Kubernetes can then schedule and manage the resource allocation of these containers to enable efficient utilization of limited computing resources, leading to more efficient data processing and analysis. I will discuss some limitations of traditional, siloed approaches to dealing with data and emphasize the need for solutions which foster collaboration. I will highlight how we are using Kubernetes at NREL to facilitate data sharing and cooperation among research teams. Kubernetes' flexible architecture enables the deployment of shared computing environments, such as Apache Superset, where researchers can seamlessly access and analyze shared datasets. Providing the ability to have one research team easily consume data generated by another, utilizing Kubernetes' as a central data platform, is one of the major wins we've encountered by adopting the platform. Finally, I will showcase real-world use cases from NREL where we have used Kubernetes to solve some persistent data challenges involving large volumes of sensor and monitoring data. I will discuss the challenges we encountered when creating our cluster and making it available as a production-ready resource. I will also discuss the specific suite of tools, including Postgres and Apache Druid for columnar and timeseries data, and Redpanda Kafka for streaming data we have deployed in our infrastructure, and the process that went into the selection of these tools.

collaborative environment↗
NASA NTRS2013

Hadoop for High-Performance Climate Analytics: Use Cases and Lessons Learned

Scientific data services are a critical aspect of the NASA Center for Climate Simulations mission (NCCS). Hadoop, via MapReduce, provides an approach to high-performance analytics that is proving to be useful to data intensive problems in climate research. It offers an analysis paradigm that uses clusters of computers and combines distributed storage of large data sets with parallel computation. The NCCS is particularly interested in the potential of Hadoop to speed up basic operations common to a wide range of analyses. In order to evaluate this potential, we prototyped a series of canonical MapReduce operations over a test suite of observational and climate simulation datasets. The initial focus was on averaging operations over arbitrary spatial and temporal extents within Modern Era Retrospective- Analysis for Research and Applications (MERRA) data. After preliminary results suggested that this approach improves efficiencies within data intensive analytic workflows, we invested in building a cyber infrastructure resource for developing a new generation of climate data analysis capabilities using Hadoop. This resource is focused on reducing the time spent in the preparation of reanalysis data used in data-model inter-comparison, a long sought goal of the climate community. This paper summarizes the related use cases and lessons learned.

analytics↗