Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Distribution”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 631 records · Page 35
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Snow distribution patterns revisited: A physics-based and machine learning hybrid approach to snow distribution mapping in the sub-Arctic Supporting Data

Snow in the Arctic and sub-Arctic is highly variable at fine scales, with deep drifts and shallow scoured areas creating a complex pattern of snow on the landscape. This fine-scale variation in snow is driven primarily by landscape and vegetation properties. Some landscape features, such as river beds, will rapidly fill in with snow during the wintertime due to high winds, while shrubs will trap blowing snow, resulting in drifts. Meanwhile, snow will blow off of exposed areas, resulting in abnormally shallow snow. These complex interactions between wind, vegetation, and terrain are difficult to represent well with physically-based models, but machine learning techniques have shown promise in the past. Here, we propose a hybrid modeling approach, where we use machine learning derived snow pattern maps to inform SnowModel, a physically-based snow process model. We develop and test this technique at the Teller 27 Seward Peninsula NGEE-Arctic study site. This dataset includes 5 *.nc files of model inputs and outputs plus one user guide (*.pdf). We present the data we use to drive SnowModel (vegetation type, digital elevation model), the snow pattern maps used to inform SnowModel (Standardized Depth Values maps, Machine Learning Snow Distribution Pattern), and our machine learning, SnowModel, and Hybrid snow depth and snow water equivalent results. The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research. The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska. Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Data efficiency and extrapolation trends in neural network interatomic potentials

Abstract Recently, key architectural advances have been proposed for neural network interatomic potentials (NNIPs), such as incorporating message-passing networks, equivariance, or many-body expansion terms. Although modern NNIP models exhibit small differences in test accuracy, this metric is still considered the main target when developing new NNIP architectures. In this work, we show how architectural and optimization choices influence the generalization of NNIPs, revealing trends in molecular dynamics (MD) stability, data efficiency, and loss landscapes. Using the 3BPA dataset, we uncover trends in NNIP errors and robustness to noise, showing these metrics are insufficient to predict MD stability in the high-accuracy regime. With a large-scale study on NequIP, MACE, and their optimizers, we show that our metric of loss entropy predicts out-of-distribution error and data efficiency despite being computed only on the training set. This work provides a deep learning justification for probing extrapolation and can inform the development of next-generation NNIPs.

36 MATERIALS SCIENCE↗
DOE OSTI2022

Electric Vehicles at Scale - Phase II - Distribution Systems Analysis

The use of electric vehicles (EVs) in the United States has grown significantly during the last decade, posing benefits to the environment but also potential challenges to our grid. To better understand this scenario , the Department of Energy (DOE) asked Pacific Northwest National Laboratory (PNNL) to perform an authoritative study of the impacts of EVs at scale on the electric grid. The Phase I study focused on the bulk power electricity impacts . This EV-at-scale Phase II work addresses key questions of interest to DOE related to the impacts of EV in the distribution systems: 1. When (which year), where, and how many EVs will be adopted and how will they be charged? 2. Given some prospective answers to the above question, what would be the EV hosting capability of a distribution system circuit? 3. How could the hosting capability be expanded to accommodate more EVs and what would be the potential measures and cost? This report documents the methodologies developed as part of this project and provides EV adoption modeling examples using distribution system circuit data provided by industry partner, Southern California Edison (SCE).

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2026

Conditional distribution estimation of building characteristics with diffusion models for urban energy modeling

Understanding current energy consumption behavior in communities is critical for informing future energy use decisions and enabling efficient energy management. Urban energy models, which are used to simulate these energy use patterns, require large datasets with detailed building characteristics for accurate outcomes. However, such detailed characteristics at the individual building level are often unknown and costly to acquire, or unavailable. Through this work, we propose using a generative modeling approach to generate realistic building attributes to fill in the data gaps and finally provide complete characteristics as inputs to energy models. Our model learns complex, building-level patterns from training on a large-scale residential building stock model containing 2.2 million buildings. We employ a tabular diffusion-based framework that is designed to handle heterogeneous (discrete and continuous) features in tabular building data, such as occupancy, floor area, heating, cooling, and other equipment details. We develop a capability for conditional diffusion, enabling the imputation of missing building characteristics conditioned on known attributes. We conduct a comprehensive validation of our conditional diffusion model, firstly by comparing the generated conditional distributions against the underlying data distribution, and secondly, by performing a case study for a Baltimore residential region, showing the practical utility of our approach. Our work is one of the first to demonstrate the potential of generative modeling to accelerate building energy modeling workflows.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2022

Sizing Considerations for EV Dynamic Wireless Charging Systems with Integrated Energy Storage

Roadways with dynamic wireless charging systems (DWCS) enable charge-sustaining in-motion EV charging, which can reduce charging idle time while increasing range capabilities. Spatially distributed transmitter coils are controlled in response to traffic load that varies significantly minute to minute with high power levels, very short charging time, and low system utilization like wind turbine power. Traffic load estimation and localized analysis may guide effective sizing and topology adoption for feasible and scalable DWCS deployment. DWCS traffic load approximation is reviewed with measured Automated Traffic Recorder (ATR) data and statistical distributions being used to create a synthetic load analyzed using proposed metrics quantifying system utilization over time. Lumped coil section segmentation is compared between second-based distance and spatial density analysis methods, offering 17-27% greater system utilization. A peak load shifting method is proposed for traffic redirection across two tracks with optional BESS integration increasing system utilization by 50-60% depending on time-based and power reserve-based sizing and control.

Lewis, Donovin D.↗
DOE OSTI2021

Event Cause Analysis in Distribution Networks using Synchro Waveform Measurements

This paper presents a machine learning method for event cause analysis to enhance situational awareness in distribution networks. The data streams are captured using time-synchronized high sampling rates synchro waveform measurement units (SWMU). The proposed method is formulated based on a machine learning method, the convolutional neural network (CNN). This method is capable of capturing the spatiotemporal feature of the measurements effectively and perform the event cause analysis. Several events are considered in this paper to encompass a range of possible events in real distribution networks, including capacitor bank switching, transformer energization, fault, and high impedance fault (HIF). The dataset for our study is generated using the real time digital simulator (RTDS) to simulate real-world events. The event cause analysis is performed using only one cycle of the voltage waveforms after the event is detected. The simulation results show the effectiveness of the proposed machine learning-based method compared to the state-of-the-art classifiers.

Niazazari, Iman↗
DOE OSTI2021

HFTS-2 Completions Design and State-of-the-Art Diagnostics Results

Various completion schemes were tested as part of the Hydraulic Fracture Test Site-2 (HFTS-2) project. The wells were instrumented with permanent fiber optic (FO) cable, and unique near-wellbore and far-field region surveys were acquired to evaluate spatial and temporal hydraulic fracture characteristics. The state-of-the-art diagnostics included substantial monitoring of both the well stimulation and longer-term production performance. The objective was to evaluate and improve stimulation distribution effectiveness (SDE) for different completion designs based on the diagnostics results. A hydraulic fracturing (HF) profile (i.e., proppant/fluid allocation among clusters) was obtained using near-wellbore Distributed Acoustic Sensing (DAS) data acquired during stimulation. Statistical analysis provided a quantitative assessment of SDE for the different completion designs. A novel workflow was developed to assess time-lapse fracture property changes at the cluster and stage levels based on Distributed Strain Sensing (DSS) data acquired during the production phase. These time-lapse fracture property changes were compared with the HF profile. Based on the statistical analysis, Rate Per Cluster (RPC) during proppant slurry placement appears to be a primary completion design variable, rather than the typical Rate Per Perf (RPP). If RPC was not maintained at a certain level, the likelihood of uneven distribution increased dramatically. The stage configurations that created low RPC while maintaining high RPP did not show improved SDE. The strain change results based on DSS data during the production phase indicated near-wellbore fracture property changes (e.g., aperture changes during well shut-in and flow periods) and aligned well with near-wellbore DAS signals acquired during stimulation. More importantly, fracture property changes based on DSS data acquired during two tests over a 7-month period show high consistency and good alignment with the near-wellbore HF profile. This workflow quantitatively evaluates SDE for different completion designs using diagnostic results and could help optimize completion strategy for future development projects. Here, the analysis and interpretation of results demonstrate the technical feasibility of using permanent fiber optic cable for stimulation and production monitoring and indic

58 GEOSCIENCES↗
DOE OSTI2023

A phased workflow to define permit‐ready locations for large volume CO 2 injection and storage

Abstract To‐date, only two UIC Class VI permits have been issued by the US Environmental Protection Agency. We illustrate a four‐phase workflow to first identify regional storage resources and then down‐select sites to yield permit‐ready locations that can accept and store large volumes of CO 2 . Specific permit requirements should guide objectives and define deliverables of respective workflow phases. In the first phase we used available regional data and screened structure and injection zones to locate resources that match CO 2 volumes planned to be captured. Available data were also used to assess presence and depth of usable groundwater, the key resource being protected via permitting. We then used advanced, closed‐form, analytical solutions (EASiTool) to estimate CO 2 injectivity into each hydrologically connected injection compartment. In the second phase we acquired and conditioned additional wireline logs and leased available seismic datasets. We interpreted the depositional systems from wireline well‐log character and mapped sandbody geometry to interpolate injection and confining‐zone distribution. Using available data, we mapped faults and locations of freshwater and overpressure (or other capacity‐limiting geologic parameters) in more detail. In the third phase, we used the augmented geologic data to develop a static model for the selected area, extracted the areas of highest interest, and generated and ran dynamic (flow) models. In a fourth phase, we reduced major uncertainties identified in earlier phases. Our case study indicates that to complete preparation of a permit application requires (1) improved lithologic characterization information (thicknesses and horizontal and vertical connectivity) and (2) better definition of poorly defined local faults. © 2023 The Authors. Greenhouse Gases: Science and Technology published by Society of Chemical Industry and John Wiley & Sons Ltd.

58 GEOSCIENCES↗
DOE OSTI2020

Cyber‐physical system testbed for power system monitoring and wide‐area control verification

The electric power system is intrinsically a cyber‐physical system (CPS) with power flowing inthe physical system and information flowing in the cyber‐network. Testbeds arecrucial for understanding the cyber‐physical interactions and provideenvironments for prototyping novel applications. This study proposes afour‐layer architecture for CPS testbeds with emphases on communication networkemulation and networked physical components. A configurable software‐definednetwork is employed to bridge physical components with wide‐area applicationsfor closed‐loop control. In order to distribute physically coupled devices intomultiple software simulations, this study proposes a data broker setup based ona distributed messaging environment to achieve low‐latency data streaming. Thedecoupled design with data streaming allows for building testbed components asmodules and running them in a distributed manner. Case studies verify the databroker setup for low‐latency sensing and actuation, as well as the communicationemulation setup for the desired network latency. Also illustrated is a replayattack scenario using synchrophasors in the Western Electricity CoordinatingCouncil (WECC) 181‐bus system for demonstrating the closed‐loop cyber‐physicalsimulation capability of the testbed.

Cui, Hantao↗
DOE OSTI2023

FPGA Acceleration of GCN in Light of the Symmetry of Graph Adjacency Matrix

Graph Convolutional Neural Networks (GCNs) are widely used to process large-scale graph data. Different from deep neural networks (DNNs), GCNs are sparse, irregular, and unstructured, posing unique challenges to hardware acceleration with regular processing elements (PEs). In particular, the adjacency matrix of a GCN is extremely sparse, leading to frequent but irregular memory access, low spatial/temporal data locality and poor data reuse. Furthermore, a realistic graph usually consists of unstructured data (e.g., unbalanced distributions), creating significantly different processing times and imbalanced workload for each node in GCN acceleration. To overcome these challenges, we propose an end-to-end hardware-software co-design to accelerate GCNs on resource-constrained FPGAs with the features including: (1) A custom dataflow that leverages symmetry along the diagonal of the adjacency matrix to accelerate feature aggregation for undirected graphs. We utilize either the upper or the lower triangular matrix of the adjacency matrix to perform aggregation in GCN to improve data reuse. (2) Unified compute cores for both aggregation and transform phases, with full support to the symmetry-based dataflow. These cores can be dynamically reconfigured to the systolic mode for transformation or as individual accumulators for aggregation in GCN processing. (3) Preprocessing of the graph in software to rearrange the edges and features to match the custom dataflow. This step improves the regularity in memory access and data reuse in the aggregation phase. Moreover, we quantize the GCN precision from FP32 to INT8 to reduce the memory footprint without losing the inference accuracy. We implement our accelerator design in Intel Stratix10 MX FPGA board with HBM2, and demonstrate 1.3x-110.5x improvement in end-to-end GCN latency as compared to the state-of the-art FPGA implementations, on the graph datasets of Cora, Pubmed, Citeseer and Reddit.

Nair, Gopikrishnan R.↗
DOE OSTI2025

Deconvoluting thermomechanical effects in X-ray diffraction data using machine learning

X-ray diffraction is ideal for probing the sub-surface state during complex or rapid thermomechanical loading of crystalline materials. However, challenges arise as the size of diffraction volumes increases due to spatial broadening and because of the inability to deconvolute the effects of different lattice deformation mechanisms. Here, we present a novel approach that uses combinations of physics-based modeling and machine learning to deconvolve thermal and mechanical elastic strains for diffraction data analysis. The method builds on a previous effort to extract thermal strain distribution information from diffraction data. The new approach is applied to extract the evolution of the thermomechanical state during laser melting of an Inconel 625 wall specimen which produces significant residual stress upon cooling. A combination of heat transfer and fluid flow, elasto-plasticity and X-ray diffraction simulations is used to generate training data for machine-learning (Gaussian process regression, GPR) models that map diffracted intensity distributions to underlying thermomechanical strain fields. First-principles density functional theory is used to determine accurate temperature-dependent thermal expansion and elastic stiffness used for elasto-plasticity modeling. The trained GPR models are found to be capable of deconvoluting the effects of thermal and mechanical strains, in addition to providing information about underlying strain distributions, even from complex diffraction patterns with irregularly shaped peaks.

36 MATERIALS SCIENCE↗
DOE OSTI2021

Ab Initio Molecular Dynamics Simulations of Amorphous Calcium Carbonate: Interpretation of Pair Distribution Function and X-Ray Absorption Spectroscopy Data

The structure and transformation of hydrous amorphous calcium carbonate (ACC) are key to understanding biomineralization pathways and their relationship with the properties of the resulting material. Quantitative interpretation of scattering experiments aimed at elucidating the structure of ACC is challenging, due to the amorphous nature of this material, and, therefore, requires models for the structure and the scattering physics. Here, we generate physically realistic ensembles of hydrated ACC structures and their vibrational disorder from ab initio molecular dynamics (AIMD) simulations with an emphasis on enabling the consistent interpretation of the finer details of three complementary structural probes: neutron and x-ray pair distribution experiments and x-ray absorption spectroscopy (XAS). In each case, we simulate the signal directly in reciprocal space and then manipulate it into the real-space pair distribution function (PDF) or spectrum using the same procedures for the experimental and theoretical data. Good agreement with experiment was obtained across the three techniques with the simulations accounting well for all features in the spectra. Remaining small discrepancies pointed to differences between real samples and the idealized simulated systems such as deviations from the nominal CaCO3·nH2O stoichiometry. Additionally, the simulations offered a more accurate description of the local coordination environment of calcium than previous shell-by-shell fits to spectra of synthetic ACC and classical molecular dynamics simulations. This work demonstrates that AIMD is a powerful approach for extracting detailed structural information from neutron PDF, x-ray PDF, and XAS of amorphous carbonate phases.

Prange, Micah P.↗
DOE OSTI2025

Knowledge Oriented Graph Unified Transformer (KOGUT) v0.1

KOGUT — Knowledge Oriented Graph Unified Transformer KOGUT implements the Relational Graph Transformer (RelGT) architecture for knowledge graph link prediction in biological domains, with a primary focus on microbial growth media prediction. While the original RelGT (arXiv:2505.10960) targets relational tables, time series, and multi-table databases, KOGUT adapts this architecture for heterogeneous biological knowledge graphs, providing first-in-class AI predictive models for microbial cultivation. Key Adaptations Beyond Original RelGT: - Knowledge Graph Focus: Applied to biological KGs with semantic node types (taxa, chemicals, media, phenotypes, environments) versus generic relational database tables, trained on the KG-Microbe knowledge graph (1.3M entities, 2.9M edges, 24 relation types). - Multimodal Node Encoding: Integrates node labels, categories, descriptions, and synonyms from KG metadata through learned embedding layers—adapting relational column features to graph node attributes with textual semantics. - Extended K-Hop Subgraph Strategy: Optimized neighborhood sampling (3-hop default, configurable up to 200 nodes) tuned for sparse biological networks, building on the original local-global attention framework with biological relation preservation. - Biolink Predicate Preservation: Type-specific transformations for 24 biological edge semantics (occurs_in, consumes, produces, has_phenotype, subclass_of) beyond standard relational foreign keys, enabling multi-relation link prediction. - Inductive Learning Support: Enables zero-shot predictions for novel taxa through feature-based embeddings (temperature, oxygen requirements, gram stain, cell shape), extending the original transductive relational benchmark scope to uncultured microorganisms. CheapSOTA Performance Optimizations (This Distribution): - VQ-EMA Centroid Attention: Vector quantization with exponential moving average for improved global context modeling (+5-10% MRR improvement). - HDF5 Precomputed Data Loading: One-time preprocessing of k-hop subgraphs to eliminate redundant graph traversals (2-5× training speedup). - Distributed Data Parallel Training: Multi-GPU support for scaling to larger knowledge graphs (tested on 4× NVIDIA A100 GPUs at NERSC Perlmutter). - Mixed Precision Training: Automatic mixed precision (AMP) for memory efficiency and faster training. Advantages Over Standard Knowledge Graph Embedding Models: Combines RelGT's proven multi-element tokenization (features, type, hop, structure) with graph-native biological representations, enabling interpretable link prediction across heterogeneous entities that standard embedding models (TransE, RotatE, ComplEx) and table-based transformers cannot directly model. Achieves near-perfect performance on microbial growth media prediction (MRR: 0.9966, Precision@1: 0.9932, Hit@10: 1.0000) while maintaining explainability through attention-based reasoning over biological pathways. Training Data: - KG-Microbe merged knowledge graph: 1,379,337 nodes, 2,960,472 edges - 24 biological relation types including taxonomic hierarchies, metabolic interactions, phenotype associations, and environmental relationships - Primary prediction task: Growth media suitability for microbial taxa (biolink:occurs_in, 50K edges) - Multi-relation capability: Predicts links for any of the 24 relation types, including chemical consumption/production, phenotype associations, and taxonomic classification Citation: Original RelGT Architecture: Dwivedi et al., "Relational Graph Transformer", arXiv:2505.10960, 2025 KOGUT Implementation: Knowledge Oriented Graph Unified Transformer for Microbial Growth Media Prediction Developed at Lawrence Berkeley National Laboratory (LBNL) Trained on NERSC Perlmutter supercomputer

Joachimiak, Marcin [Lawrence Berkeley National Lab↗
DOE OSTI2024

Modeling Plutonium Decorporation in a Female Nuclear Worker Treated with Ca-DTPA after Inhalation Intake

The present work models plutonium (Pu) biokinetics in a female former nuclear worker. Her bioassay measurements are available at the US Transuranium and Uranium Registries. The worker was internally exposed to a plutonium-americium mixture via acute inhalation at a nuclear weapons facility. She was medically treated with injections of 1 g Ca-DTPA on days 0, 5, and 14 after the intake. Between days 0 and 20, fecal and urine samples were collected and analyzed for 239 Pu and 241 Am. Subsequently, she was followed up for bioassay monitoring over 14 y, with additional post-treatment urine samples collected and analyzed for 239 Pu. The uniqueness of this dataset is due to the availability of: (1) both early and long-term bioassay data from a female with plutonium intake; (2) data on chelation therapy for a female; and (3) fecal measurement results. Chelation therapy with Ca- and/or Zn-salts of DTPA is known to aid in reducing the internal radiation dose by enhancing the excretion of plutonium and americium from the body. Such enhancement affects plutonium biokinetics in the human body, posing a challenge to the internal dose assessment. The current radiation dose assessment practice is to exclude the data affected by Ca-DTPA from the analysis. The present analysis is the first to explicitly model the chelation-affected bioassay data in a female by using a newly developed chelation model. Thus, the bioassay data collected during and after the Ca-DTPA administrations were used for biokinetic modeling and dose assessment. The Markov Chain Monte Carlo method was used to investigate model parameter uncertainty, based on the bioassay data and assumed prior probability distributions. A χ 2 /nData (number of data points) ≈ 1 was observed in this study, which indicates self-consistency of the data with the model. Results of this study show that the worker’s 239 Pu intake was 12 Bq, with a committed effective dose to the whole-body of 1.2 mSv and a committed equivalent dose to the bone surfaces, liver, and lungs of 37.8, 9.1, and 0.8 mSv, respectively. This study also discusses the worker’s dose reduction due to chelation treatment.

61 RADIATION PROTECTION AND DOSIMETRY↗
DOE OSTI2025

Fast Response Temperature by airborne measurements over BNF

The original data were collected during the AAF Engineering Flights (AEF2025) in the vicinity of the ARM Bankhead National Forest (BNF) Atmospheric Observatory (https://www.arm.gov/capabilities/observatories/bnf ) in northwestern Alabama in March 2025. The ARM Aerial Facility ArcticShark uncrewed aerial system (UAS, https://www.arm.gov/capabilities/observatories/aaf/uas) was based at the public-use airport of Posey Field, Alabama (FAA LID: 1M4, 34.28027778° N, 87.60055556° W, 283m MSL) from March 9 through March 24, 2025. The ArcticShark UAS performed nine flights, including eight research flights over the AMF3 (BNF Main Site) and Supplemental Facilities to measure atmospheric state, turbulence, surface IR temperature and imagery, and aerosol number concentration and size distribution. The current data set presents fast response temperature in the atmospheric boundary layer and lower free troposphere measured on the airborne platform throughout the field campaign. The primary instruments used to create the current data set were the fine wire thermocouple probe, the Aircraft Integrated Meteorological Measurement System (AIMMS-30), the Pitot-static system (part of UAS flight control), and the infrared gas analyzer sensor for H2O and CO2 (LI-840). All parameters used in temperature calculations (static pressure, True Air Speed, and absolute humidity in form of dew point temperature) were included in the data set. For user convenience, one additional parameter was also included: the type of flight flag (level, up, down, turn, and combination of thereof).

Air temperature, fast response↗
DOE OSTI2025

Turbulent Parameters by airborne measurements over BNF in March 2025

The original data were collected during the AAF Engineering Flights (AEF2025) in the vicinity of the ARM Bankhead National Forest (BNF) Atmospheric Observatory (https://www.arm.gov/capabilities/observatories/bnf ) in northwestern Alabama in March 2025. The ARM Aerial Facility ArcticShark uncrewed aerial system (UAS, https://www.arm.gov/capabilities/observatories/aaf/uas) was based at the public-use airport of Posey Field, Alabama (FAA LID: 1M4, 34.28027778° N, 87.60055556° W, 283m MSL) from March 10 through March 24, 2025. The ArcticShark UAS performed nine flights, including eight research flights over the AMF3 (BNF Main Site) and Supplemental Facilities to measure atmospheric state, turbulence, surface IR temperature and imagery, and aerosol number concentration and size distribution. The current data set presents a collection of turbulent parameters in the atmospheric boundary layer or lower free troposphere based on airborne measurement throughout the field campaign. The primary instruments used to create the current data set were the Aircraft Integrated Meteorological Measurement System (AIMMS-30) and the fine-wire thermocouple probe.

Atmosphere↗
DOE OSTI2023

Conquering Data Chaos: Research Data Management with Kubernetes

Managing massive volumes of data and effectively making it accessible to researchers poses significant challenges and is a barrier to scientific discovery. In many cases, critical data is locked up in unwieldy file formats or one-off databases and is too large to effectively process on a single machine. This talk explores the role of Kubernetes, an open-source container orchestration platform, in addressing research data management challenges. I will discuss how we are using a set of publicly available open-source and home-grown tools in the National Renewable Energy Lab (NREL) Data, Analysis, and Visualization (DAV) group to help researchers overcome data-related bottlenecks. The talk will begin by providing an overview of the data challenges faced in research data management, including data storage, processing, and analysis. I will highlight Kubernetes' ability to handle large-scale data by leveraging containerization and distributed computing, including distributed storage. Kubernetes allows researchers to encapsulate data processing infrastructure and workflows into portable containers, enabling reproducibility and ease of deployment. Kubernetes can then schedule and manage the resource allocation of these containers to enable efficient utilization of limited computing resources, leading to more efficient data processing and analysis. I will discuss some limitations of traditional, siloed approaches to dealing with data and emphasize the need for solutions which foster collaboration. I will highlight how we are using Kubernetes at NREL to facilitate data sharing and cooperation among research teams. Kubernetes' flexible architecture enables the deployment of shared computing environments, such as Apache Superset, where researchers can seamlessly access and analyze shared datasets. Providing the ability to have one research team easily consume data generated by another, utilizing Kubernetes' as a central data platform, is one of the major wins we've encountered by adopting the platform. Finally, I will showcase real-world use cases from NREL where we have used Kubernetes to solve some persistent data challenges involving large volumes of sensor and monitoring data. I will discuss the challenges we encountered when creating our cluster and making it available as a production-ready resource. I will also discuss the specific suite of tools, including Postgres and Apache Druid for columnar and timeseries data, and Redpanda Kafka for streaming data we have deployed in our infrastructure, and the process that went into the selection of these tools.

collaborative environment↗