Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “model data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 433 records · Page 24
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Reconstruction framework advancements to support streaming for the ePIC detector at the EIC

The ePIC collaboration adopted the JANA2 framework to manage its reconstruction algorithms. This framework has since evolved substantially in response to ePIC’s needs. There have been three main design drivers: integrating cleanly with the Podio-based data models and other layers of the key4hep stack, enabling external configuration of existing components, and supporting timeframe splitting for streaming readout. The result is a unified component model featuring a new declarative interface for specifying inputs, outputs, parameters, services, and resources. This interface enables the user to instantiate, configure, and wire components via an external file. One critical new addition to the component model is a hierarchical decomposition of data boundaries into levels such as Run, Timeframe, PhysicsEvent, and Subevent. Two new component abstractions, Folder and Unfolder, are introduced in order to traverse this hierarchy, e.g. by splitting or merging. The pre-existing components can now operate at different event levels, and JANA2 will automatically construct the corresponding parallel processing topology. This means that a user may write an algorithm once, and configure it at runtime to operate on timeframes or on physics events. Overall, these changes mean that the user requires less knowledge about the framework internals, obtains greater flexibility with configuration, and gains the ability to reuse the existing abstractions in new streaming contexts.

Brei, Nathan [Thomas Jefferson National Accelerato↗
DOE OSTI2022

Simulated tree recruitment at PA-BCI under four precipitation scenarios

Simulated recruitment rates are provided for four tropical tree plant functional types (PFTs) which vary in drought and shade tolerance, at Barro Colorado Island, Panama (PA-BCI). Recruitment rates are predicted using the Tree Recruitment Scheme (TRS; Hanbury-Brown et al., 2022) and model data output from the Ecosystem Demography model version 2 with hydrodynamics (ED2; Medvigy et al., 2009, Powell et al., 2018). Predictions of recruitment are provided under four precipitation scenarios: “BASE” = recycled 2008-2014 observed meteorology (Faybishenko & Paton, 2021), “SYN-ENSO” = two exceptionally strong El Niño events within 30 years (Powell et al., 2018), “WET” = 30% increase in precipitation compared to BASE, and DRY-DS = dry season (January–April) precipitation reduced by 75% compared to BASE. In addition to recruitment rates, this dataset provides TRS predictions of PFT-specific seed bank and seedling pool dynamics (tracked in units of carbon) and ED predictions of understory photosynthetically active radiation and soil matric potential at 6 cm soil depth.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Future Response of Coastal Wetlands to Environmental Stresses: Algorithm Comparison of Numerical Models

Coastal wetlands are a critical component of the earth system that strongly influence the global water and biogeochemical cycles. They are also likely important sentinel of climate change. Because eco-geomorphological processes have long been recognized to be important for coastal wetland survival under accelerated sea-level rise (SLR), many eco-geomorphology models have been developed to assess the impact of climate change on coastal wetlands. Although these models differ substantially in complexity and numerical methods, few studies have investigated the algorithm-level uncertainties in these models. In this study, we developed a multiple-algorithm model framework of coastal wetlands that represents coastal hydrodynamics (such as water level, significant wave height and bottom shear stress) and four eco-geomorphological processes: mineral accretion, organic matter accretion, storm surge erosion and landward migration. We validated the model at three representative coastal wetland sites (Venice Lagoon, Plum Island Estuary and Hunter Estuary) for hydrodynamics, mineral accretion and organic matter accretion. Through model-data comparison, we showed that the model can well capture the dynamics of hydrodynamical and eco-geomorphological conditions in the study sites. Importantly, analysis of the multiple-algorithm simulations suggests that differences in the process representation of mineral and organic matter accretion may contribute to the recent contradicting predictions of coastal wetland evolution under accelerated SLR.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Improving the Accessibility and Usability of Geothermal Information with Data Lakes and Data Pipelines on the Geothermal Data Repository: Preprint

The Geothermal Data Repository (GDR) provides universal access to data and information resulting from research and development activities funded by the Department of Energy (DOE). The GDR has extended this universal access to big data through integration with data lakes developed by the Open Energy Data Initiative (OEDI). Previously, large datasets such as seismic waveform or distributed acoustic sensing (DAS) data could only be accessed by institutions with high performance data storage and compute capabilities, effectively limiting the accessibility of big data to national labs, larger universities, and major corporations. Moreover, the time and resources needed to transport big data and configure them can produce additional barriers to use. Many of the standard formats used for structured data models (also known as content models) are incapable of handling big data and can introduce additional usability problems, often requiring data to be reformatted prior to use. This paper will explore how recent integrations between the GDR and the OEDI data lake have improved the accessibility and usability of geothermal data in a big way, making the data available to a broader audience, and enabling collaborative analysis and innovation across the greater geothermal industry.

access↗
DOE OSTI2021

Improving the Accessibility and Usability of Geothermal Information with Data Lakes and Data Pipelines on the Geothermal Data Repository

The Geothermal Data Repository (GDR) provides universal access to data and information resulting from research and development activities funded by the Department of Energy (DOE). The GDR has extended this universal access to big data through integration with data lakes developed by the Open Energy Data Initiative (OEDI). Previously, large datasets such as seismic waveform or distributed acoustic sensing (DAS) data could only be accessed by institutions with high performance data storage and compute capabilities, effectively limiting the accessibility of big data to national labs, larger universities, and major corporations. Moreover, the time and resources needed to transport big data and configure them can produce additional barriers to use. Many of the standard formats used for structured data models (also known as content models) are incapable of handling big data and can introduce additional usability problems, often requiring data to be reformatted prior to use. This paper will explore how recent integrations between the GDR and the OEDI data lake have improved the accessibility and usability of geothermal data in a big way, making the data available to a broader audience, and enabling collaborative analysis and innovation across the greater geothermal industry.

access↗
DOE OSTI2023

Modeled production, oxidation, and transport processes of wetland methane emissions in temperate, boreal, and Arctic regions

Abstract Wetlands are the largest natural source of methane (CH 4 ) to the atmosphere. The eddy covariance method provides robust measurements of net ecosystem exchange of CH 4 , but interpreting its spatiotemporal variations is challenging due to the co‐occurrence of CH 4 production, oxidation, and transport dynamics. Here, we estimate these three processes using a data‐model fusion approach across 25 wetlands in temperate, boreal, and Arctic regions. Our data‐constrained model—iPEACE—reasonably reproduced CH 4 emissions at 19 of the 25 sites with normalized root mean square error of 0.59, correlation coefficient of 0.82, and normalized standard deviation of 0.87. Among the three processes, CH 4 production appeared to be the most important process, followed by oxidation in explaining inter‐site variations in CH 4 emissions. Based on a sensitivity analysis, CH 4 emissions were generally more sensitive to decreased water table than to increased gross primary productivity or soil temperature. For periods with leaf area index (LAI) of ≥20% of its annual peak, plant‐mediated transport appeared to be the major pathway for CH 4 transport. Contributions from ebullition and diffusion were relatively high during low LAI (<20%) periods. The lag time between CH 4 production and CH 4 emissions tended to be short in fen sites (3 ± 2 days) and long in bog sites (13 ± 10 days). Based on a principal component analysis, we found that parameters for CH 4 production, plant‐mediated transport, and diffusion through water explained 77% of the variance in the parameters across the 19 sites, highlighting the importance of these parameters for predicting wetland CH 4 emissions across biomes. These processes and associated parameters for CH 4 emissions among and within the wetlands provide useful insights for interpreting observed net CH 4 fluxes, estimating sensitivities to biophysical variables, and modeling global CH 4 fluxes.

Ueyama, Masahito↗
DOE OSTI2026

Modeling Framework for Data Center

This chapter highlights the critical need for advanced modeling of data centers due to their rapidly increasing energy consumption and impact on grid reliability. Driven by the demand for AI applications, data centers are projected to consume a significant portion of US energy by 2028, putting stress on an already challenged power grid. The chapter emphasizes the importance of "fast" time-scale models to understand the dynamic interactions between data centers and the grid, especially given the rapid power fluctuations of AI workloads. It outlines a modeling framework that includes both offline and real-time EMT domain simulations, detailing the necessary representations for various components like utility interfaces, transformers, IT loads, UPS, cooling loads, Battery Energy Storage Systems (BESS), generators, protection systems, and higher-level control systems. While standard simulation tools like PSCAD offer basic models, custom development is often required to accurately capture the unique and fast-changing behaviors of modern data centers. The chapter also discusses key metrics and test cases for validating these models, focusing on transient load responses, protection relay coordination, and demand flexibility. Finally, it addresses the challenges of modeling large-scale data centers, such as computational complexity and the trade-off between model fidelity and practicality, suggesting hybrid modeling approaches as a solution. The overarching goal is to create a robust framework that helps assess data center impacts on grid stability, identify vulnerabilities, and inform the development of standards for reliable integration of these large loads into the bulk power system.

25 ENERGY STORAGE↗
DOE OSTI2023

Artificial intelligence models, photos, and data associated with the manuscript “Quantifying Streambed Grain Size, Uncertainty, and Hydrobiogeochemical Parameters Using Machine Learning Model YOLO” (v2)

This data package is associated with the manuscript “Quantifying Streambed Grain Size, Uncertainty, and Hydrobiogeochemical Parameters Using Machine Learning Model YOLO” published in Water Resources Research (Chen et al., 2024). This data package includes the training, validation, testing, and prediction data used by the artificial intelligence (AI) model for automated grain size and hydro-biogeochemistry quantification using streambed photos. The grain size data are extracted for each photo using You Look Only Once (YOLO), a pre-trained object detection model. This data package was originally published in October 2023. It was updated August 2025 (v2; new and modified files). File and folder names were not revised to indicate changes. See the change history section in the readme for more details. Please see flmd.csv for a list of all files contained in this data package and descriptions for each. Please see dd.csv for a data dictionary that defines the column headers of .csv files in the data package. This dataset is comprised of one data folder containing (1) file-level metadata; (2) data dictionary; (3) readme; and (4) six subfolders. Subfolders 1 to 4 include the training, validation, testing, and prediction data. Subfolder 5_Summary includes the summary results of different combinations of training, validation, testing, and prediction data. Subfolder 6_SupplementalData includes additional data downloaded from public sources (Kaufman et al., 2023a; Kaufman et al., 2023b; Garefalakis et al., 2023; Mair et al., 2024; https://github.com/river-corridors-sfa/Geospatial_variables). In total, the data package includes 110 folders and 44,283 files. These files include 9,047 .jpg photos, 1 .png photo, 3 .tif photos; 26,639 photo labels and individual grain sizes and probability from AI (.txt); 8,447 grain size distribution data (.dat); and 126 CSV files for results summary, and 14 required metadata files (.xlsx). The summary CSV files contain 68 columns and approximately 2,200 rows that represent photo names, site locations, recording time, GPS coordinates, grains sizes (D10, D50, D60, and D84), number of grains, and additional hydro-biogeochemical data such as water depth, flow velocity, Manning’s coefficient, friction factor, hydraulic conductivity, permeability, streambed interstitial velocity magnitude, mass transfer rate, and nitrate uptake velocity. The photos were obtained from 75 sites in the Yakima River Basin and the Columbia River shorelines, and other associated data from samples and sensors obtained when the photos were taken are publicly available (Fulton et al. 2022; Grieger et al. 2023). All files are .csv, .txt, .dat, .jpg, or .pdf. We acknowledge the Yakama Nation as owners and caretakers of the lands where we collected some of these data. We thank the Confederated Tribes and Bands of the Yakama Nation Tribal Council and Yakama Nation Fisheries for working with us to facilitate sample collection and optimization of data usage according to their values and worldview.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

The critical role of soil moisture in compound hazards

Soil moisture regulates the exchange of energy, water, and carbon across land–vegetation–atmosphere interfaces. Extremes in soil moisture can amplify natural hazards through interactions with diverse Earth system processes. Despite its mechanistic importance, soil moisture remains underrepresented in hazard research and predictive frameworks. Here, in this study, we review our current understanding of the role of soil moisture in the evolution and onset of diverse compound hazards by synthesizing the latest findings from observational and modelling studies. We highlight key soil moisture mechanisms, including atmospheric feedbacks that amplify drought–heatwave–wildfire events, precipitation couplings that promote clustered storms, and threshold responses that drive vegetation die-offs, trigger landslides, and induce flooding. Persistent challenges in observational data, model representation and operational implementation have limited the integration of soil moisture into hazard early-warning systems. Addressing these gaps through advances in observations, data assimilation, and physics-based and data-driven modelling will enhance hazard prediction and preparedness in a rapidly changing world.

Li, Chuxuan [University of California, Los Angeles↗
DOE OSTI2024

Earth System Reanalysis in Support of Climate Model Improvements

Recent climate model developments, established through increased model resolution, have led to substantial improvements in model simulations of the time-evolving, coupled Earth system and its subcomponents. However, regardless of resolution, climate models will always produce climate features and variability that differ from the real world and will be prone to biases. This is due to many remaining uncertainties, such as in parametric and structural model uncertainty, in the initial conditions prescribed, and in the prescribed (scenario) forcing which varies on decadal to centennial timescales. Further model improvements are expected to arise specifically from improved representation of physical processes realized through model-data fusion. This will create an unprecedented opportunity to better exploit a large array of Earth observations, from in situ measurements to weather radars and satellite observations, as the resolved scales of the models approach those of the observations. For this, climate DA will be the central tool to bring models and observations into consistency, by improving initial conditions, inferring uncertain model parameters and structure, and quantifying uncertainty. Generally, there will be advantages and complementarities of adjoint-based smoother approaches, ensemble-based filter approaches, or new ML-inspired approaches. Yet, the ever-increasing model resolution will present growing challenges arising from computational cost, calling for new ways of performing data assimilation and model optimization. Using the complementarity in a hybrid approach, blending tools and concepts from variational, ensemble and ML methods might be what is required in the future. In this context ML could be important to handle non-linear responses, and to better approximate non-Gaussian distributions.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Impact of Vegetation Assimilation on Flash Drought Characteristics across the Continental United States

Predicting and managing the impacts of flash droughts is difficult owing to their rapid onset and intensification. Flash drought monitoring often relies on assessing changes in root-zone soil moisture. However, the lack of widespread soil moisture measurements means that flash drought assessments often use process-based model data like that from the North American Land Data Assimilation System (NLDAS). Such reliance opens flash drought assessment to model biases, particularly from vegetation processes. Here, we examine the influence of vegetation on NLDAS-simulated flash drought characteristics by comparing two experiments covering 1981–2017: open loop (OL), which uses NLDAS surface meteorological forcing to drive a land surface model using prognostic vegetation, and data assimilation (DA), which instead assimilates near-real-time satellite-derived leaf area index (LAI) into the land surface model. The OL simulation consistently underestimates LAI across the United States, causing relatively high soil moisture values. Both experiments produce similar geographic patterns of flash droughts, but OL produces shorter duration events and regional trends in flash drought occurrence that are sometimes opposite to those in DA. Across the Midwest and Southern United States, flash droughts are 4 weeks (about 70%) longer on average in DA than OL. Moreover, across much of the Great Plains, flash drought occurrence has trended upward according to the DA experiment, opposite to the trend in OL. This sensitivity of flash drought to the representation of vegetation suggests that representing plants with greater fidelity could aid in monitoring flash droughts and improve the prediction of flash drought transitions to more persistent and damaging long-term droughts.

Drought↗
DOE OSTI2023

Plant-wide modeling and techno-economic analysis of a direct non-oxidative methane dehydroaromatization process via conventional and microwave-assisted catalysis

Direct non-oxidative methane dehydroaromatization (DHA) process via conventional and microwave (MW)-assisted thermo-catalytic catalysis is studied. Rate models for methane DHA reactions, including the effect of catalyst deactivation, are developed by using the in-house experimental data. Model results for gas concentration profile and catalyst deactivation are in good agreement with the experimental data. This rate model is then used for the development of dynamic multi-scale, multi-physics commercial-scale reactor models. Total number of fixed bed reactors desired for a cyclic steady state process is estimated. Plant-wide models are then developed for conventional and MW-assisted processes for producing products of desired specifications. Techno-economic analysis of the methane DHA process is undertaken. Economics of these methane DHA processes are compared with the typical multi-step natural gas to aromatics production process via methanol synthesis. Sensitivity of internal rate of return (IRR) and net present value (NPV) to various economic and process parameters such as plant scale, desired rate of return, reactor cost, feedstock and utility cost, catalyst variable cost, and MW reactor cost is studied. Here, electric equivalent efficiency of the conventional methane DHA process is found to be 69.2 % and 67.3 % at 750 °C and 800 °C, respectively, while the MW-assisted methane DHA process has the electric equivalent efficiency of 48.9 % at 800 °C. IRRs of the conventional methane DHA process at 750 °C and 800 °C, and MW-assisted process are 15.2 %, 17.5 %, and 18.8 %, respectively for a methane feed flowrate of 19,782 kg/h, while the IRR of the multi-step natural gas to aromatics production process is estimated to be 0 % for the same plant scale. Impact of change in the methane price, electricity price, and catalyst cost is found to be considerable on the process economics, while the cost of the MW reactor is found to have negligible impact.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2020

Hydrodynamic and Radiographic Toolbox (HART)

With a multi-lab and university team, we propose to develop new methods for a Hydrodynamic and Radiographic Toolbox (HART) that will enable a fuller and more extensive use of experimental radiographic data towards better characterizing and reducing uncertainties in predictive modeling of weapons performance. We will achieve this by leveraging recent developments in areas of computational imaging, statistical and machine learning, and reduced order modeling of hydrodynamics. In terms of software practices, by partnering with XCP (RISTRA Project) we will conform to recent XCP standards that are inline with modern software practices and standards and ensure compatibility and ease of inter-operability with existing codes. Three new activities under this proposal include (a) the development and use of deep learning-based surrogates to accelerate reconstruction and variational inference of density fields from radiographs of hydrotests, (b) a model-data fusion strategy that couples deep learning-based density reconstructions with fast hydrodynamics simulators to better constrain the reconstruction, and (c) a method for treating asymmetries using techniques adopted from limited view tomography. All three new activities will be based on improved treatment of scatter, noise, beam spot movement, detector blur, and flat fielding in the forward model, and a use of sophisticated priors to aid in the re construction. The improvements to the forward model and improved algorithmic design of the reconstruction when complete will be contained in the iterative reconstruction code SHIVA—a code project that we have recently initiated. The many ways in which machine learning can be used in the reconstruction work will be contained in a code HERMES that has been initiated with DTRA support. For example, the significant levels of acceleration that will likely be achieved by the use of machine learning techniques will permit us (and are required) to quantify uncertainties in density retrievals. Next, the two-way coupling between density reconstruction and model-based simulation of the hydrodynamics will be contained in code EREBUS, and will permit a fuller realization of the potential of the data to constrain the hydrodynamic model and better address issues related to asymmetries in the problem. Finally, we anticipate that the better consistency with physics achieved in our reconstructions will allow them to be used by X-Division more so than today.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗
DOE OSTI2020

Automated Cloud Based Long Short-Term Memory Neural Network Based SWE Prediction

Snow derived water is a critical component of the US water supply. Measurements of the Snow Water Equivalent (SWE) and associated predictions of peak SWE and snowmelt onset are essential inputs for water management efforts. This paper aims to develop an integrated framework for real-time data ingestion, estimation, prediction and visualization of SWE based on daily snow datasets. In particular, we develop a data-driven approach for estimating and predicting SWE dynamics using the Long Short-Term Memory neural network (LSTM) method. Our approach uses historical datasets (precipitation, air temperature, SWE, and snow thickness) collected at NRCS Snow Telemetry (SNOTEL) stations to train the LSTM network and current year data to predict SWE behavior. The performance of our prediction was compared for different prediction dates and prediction training datasets. Our results suggest that the proposed LSTM network can be an efficient tool for forecasting the SWE timeseries, as well as Peak SWE and snowmelt timing. Results showed that the window size impacts the model performance (where the Nash Sutcliffe efficiency (NSE) ranged from 0.96 to 0.85 and the Rooted Mean Square Error (RMSE) ranged from 0.038 to 0.07) with an optimum number that should be calibrated for different stations and climate conditions. In addition, by implementing the LSTM prediction capability in a cloud based site-monitoring platform, we automate model-data integration. By making the data accessible through a graphical web interface and an underlying API which exposes both training and prediction capabilities. The associated results can be made easily accessible to a broad range of stakeholders.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Standardising the “Gregory method” for calculating equilibrium climate sensitivity

The equilibrium climate sensitivity (ECS) – the equilibrium global mean temperature response to a doubling of atmospheric CO 2 – is a high-profile metric for quantifying the Earth system's response to human-induced climate change. A widely applied approach to estimating the ECS is the “Gregory method” (Gregory et al., 2004), which uses an ordinary least squares (OLS) regression between the net radiative flux, N, and surface air temperature anomalies, ΔT, from a 150 year experiment in which atmospheric CO 2 concentrations are quadrupled. The ECS is determined by extrapolating the linear fit to N=0, i.e. the ΔT-intercept, indicating the point at which the system is back in equilibrium. This method has been used to compare ECS estimates across the CMIP5 and CMIP6 ensembles and will likely be a key diagnostic for CMIP7. Despite its widespread application, there is little consistency or transparency between studies in how the climate model data is processed prior to the regression, leading to potential discrepancies in ECS estimates. We identify 32 alternative data processing pathways, varying by differences in global mean weighting, net radiative flux variable, anomaly calculation method, and linear regression fit. Using 44 CMIP6 models, we systematically assess the impact of these choices on ECS estimates and calculate uncertainty ranges using two bootstrap approaches. While the inter-model ECS range is insensitive to the data processing pathway, individual outlier models exhibit notable differences. Approximating a model's native grid cell area (if irregular) with cosine of the latitude can decrease the ECS by 11 %, the choice of N-variable can change the ECS by 6 %, and some anomaly calculation methods can introduce spurious temporal correlations in the processed data. Beyond data processing choices, we also evaluate an alternative linear regression method – total least squares (TLS) – which has a more statistically robust basis than OLS. However, for consistency with previous literature, and given TLS may reduce the ECS compared to OLS (by up to 24 %), thereby making a known bias in the Gregory method worse, we do not feel there is sufficient clarity to recommend a transition to TLS in all cases. To improve reproducibility and comparability in future studies, we recommend a standardised Gregory method: weighting the global mean by cell area, using the top of the atmosphere (as opposed to the top of model) N-variable, and calculating anomalies by first applying a rolling average to the preindustrial control timeseries then subtracting from the raw CO 2 quadrupling experiment. This approach accounts for model drift while reducing noise in the data to best meet the pre-conditions of the linear regression. While CMIP6 results of the multi-model mean ECS appear insensitive to these processing choices, similar assumptions may not hold for CMIP7, underscoring the need for standardised data preparation in future climate sensitivity assessments.

Geosciences↗
DOE OSTI2020

A physics-informed operator regression framework for extracting data-driven continuum models

The application of deep learning toward discovery of data-driven models requires careful application of inductive biases to obtain a description of physics which is both accurate and robust. We present here a framework for discovering continuum models from high fidelity molecular simulation data. Our approach applies a neural network parameterization of governing physics in modal space, allowing a characterization of differential operators while providing structure which may be used to impose biases related to symmetry, isotropy, and conservation form. Here, we demonstrate the effectiveness of our framework for a variety of physics, including local and nonlocal diffusion processes and single and multiphase flows. For the flow physics we demonstrate this approach leads to a learned operator that generalizes to system characteristics not included in the training sets, such as variable particle sizes, densities, and concentration.

42 ENGINEERING↗
DOE OSTI2022

Proximity Portability and in Transit , M-to-N Data Partitioning and Movement in SENSEI [Book Chapter]

In high-performance parallel in situ processing, the term in transit processing refers to those configurations where data must move from a producer to a consumer that runs on separate resources. In the context of parallel and distributed computing on an HPC platform one of the central challenges is to determine a mapping of data from producer ranks to consumer ranks. This problem is complicated by the heterogeneity that arises in producer-consumer pairs, such as when producer and consumer codes have different levels of concurrency, different scaling characteristics, or different data models. The resulting mapping and movement of data from M producer to N consumer ranks can have a significant impact on aggregate application performance, particularly when the data consumer requires only a subset of the overall data for its task. This chapter focuses on the design considerations that underlie SENSEI’s implementation to this challenging problem. These design considerations extend the core SENSEI architecture and include ideas like the need to accommodate flexibility in the choice of different partitioning methods, the ability for a data consumer to request and receive only the subset of data needed for its particular operation, and the ability to leverage any of several different data transport tools. The idea of proximity portability, being able to use different data transport methods as part of an in transit workflow, is illustrated through the use of three different transport layers where switching from one transport tool to another is accomplished with only a configuration file change. Here, the chapter also includes a performance analysis summary showing the performance gains that are possible in terms of multiple metrics, such as memory footprint, time to solution, and amount of data moved, when using optimized partitioners in an in transit setting, gains that are made possible by the implementation shaped by specific design considerations.

Bethel, E. Wes↗