Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data-driven dynamical systems”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Stoichiometrically-informed symbolic regression for extracting chemical reaction mechanisms from data

A data-driven computational method is introduced to extract chemical reaction mechanisms from time series chemical concentration data. It is realized through the use of dynamic symbolic regression in which a sparse analytical form for a dynamical system is discoverable from the underlying data. We specifically develop the stoichiometrically-informed symbolic regression (SISR) method to address a standing challenge in complex chemical reaction networks: given a time-series dataset of concentrations of several components, what is the mechanism and the associated rate constants? SISR finds the optimal mechanism, kinetic equations and rate constants by combining differential optimization with a genetic optimization approach that searches a symbolic space of possible reaction mechanisms. Use of SISR in several paradigmatic examples spanning linear and nonlinear reaction schemes results in excellent agreement between true and predicted mechanisms, including when the method is applied to noisy data. The advantages of a stoichiometrically-informed approach such as SISR to address reaction discovery is illustrated through comparison with the use of generic state-of-the-art data-driven approaches.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Physics-Informed Neural Network (PINN) Prediction of Mixed Mass-Heat-Crystallization Limited Methane Hydrate Formation and Dissociation in Micro-Confinement

The creation and use of Physics-Informed Neural Networks (PINNs) for simulating the dynamics of methane hydrate formation and dissociation will be presented. The PINN framework's main benefit is its capacity to impose physical consistency with only a partial comprehension of the governing equations. This makes the algorithm especially useful for systems with little experimental evidence or a lack of theoretical knowledge. A strong basis for forecasting methane hydrate behavior over the verified operating ranges of 30.0-80.9 bar pressure and 1.0-4.0 K sub-cooling conditions is provided by the combination of conductive heat transfer equations and mixed mass-transfer–crystallization kinetics. PINNs were more accurate at predicting the mixed mass-heat-crystallization limited kinetics than conventional Artificial Neural Networks (ANNs), demonstrating remarkable predictive accuracy for methane hydrate production over the ANN model. The efficiency of incorporating physical limitations from first principles into machine learning frameworks for methane hydrate crystallizations is reinforced by these findings. For hydrate-related applications in energy generation, carbon sequestration, and climate modelling, our study establishes PINNs as a computational tool that is both scalable and efficient. The proven capacity to close the gap between conventional physics-based simulations and solely data-driven models creates new opportunities for expedited hydrate research and practical applications.

Hartman, Ryan L [NYU Tandon School of Engineering]↗
DOE OSTI2025

A Data-Driven Framework for Predicting the Sorting and Screening Performance of an Integrated Biomass Feedstock Preprocessing System

The characteristics of mechanically sorted and screened lignocellulosic biomass, such as the mass contents of corn stover anatomical fractions (leaves, husks, stalks, cobs, etc.), can be used to calculate the intermediate feedstock quality attributes “yield” and “purity” that indicate the conversion efficiency of biocrude. No prior study has investigated the correlations from the characteristics of raw biomass and preprocessing unit operation parameters to those intermediate feedstock quality attributes. This work presents a data-driven framework for assessing and predicting the intermediate feedstock quality attributes in an integrated biomass feedstock preprocessing system. Our study used corn stover as a typical type of herbaceous biomass because of its abundance in the U.S. It began with data acquisition of moisture content, particle size distribution, and anatomical fractions of the materials after each unit operation in the system. The objective of this preprocessing system is to minimize husks and leaves and maximizing cobs and stalks by mechanically separating the materials into three streams via disc screen and air separator. Prototype neural network models were then developed to evaluate the feasibility of predicting process outcomes based on measurable parameters. It is found that incorporating physical constraints into these prediction models significantly enhances the accuracy of the predicted yield and purity against the ground truth data. The experimental data and model predictions indicate that decreasing throughput increases purity, while higher throughput results in lower purity. Finally, an optimization problem was introduced to search optimal combinations of feed material properties and preprocessing unit operation parameters, as the intermediate feedstock quality attributes – yield and purity, appeared to be competing factors. The study also suggests the continual need to improve the data-driven framework’s predictability by incorporating more accurate physical models to describe the dynamics in the preprocessing units such as the air separator.

09 - BIOMASS FUELS↗
DOE OSTI2026

Contrasting Carbon–Water–Energy Dynamics in Perennial and Annual Bioenergy Agroecosystems Using Eddy Covariance and Interpretable Machine Learning

Understanding how agroecosystems respond to environmental variability is fundamental to predicting productivity and sustainability under a changing climate. We analyzed 55 site-years of high-frequency eddy covariance observations from five agroecosystems—two perennial grasses (miscanthus and switchgrass), two annual rotation systems (maize–soybean and sorghum–soybean), and a restored native prairie—to examine ecosystem-scale carbon, water, and energy fluxes. Using an interpretable machine-learning framework with regression tree ensembles, Shapley Additive Explanations, and Accumulated Local Effects, we quantified how environmental and temporal factors regulate gross primary productivity (GPP), evapotranspiration (ET), water-use efficiency, and the Bowen ratio. Perennials exhibited stronger physiological buffering and maintained fluxes across a broader range of temperature and moisture conditions, reflecting deeper rooting and persistent canopy cover. Annuals, in contrast, showed greater short-term variability and stronger coupling to atmospheric demand, with GPP and ET declining rapidly under low humidity or soil moisture. Differences in temperature sensitivity of Bowen ratio further revealed that perennials sustained proportionally greater sensible heat flux under cool conditions, whereas annuals exhibited constrained energy exchange when evaporative demand was low. Together, these results demonstrate that crop life cycle and canopy structure are fundamental determinants of ecosystem-scale carbon–water–energy coupling. By integrating long-term flux observations with interpretable machine learning, this study identifies the environmental drivers that shape agroecosystem function and highlights how conversion from annual to perennial feedstocks can enhance climatic resilience and alter land–atmosphere energy feedbacks. These findings provide a data-driven basis for improving crop and Earth-system models and for guiding bioenergy landscape design under future climate scenarios.

Accumulated Local Effects↗
DOE OSTI2026

Data-Driven Modeling and Correction of Vehicle Dynamics

We develop a data-driven framework for learning and correcting nonautonomous vehicle dynamics. Physics-based vehicle models are often simplified for tractability and therefore exhibit inherent model-form uncertainty, motivating the need for data-driven correction. Moreover, nonautonomous dynamics are governed by time-dependent control inputs, which pose challenges in learning predictive models directly from temporal snapshot data. To address these, we reformulate the vehicle dynamics via a local parameterization of the time-dependent inputs, yielding a modified system composed ofa sequence of local parametric dynamical systems. Here, we approximate these parametric systems using two complementary approaches. First, we employ the dimension reduction and interpolation in parameter space (DRIPS) methodology to construct efficient linear surrogate models, equipped with lifted observable spaces and manifold-based operator interpolation. This enables data-efficient learning of vehicle models whose dynamics admit accurate linear representations in the lifted spaces. Second, for more strongly nonlinear systems, we employ flow map learning (FML), a deep neural network (DNN) approach that approximates the parametric evolution map without requiring special treatment of nonlinearities. We further extend FML with a transfer-learning-based model correction procedure, enabling the correction of misspecified prior models using only a sparse set of high-fidelity or experimental measurements, without assuming a prescribed form for the correction term. Through a suite of numerical experiments on unicycle, simplified bicycle, and slip-based bicycle models, we demonstrate that DRIPS offers robust and highly data-efficient learning of nonautonomous vehicle dynamics, while FML provides expressive nonlinear modeling and effective correction of model-form errors under severe data scarcity.

data-driven modeling↗
DOE OSTI2026

Regional surrogates for predictive control of digital twins

Digital twins of complex systems must involve a model that is fast, generalizable, and usable for real-time control. For example, high-fidelity nonlinear multiphysics simulations can capture laser-material interactions, but are too slow for optimization or model predictive control (MPC). Reduced-order models, used to accelerate such computation, frequently fail to generalize to unseen inputs or control states. We show theoretically that this failure is intrinsic, i.e., that a learned model is non-unique outside the sampled subspace when its low-rank structure arises from limited excitation and clustered eigenvalues, rather than from a user-imposed truncation alone. Motivated by this result, we propose a control-ready regional surrogate-construction framework for both autonomous and nonautonomous dynamics; it employs Koopman lifting to represent nonlinearities, while preserving spatial locality. We illustrate our approach by constructing a control-ready surrogate for the digital twin of a thermal component of additive-manufacturing process. Our surrogate, localized in space through a von Neumann stencil, is learned from noisy high-fidelity simulations that emulate thermal-camera images collected during the manufacturing. It is linear in thermo-physically augmented states so that MPC reduces to a convex quadratic program. The surrogate requires no online correction, generalizes to unseen scan paths and power profiles of the laser, and is more than three orders of magnitude faster than a finite-difference solver. Furthermore, when the MPC sequence computed on the digital twin is applied to this solver, closed-loop temperature regulation is recovered, showing that the surrogate preserves control-relevant input-output behavior.

Data-driven model↗
DOE OSTI2026

Toward Intelligent Multimodal Holography for Real-Time Chemical Imaging of Dynamic Ion Separation

Molecular-level visualization of ion transport and separation dynamics in complex environments is crucial for advancing energy systems, water purification, and critical materials recovery. Achieving this requires imaging platforms that combine structural sensitivity, chemical specificity, and real-time operation. Digital off-axis holography (DOAH) provides high-throughput, label-free quantitative phase imaging but inherently lacks chemical selectivity. Integrating DOAH with complementary spectroscopic channels such as fluorescence or hyperspectral imaging introduces the needed molecular specificity, while also creating challenges in multimodal data fusion, synchronization, and computational throughput. Artificial intelligence offers a powerful route to address these limitations by uniting physics-based reconstruction with data-driven interpretation. In this Perspective, we outline a framework for intelligent multimodal holography and demonstrate its potential using a preliminary AI-driven test case. Raw DOAH holograms of lanthanide solutions subjected to magnetic field gradients were analyzed using multi-agent AI workflows that autonomously selected reconstruction tools, extracted NMF components, and generated scientific claims consistent with true paramagnetic and diamagnetic behavior. This demonstration shows how AI-enabled reasoning can deliver real-time chemical–structural interpretation directly from raw holograms. Together, these advances define a path toward adaptive, intelligent holography platforms capable of supporting in situ chemical separations, dynamic ion transport analysis, and next-generation interfacial science.

Ricchiuti, Giovanna↗
DOE OSTI2026

A Data-Driven Method for Modeling Creep-Fatigue Stress- Strain Behavior Using Neural ODEs

In this paper, we introduce a data-driven machine learning approach for modeling one-dimensional stress–strain behavior under cyclic loading, utilizing experimental data from the nickel-based Alloy 617. The study employs uniaxial creep–fatigue test data acquired under various loading histories and compares two distinct neural network-based ODE models. The first model, known as the black-box model, comprehensively describes the strain–stress relationship using a Neural ODE equation. To interpret this black-box model, we apply the Sparse Identification of Nonlinear Dynamical Systems (SINDy) technique, transforming the black-box model into an equation-based model using symbolic regression. The second model, the Neural flow rule model, incorporates Hooke’s Law for the linear elastic component, with the nonlinear part characterized by a Neural ODE. Both models are trained with experimental data to accurately reflect the observed stress–strain behavior. We conduct a detailed comparison with the standard Chaboche model, which includes three back stresses. Our results demonstrate that the neural network-based ODE models precisely capture the experimental creep–fatigue mechanical behavior, exceeding the standard Chaboche model’s accuracy. Furthermore, an interpretable model derived from the black-box neural ODE model through symbolic regression achieves accuracy comparable to the Chaboche model, enhancing its interpretability. The results highlight the potential of neural network-based ODE models to depict complex creep–fatigue behavior, eliminating the necessity for experts to define a specific, material-focused model form.

creep-fatigue↗
DOE OSTI2026

Neural network potentials with effective charge separation for non-equilibrium dynamics of ionic solids: a ZnO case study

Developing neural network potentials (NNPs) accurate under non-equilibrium dynamics is challenging, as such systems require extensive sampling beyond equilibrium phases. Here we construct high-fidelity NNPs for zinc oxide (ZnO), a polymorphic ionic solid, using density functional theory (DFT) reference data. To efficiently capture transitional configurations, we combine enhanced-sampling molecular dynamics with empirical potentials, data distillation, and pretraining on short-range atomic energies (A-Train), followed by transfer learning with DFT-relabeled datasets. This hierarchical approach improves transferability across polymorphs and stress states. We further introduce effective charge separation, treating long-range Coulombic terms analytically while short-range residual interactions are learned by the NNP. The optimal effective charges fall in the range 0.5–1.0 q e , consistent with dielectric-screened values derived from formal charges but distinct from Bader estimates. Motivated by this observation, we propose a simple data-driven protocol in which effective charges are optimized by comparing DFT reference energies with explicit Coulomb calculations, without additional NNP training. This strategy improves accuracy and transferability in DFT-level predictions of energies, forces, and stress. Together, these results provide a practical charge-selection framework for robust NNP development in ionic solids, enabling reliable simulation of polymorphic phase transformations and non-equilibrium dynamics.

Chemistry↗
DOE OSTI2026

A simulation framework for evaluating electronic order workflows in integrated health records

Electronic health record (EHR) systems are critical to modern healthcare delivery, yet the dynamic workflows that govern electronic order processing remain underexplored. Inefficiencies in these digital pathways can cause delays in care, repetitive workloads, and even patient harm. This study presents a discrete-event simulation framework used to reconstruct and evaluate EHR-based order workflows in a large integrated healthcare system. Using real-world data extracted from the Veterans Health Administration’s Corporate Data Warehouse, the authors mapped order events to standardized state transitions and modeled their progression across different facilities of varying complexity levels. After being calibrated with empirical distributions of transition times and validated against observed time-in-system metrics, the simulation demonstrates close alignment with historical performance. Scenario analyses reveal that resource capacity constraints significantly amplify the impact of electronic order surges, which are reflected in the disproportionate growth in backlogs and processing delays. Adjustments in transition probabilities further increased recirculation and extended workflow paths. Network-based analysis identified Reserved, InProgress, and Completed as structurally critical states that function as hubs within the process network but the transitions in-between also act as major bottlenecks. These results showcased the effectiveness of simulation-based approaches in monitoring EHR order processing performance and evaluating consequences of workflow changes on healthcare network resources planning. The proposed simulation framework provides a scalable data-driven tool to support operational decision-making and improve the efficiency of electronic order management in complex healthcare environments.

Engineering↗
DOE OSTI2026

Superionic conduction in solid polymer electrolytes – decoupling ion transport from segmental relaxation

Solvent-free, solid polymer electrolytes (SPEs) are promising candidates for next-generation, electrochemical energy storage systems due to their potential to enhance safety and performance, enable flexible device architectures, and streamline manufacturing processes. Conventional SPEs suffer from limited ionic conductivity due to the strong coupling between ion transport and (generally slow) polymer segmental relaxation. The realization of superionic conduction in SPEs, in which ions move faster than the structural relaxation of the polymers, requires a shift in design principles to promote this type of decoupled ion motion. In this perspective, we discuss how polymer architecture, ion–ion correlations, and ion–polymer interactions can unlock superionic behavior. We highlight several key design features, such as crystallinity, bulky side groups, high molecular weight, and percolating ionic aggregation, with a focus on creating low-barrier transport pathways in various polymer systems. We also demonstrate opportunities to combine polymer chemistry and data science through high-throughput and automated screening approaches to reveal how phase behavior, ion dynamics, and ionic interactions govern transport, thereby potentially enabling data-driven discovery of superionic polymer electrolyte materials.

Yang, Mengying [Univ. of Delaware, Newark, DE (Uni↗
DOE OSTI2026

Explaining System-Level Prognostics with Established Machine Learning Methods

System-level prognostics is crucial for ensuring reliability and enabling predictive maintenance in complex systems with interconnected components. This study presents a framework that integrates data-driven methods to predict the remaining useful life (RUL) of a subsystem under multiple and concurrent faults within a nuclear power plant system with explainable artificial intelligence (XAI). A nuclear power plant (NPP) operation was simulated to model the degradation behavior of NPP components, and four machine learning models—Gradient Boosting Regressor (GBR), Support Vector Regressor (SVR), Fully Connected Neural Network (FCNN), and Long Short-Term Memory (LSTM)—were evaluated for prognostics with a novel system RUL parameter. The LSTM model demonstrated potential superior repeatability, while SHAP (SHapley Additive exPlanations) for explainability provided consistent and trustworthy global explanations. In contrast, LIME (Local Interpretable Model-agnostic Explanations) offered localized interpretability but showed reduced stability for sequential data. Key findings include the interplay between component-level degradation and system-wide performance, with LSTM effectively capturing these dynamics through sequence-level predictions. The XAI techniques enhanced transparency by identifying critical features influencing model predictions and aligning with domain knowledge. Furthermore, this framework has significant implications for improving trust and understanding in predictive maintenance, particularly in safety-critical industries like nuclear energy.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2026

Latent Twins

Over the past decade, scientific machine learning has transformed the development of mathematical and computational frameworks for analyzing, modeling, and predicting complex systems. From inverse problems to numerical partial differential equations (PDEs), dynamical systems, and model reduction, these advances have pushed the boundaries of what can be simulated. Yet they have often progressed in parallel, with representation learning and algorithmic solution methods evolving largely as separate pipelines. With Latent Twins, we propose a unifying mathematical framework that creates a hidden surrogate in latent space for the underlying equations. Whereas digital twins mirror physical systems in the digital world, Latent Twins mirror mathematical systems in a learned latent space governed by operators. Through this lens, classical modeling, inversion, model reduction, and operator approximation all emerge as special cases of a single principle. We establish the fundamental approximation properties of Latent Twins for both ordinary differential equations (ODEs) and PDEs and demonstrate the framework across three representative settings: (i) canonical ODEs, capturing diverse dynamical regimes; (ii) a PDE benchmark using the shallow-water equations, contrasting Latent Twin simulations with deep operator network and forecasts with a four-dimensional variational method baseline; and (iii) a challenging real-data geopotential reanalysis dataset, reconstructing and forecasting from sparse, noisy observations. Latent Twins provide a compact, interpretable surrogate for solution operators that evaluate across arbitrary time gaps in a single-shot, while remaining compatible with scientific pipelines such as assimilation, control, and uncertainty quantification. Looking forward, this framework offers scalable, theory-grounded surrogates that bridge data-driven representation learning and classical scientific modeling across disciplines.

Latent Twins↗
DOE OSTI2026

Towards a self-driving trigger at the LHC: adaptive response in real time

Real-time data filtering and selection—or trigger—systems at high-throughput scientific facilities such as the experiments at the Large Hadron Collider must process extremely high-rate data streams under stringent bandwidth, latency, and storage constraints. Yet these systems are typically designed as static, hand-tuned menus of selection criteria grounded in prior knowledge and simulation. In this work, we further explore the concept of a self-driving trigger, an autonomous data-filtering framework that reallocates resources and adjusts thresholds dynamically in real-time to optimize signal efficiency, rate stability, and computational cost as instrumentation and environmental conditions evolve. We introduce a benchmark ecosystem to emulate realistic collider scenarios and demonstrate real-time optimization of a menu including canonical energy sum triggers as well as modern anomaly-detection algorithms that target non-standard event topologies using machine learning. Using simulated data streams and publicly available collision data from the Compact Muon Solenoid experiment, we demonstrate the capability to dynamically and automatically optimize trigger performance under specific cost objectives without manual retuning. Our adaptive strategy shifts trigger design from static menus with heuristic tuning to intelligent, automated, data-driven control, unlocking greater flexibility and discovery potential in future high-energy physics analyses.

Emami, Shaghayegh [Michigan U.] (ORCID:00090007589↗
DOE OSTI2026

Conditional diffusion machine-learning framework for mapping valence electron distribution from convergent beam electron diffraction

Quantitative convergent beam electron diffraction (CBED) enables determination of aspherical valence electron distributions through refinement of low-order structure factors, which are highly sensitive to chemical bonding and charge density variations. However, conventional quantitative CBED (QCBED) requires solving a highly nonlinear inverse problem with many coupled parameters, and computationally intensive dynamical diffraction calculations, making it time-consuming and difficult to apply to complex systems. More broadly, reconstructing charge density and orbital electron distribution from diffraction data has long been a central challenge in both x-ray and electron crystallography. Here, in this study, we introduce an artificial-intelligence (AI)-based framework that replaces traditional refinement with a data-driven inverse solver. Using a large synthetic CBED dataset generated by Bloch-wave simulations, we train a conditional diffusion model to directly infer crystal structural parameters and multipole density formalism parameters, and hence valence electron distributions, from CBED patterns alone. By learning from forward simulations across realistic parameter space, the model effectively solves the inverse problem. Compared with direct regression approaches, the diffusion-based framework provides posterior parameter distributions for rigorous uncertainty quantification while preserving quantitative fidelity and reducing analysis time by orders of magnitude. By eliminating the need for external single-crystal x-ray diffraction data and complex nonlinear refinement, this approach enables practical, high-throughput, and in situ quantitative CBED, enabling real-time mapping of valence electron distributions and their correlation with functional responses in quantum and energy materials.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2026

Bayesian Physics Informed Spatio-Temporal Network for Streamflow Data Imputation

Reliable reconstruction of incomplete streamflow records is critical for improving hydrological forecasting, flood preparedness, and water resource management. However, large observational gaps and uncertainties in governing physical parameters limit the accuracy of traditional statistical and machinelearning imputation frameworks. To address these challenges, we develop a Bayesian Physics-Informed Spatio-Temporal Network (BPI-STNet) that jointly captures spatial and temporal dependencies while enforcing hydrologic consistency through embedded physical constraints. The framework integrates a GraphSAGE-LSTM architecture to model spatial connectivity across gauges and temporal flow dynamics, coupled with a Bayesian update mechanism to estimate uncertain parameters in a simplified water-balance framework. Unlike conventional physics-informed networks that rely on sampling-based posterior estimation, BPI-STNet derives an analytic solution to the inverse problem, allowing closed-form Bayesian updates of uncertain parameters Λ={α,β,k} using Gaussian priors and likelihoods. Applied to daily observations from the Susquehanna River Basin (1980-2022), BPI-STNet achieves substantial improvements over a purely data-driven RGNN baseline, which reduced RMSE by 23 % and MAE by 9 %, and achieving an average NSE values up to 0.96. The results demonstrate that coupling Bayesian inference with physics-informed learning yields physically consistent, uncertainty-aware reconstructions that preserve the temporal persistence and statistical distribution of observed flows. The proposed framework establishes a generalizable paradigm for data-sparse hydrologic systems where both data fidelity and physical interpretability are essential.

Krishnan Kutty Ambika, Anukesh [ORNL] (ORCID:00000↗
DOE OSTI2026

Multi‐Decadal Dynamics of Wetland Methane Emissions Revealed by Knowledge‐Guided Machine Learning

Measurement of methane fluxes (FCH 4 ) from natural systems, such as wetlands, has lagged far behind carbon dioxide fluxes. Short and fragmented wetland FCH 4 data limit our ability to assess its long-term dynamics and potential climate feedbacks. Extrapolating short-term FCH 4 records to recent decades remains challenging for both process-based models and data-driven machine learning (ML) approaches. Here, we develop a knowledge-guided ML framework that integrates eddy covariance (EC) FCH 4 observations, field warming experiments, and biogeochemical knowledge to reconstruct the long-term FCH 4 budgets and trends. Focusing on the 11 longest EC monitoring sites in the AmeriFlux network, we found considerable variability in multi-decadal trends of wetland FCH 4 , with increases up to 14% per decade from 2000 to 2024. We also found that the strength of these increasing trends declines from high to low latitudes, highlighting the vulnerability of northern wetlands. This work presents novel and robust reconstructions of long-term wetland FCH 4 , offering critical benchmark datasets for bottom-up ecosystem models and advancing fundamental understanding of wetland biogeochemistry.

AmeriFlux site↗
DOE OSTI2026

iDDS: intelligent distributed dispatch and scheduling for workflow orchestration

The intelligent distributed dispatch and scheduling (iDDS) service is a versatile workflow orchestration system designed for large-scale, distributed scientific computing. iDDS extends traditional workload and data management by integrating data-aware execution, conditional logic, and programmable workflows, enabling automation of complex and dynamic processing pipelines. Originally developed for the ATLAS experiment at the large hadron collider, iDDS has evolved into an experiment-agnostic platform that supports both template-driven workflows and a Function-as-a-Task model for Python-based orchestration. This paper presents the architecture and core components of iDDS, highlighting its scalability, modular message-driven design, and integration with systems such as PanDA and Rucio. We demonstrate its versatility through real-world use cases: fine-grained tape resource optimization for ATLAS, orchestration of large Directed Acyclic Graph (DAG) workflows for the Rubin Observatory, distributed hyperparameter optimization for machine learning applications, active learning for physics analyses, and AI-assisted detector design at the electron–ion collider. By unifying workload scheduling, data movement, and adaptive decision-making, iDDS reduces operational overhead and enables reproducible, high-throughput workflows across heterogeneous infrastructures. We conclude with current challenges and future directions, including interactive, cloud-native, and serverless workflow support.

97 MATHEMATICS AND COMPUTING↗