Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “large scale analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Extremely Scalable Distributed Computation of Contour Trees via Pre-Simplification

Contour trees offer an abstract representation of the level set topology in scalar fields and are widely used in topological data analysis and visualization. However, applying contour trees to large-scale scientific datasets remains challenging due to scalability limitations. Recent developments in distributed hierarchical contour trees have addressed these challenges by enabling scalable computation across distributed systems. Building on these structures, advanced analytical tasks—such as volumetric branch decomposition and contour extraction—have been introduced to facilitate large-scale scientific analysis. Despite these advancements, such analytical tasks substantially increase memory usage, which hampers scalability. In this paper, we propose a pre-simplification strategy to significantly reduce the memory overhead associated with analytical tasks on distributed hierarchical contour trees. We demonstrate enhanced scalability through strong scaling experiments, constructing the largest known contour tree—comprising over half a trillion nodes with complex topology—in under 15 minutes on a dataset containing 550 billion elements.

Li, Mingzhe [University of Utah]↗
DOE OSTI2024

Pando

SAND2025-02006O Pando is a distributed data analysis software tool. It is designed to handle large-scale graph analysis problems, often with a specific focus on blockchain/cryptocurrency data. Pando handles scalability by running on a distributed cluster of servers. Users can customize the output using the program’s plugin/extension design methodology. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Gabert, Kasimir↗
DOE OSTI2022

Simulation process and data flow for a large system dynamics model

This paper documents the workflow and supporting technologies that a large system dynamics model, the biomass scenario model, employs to streamline the data preparation, simulation, quality control, and analysis process at the National Renewable Energy Laboratory. The workflow centers on automation of routine aspects of the flow of data between data stores, simulations, and visualizations. It enforces quality checks on data, reproducibility of computations, and traceability of results, while maintaining complete archives of modeling and analysis artifacts. The resulting frictionless simulation/analysis environment supports large-scale sensitivity analysis, interactive creation of ensembles of simulations, and rapid visualization-based exploration of simulation results.

09 BIOMASS FUELS↗
DOE OSTI2025

A Scale‐Adaptive Urban Hydrologic Framework: Incorporating Network‐Level Storm Drainage Pipes Representation

Abstract Below‐ground urban stormwater networks (BUSNs) significantly influence urban flood dynamics, yet their representation at the watershed or larger scales remains challenging. We introduce a scalable urban hydrologic framework that centers on a novel network‐level BUSN representation, balancing the needs for physical basis, parameter parsimony, and computational efficiency. Our framework conceptualizes an urban watershed into four interacting zones: hillslopes (natural), storm‐sewersheds (urban), a sub‐network channel (tributaries), and a main channel. We develop an innovative Graph Theory‐based algorithm to derive network‐level BUSN parameters from publicly available datasets, enabling efficient, scalable parameterization. We demonstrate this framework's applicability at nine representative watersheds in the Houston metropolitan region, USA, with urban imperviousness ranging from 0% to 64% and drainage areas ranging from 24 to 302 . Our model achieves satisfying computational efficiency, completing hourly time step simulations for 18 years in less than 5 sec per watershed on a standard PC. Validation against observed daily streamflow confirms that the model can capture small‐to‐large flood peaks and seasonal and annual water balance over these watersheds. Comparisons with the National Water Model show better performance in predicting flood peaks and overall water balance, underscoring the promises of our new framework for urban hydrologic modeling at large scales. Furthermore, analysis reveals nonlinear relationships between BUSNs' designed capacities and flood reduction effects. Our approach bridges the gap between detailed hydraulic and large‐scale hydrologic models, providing a valuable tool for urban flood prediction and management across broader spatial and temporal scales.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Spectral Clustering-Based Partitioning of Large-Scale Power Electronics-Based Power Systems for Small-Signal Stability Analysis

The nodal admittance matrix (NAM)-based approach is well-suited for small-signal stability analysis of large-scale power electronics-based power systems (PEPSs), as it preserves the system structure through its admittance matrix. Previous studies have explored partitioning such systems into subareas and interconnections to reduce computational burden; however, they lacked a formal algorithmic procedure for determining feasible partitions. While several grid partitioning methods, such as those based on graph theory or machine learning, exist in the literature, they cannot be directly applied to NAM-based analysis due to differing objectives and constraints. Here, this paper addresses this gap by presenting a systematic, step-by-step procedure for applying a spectral partitioning algorithm that yields a division of the system into subareas suitable for NAM-based analysis. The computational complexity of the proposed method is also derived to demonstrate its efficiency and justify the practicality of the resulting subarea decomposition. The performance of the partitioning method is evaluated by applying the spectral clustering-derived subareas and interconnections to the NAM-based partitioning approach on a 140-bus system. Computational times for the full-system and partitioned NAM analyses are compared using MATLAB. Additionally, PSCAD simulations of the complete system and partitioned subareas are carried out to verify the effectiveness of the proposed method.

Nupur [Univ. of Tennessee, Knoxville, TN (United S↗
DOE OSTI2021

Advanced Computing, Data Science, and Artificial Intelligence Research Opportunities for Energy-Focused Transportation Science

The Energy Efficient Mobility Systems (EEMS) technology landscape is complex and rapidly evolving, which provides both tremendous opportunities and formidable challenges. Significant alterations to the mobility landscape are underway due to the advent of vehicle and infrastructure connectivity, autonomous driving, and rapid passenger- and freight-vehicle electrification. Advanced computing will play an increasingly important role in enabling the EEMS program to understand and identify the most important levers to improve the energy productivity of future integrated mobility systems. It is also driving new approaches to mobility and the research to unlock an affordable, efficient, safe, and accessible transportation future. Driving much of this change is the collection, analysis, and strategic use of massive amounts of diverse, complex data from infrastructure and vehicles with on-board sensors and data storage and transmission capabilities. Diverse and representative data are key to implementing approaches to maximize mobility energy productivity. While high-fidelity modeling of integrated transportation networks has strengthened our understanding of dynamic movement and behavior patterns, existing tools must be expanded beyond their current focus. This work necessitates data infrastructure investments (e.g., secure-streaming data platforms driven by ubiquitous sensors and video analytics) as well as investments in critical capabilities for large-scale automated analysis and organization using modern machine learning, statistics, and artificial intelligence. Other chief needs include agile, large-scale storage that can be quickly searched and queried for relevant data to support validation and model development, data-sharing agreements, and formatting standards for key data types. The future of public transit must be explored in greater detail, research must inform design, and opportunities must be identified for improving the mobility productivity of public transit in both urban and rural America.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2021

Contingency Analysis Based on Partitioned and Parallel Holomorphic Embedding

In the steady-state contingency analysis, the traditional Newton-Raphson method suffers from non-convergence issues when solving post-outage power flow problems, which hinders the integrity and accuracy of security assessment. In this paper, we propose a novel robust contingency analysis approach based on holomorphic embedding (HE). Here, the HE-based simulator provides theoretical convergence guarantee, which is desirable because it avoids the influence of numerical issues and provides a credible security assessment conclusion. In addition, based on the multi-area characteristics of real-world power systems, a partitioned HE (PHE) method is proposed with an interfacebased partitioning of HE formulation. The PHE method does not undermine the numerical robustness of HE and significantly reduces the computation burden in large-scale contingency analysis. The PHE method is further enhanced by parallel or distributed computation to become parallel PHE (P2HE). Tests on a 458-bus system, a synthetic 419-bus system and a large-scale 21447-bus system demonstrate the advantages of the proposed methods in robustness and efficiency.

42 ENGINEERING↗
DOE OSTI2021

Sentiment Analysis based Error Detection for Large-Scale Systems

Today's large-scale systems such as High Performance Computing (HPC) Systems are designed/utilized towards exascale computing, inevitably decreasing its reliability due to the increasing design complexity. HPC systems conduct extensive logging of their execution behaviour. In this paper, we leverage the inherent meaning behind the log messages and propose a novel sentiment analysis-based approach for the error detection in large-scale systems, by automatically mining the sentiments in the log messages. Our contributions are four-fold. (1) We develop a machine learning (ML) based approach to automatically build a sentiment lexicon, based on the system log message templates. (2) Using the sentiment lexicon, we develop an algorithm to detect system errors. (3) We develop an algorithm to identify the nodes and components with erroneous behaviors, based on sentiment polarity scores. (4) We evaluate our solution vs. other state-of-the-art machine/deep learning algorithms based on three representative supercomputers' system logs. Experiments show that our error detection algorithm can identify error messages with an average MCC score and f -score of 91% and 96% respectively, while state of the art ML/deep learning model (LSTM) obtains only 67% and 84%. To the best of our knowledge, this is the first work leveraging the sentiments embedded in log entries of large-scale systems for system health analysis.

error detection↗
DOE OSTI2020

A Scalable Parallel Hypergraph Generator (HyGen)

Graphs are extensively used to model real-world complex systems. An edge in a graph can model pairwise relationships. However, multiway relationships (connections between three or more vertices) are common in many complex systems such as cellular process, image segmentation, and circuit design. A graph edge cannot model multiway relationships. A hypergraph, which can connect more than two vertices, is thus a better option to model multiway relationships. A large-scale hypergraph analysis has the potential to find useful insights from a complex system and assist in knowledge discovery. Currently a limited number of hypergraphs exists that are representative of real-world datasets. Moreover, real-world hypergraph datasets are small in size and inadequate to incorporate future needs. A graph generator that can produce large-scale synthetic hypergraphs can solve the above mentioned problems. In this paper, we present a scalable parallel hypergraph generator (HyGen) based on the Message Passing Interface (MPI) standard. To generate hypergraphs, HyGen takes the following parameter values as inputs: i) number of vertices, ii) number of hyperedges, iii) number of clusters, iv) vertex distribution, v) hyperedge distribution, vi) local cluster cardinality, and vii) global cluster cardinality. We have demonstrated that HyGen can generate hypergraphs of various sizes in a scalable fashion. HyGen takes approximately four minutes to generate a hypergraph with 4.8 million vertices, 1.6 million hyperedges, and 800 clusters using 1,024 processes on a leadership class computing platform. Our strong and weak scaling experiments on supercomputers demonstrate that HyGen can quickly create large-scale hypergraphs in a parallel manner, thus providing a useful capability for hypergraph analysis.

Hasan, S M Shamimul↗
DOE OSTI2020

Real-Time Coupling of Geographically Distributed Research Infrastructures: Taxonomy, Overview, and Real-World Smart Grid Applications

Novel concepts enabling a resilient future power system and their subsequent experimental evaluation are experiencing a steadily growing challenge: large scale complexity and questionable scalability. The requirements on a research infrastructure (RI) to cope with the trends of such a dynamic system therefore grow in size, diversity and costs, making the feasibility of rigorous advancements questionable by a single RI. Analysis of large scale system complexity has been made possible by the real-time coupling of geographically separated RIs undertaking geographically distributed simulations (GDS), the concept of which brings the equipment, models and expertise of independent RIs, in combination, to optimally address the challenge. This article presents the outputs of IEEE PES Task Force on Interfacing Techniques for Simulation Tools towards standardization of GDS as a concept. First, the taxonomy for setups utilized for GDS is established followed by a comprehensive overview of the advancements in real-time couplings reported in literature. The overview encompasses fundamental technological design considerations for GDS. The article further presents four application oriented case studies (real-world implementations) where GDS setups have been utilized, demonstrating their practicality and potential in enabling the analysis of future complex power systems.

distributed laboratories↗
DOE OSTI2025

3D Multiphysics Model for Large Scale Planar Cell: Inductance Investigation in Impedance Analysis

Developed a comprehensive 3D multiphysics model to analyze impedance behavior in large-scale planar Solid Oxide Fuel Cells (SOFCs). The model investigates how impedance varies across operating conditions and cell components, with a particular focus on inductive loops or negative capacitance at the air electrode. It was found that the Inductive loop disappears in the low-frequency zone when the activation overpotential or faradic current is not temperature-dependent. These findings offer new insights into SOFC impedance behavior.

impedence analysis↗
DOE OSTI2025

The Analysis Description Language Ecosystem: Latest developments and physics applications

We present latest developments in Analysis Description Language (ADL), a declarative domain-specific language describing the physics algorithm of a HEP data analysis decoupled from software frameworks. Analyses written in ADL can be integrated into any framework for various tasks. ADL is a multipurpose construct with uses ranging from analysis design to preservation, reinterpretation, queries, visualisation, combination, etc. The most advanced infrastructure to execute ADL on events is the CutLang runtime interpreter. Recent technical developments include an automated interface with different data types, generation of the abstract syntax tree, a visualization tool that that auto-converts analysis flows to graphs, incorporation of trained machine learning models and a Jupyter-based plotting tool. We also report physics implications including a large scale LHC analysis implementation and validation effort for beyond the standard model reinterpretation purposes and studies with ATLAS and CMS open data.

Sekmen, Sezen [Kyungpook National Univ., Daegu (Ko↗
DOE OSTI2025

Quantitative approaches for multiscale structural analysis with atomic resolution electron microscopy

Atomic-resolution imaging with scanning transmission electron microscopy is a powerful tool for characterizing the nanoscale structure of materials, in particular features such as defects, local strains, and symmetry-breaking distortions. In addition to advanced instrumentation, the effectiveness of the technique depends on computational image analysis to extract meaningful features from complex datasets recorded in experiments, which can be complicated by the presence of noise and artifacts, small or overlapping features, and the need to scale analysis over large representative areas. Here, we present image analysis approaches which synergize real and reciprocal space information to efficiently and reliably obtain meaningful structural information with picometer scale precision across hundreds of nanometers of material from atomic-resolution electron microscope images. Damping superstructure peaks in reciprocal space allows symmetry-breaking structural distortions to be disentangled from other sources of inhomogeneity and measured with high precision. Real-space fitting of the wavelike signals resulting from Fourier filtering enables absolute quantification of lattice parameter variations and strain, as well as the uncertainty associated with these measurements. Implementations of these algorithms are made available as an open source python package.

36 MATERIALS SCIENCE↗
DOE OSTI2021

Observability Analysis of a Power System Stochastic Dynamic Model Using a Derivative-Free Approach

Serving as a prerequisite to power system dynamic state estimation, the observability analysis of a power system dynamic model has recently attracted the attention of many power engineers. However, because this model is typically nonlinear and large-scale, the analysis of its observability is a challenge to the traditional derivative-based methods. Indeed, the linear-approximation-based approach may provide unreliable results while the nonlinear-technique-based approach inevitably faces extremely complicated derivations. Furthermore, because power systems are intrinsically stochastic, the traditional deterministic approaches may lead to inaccurate observability analyses. In this work, facing these challenges, we propose a novel polynomial-chaos-based derivative-free observability analysis approach that not only is free of any linear approximations, but also accounts for the stochasticity of the dynamic model while bringing a low implementation complexity. Furthermore, this approach enables us to quantify the degree of observability of a stochastic model, what conventional deterministic methods cannot do. The excellent performance of the proposed method has been demonstrated by performing extensive simulations using a synchronous generator model with IEEE-DC1A exciter and the TGOV1 turbine governor.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2024

CORRLA-RS

The CORRLA-RS package provides a suite of statistical methods for sampling multidimensional distributions and to conduct sensitivity and correlation analysis of large scale data in the Rust programming language. The software provides a unique solution to multidimensional constrained sampling problems utilizing a combination of parallelized Markov Chain Monte Carlo methods and traditional rejection sampling. The sensitivity and correlation analysis methods are backed by a high performance randomized singular value decomposition implementation which enables datasets larger than the random access memory (RAM) size to be analyzed. Additionally, CORRLA-RS implements the active subspace identification method using a KD-Tree and the randomized singular value decomposition acting in concert.

Gurecky, William [Oak Ridge National Laboratory (O↗
DOE OSTI2020

Improvement of Atmospheric Objective Analysis Over Sloping Terrain and Its Impact on Shallow-Cumulus Clouds in Large-Eddy Simulations

Surface topography strongly impacts the regional atmospheric circulation dynamically and thermodynamically. Over the Great Plains in the United States, the gently tilted slope is an important factor that impacts clouds, convection and regional circulations. This study enhances an atmospheric constrained variational analysis by using a terrain-following sigma vertical coordinate and applies to the data collected at the Southern Great Plain (SGP) site of the Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) program. Sensitivity studies are performed to examine the impact of the terrain effects on the derived large-scale atmospheric forcing fields and the simulated shallow-cumulus clouds in large-eddy simulation (LES) driven by the forcing. We found that the terrain impacts on the large-scale forcing fields are mainly at lower levels and are strongly controlled by up/downslope winds. The response of the derived forcing to the slope of the terrain is monotonic, but the response of the simulated shallow-cumulus clouds is more complex and depends on several factors. Overall, the terrain impact is small over SGP due to the small slope angle. However, the flat-surface assumption may cause larger biases in the large-scale forcing fields at locations with steeper terrain. The new sigma-coordinate algorithm, with its consideration of surface slope, should be more suitable to derive large-scale objective analysis over regions with steep terrain for application to force single-column models, cloud resolving models and LES models.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Analysis of Bis(trifluoromethylsulfonyl)imide Interactions with Metal Cations Through a Chemical Informatics Approach

Nominally weakly coordinating anions are useful for modulating the solubility and chemical properties of metal complexes, but identification and analysis of the systematics of the interactions of anions with cationic metal complexes has not received the attention it deserves. Here, a chemical informatics approach is demonstrated for identifying and quantitatively analyzing the ways that the bis(trifluoromethylsulfonyl)imide anion (TFSI) can interact with metal-containing species. An open access computer program (PyCIFTer) was developed to facilitate large-scale structural analysis of TFSI-containing species by utilization of experimental atomic coordinate data from single-crystal X-ray diffraction (XRD) studies obtained from the Cambridge Structural Database (CSD). PyCIFTer establishes a three-dimensional vector space from the raw atomic coordinates, generating acyclic, undirected graphs that are used to rapidly analyze the structural properties (bond lengths and angles) of TFSI in individual structures in sequential/batch fashion. The structures are sorted by PyCIFTer into groups based on pre-set and chemically sensible criteria, affording a comprehensive and systematic view of TFSI structural chemistry. This approach avoids tedious one-at-a-time interrogation of structures, a prospect unreasonable in this case, and many others of contemporary chemical relevance; there were over 1500 structures in the CSD containing TFSI as of November 2024. The results demonstrate that TFSI only rarely binds to cations in the solid state, favoring the formation of species in which TFSI is found in cations’ outer coordination spheres. The prospect of applying PyCIFTer to other moieties is also discussed. PyCIFTer is also schematically compared to the commercial CSD Python application programming interface (API). Taken together, this work demonstrates the usefulness of modular workflows for sequential/batch analysis of structural data from XRD, an approach that appears poised to accelerate the translation of legacy structural results into new chemical insights and hypotheses.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗