Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Graph Processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 991 records · Page 55
All sourcesNASA NTRSDOE OSTI
NASA NTRS1977

On the photographic processing and digital texture for remote sensing of Kujukuri Coast of Chiba in Japan

Remote sensing of various coastal phenomena on the Kujukuri Coast and Kashimanada Coast was done by the use of aircraft for the purpose of investigating the characteristics of shore reefs and floating sand, and the depth of the sea. A multispectral camera and a video ITV camera were used as sensors. The first flight was over the Kashimanada and Kujukuri Coasts and the next flight was over the Katsuura Bay. The shape of shore reefs, the state of floating sand, the depth of the sea, etc., are represented by equidensitographs using texture. The interval of density slices is 0.05. Correlations between the textures represented by equidensitographs, digital graphs and analog display are estimated.

Genda, H.↗
DOE OSTI2018

A multiscale multi-permeability poroplasticity model linked by recursive homogenizations and deep learning

Many geological materials, such as shale, mudstone, carbonate rock, limestone and rock salt are multi-porosity porous media in which pores of different scales may co-exist in the host matrix. When fractures propagate in these multi-porosity materials, these pores may enlarge and coalesce and therefore change the magnitude and the principal directions of the effective permeability tensors. The pore-fluid inside the cracks and the pores of host matrix may interact and exchange fluid mass, but the difference in hydraulic properties of these pores often means that a single homogenized effective permeability tensor field is insufficient to characterize the evolving hydraulic properties of these materials at smaller time scale. Furthermore, the complexity of the hydro-mechanical coupling process and the induced mechanical and hydraulic anisotropy originated from the micro-fracture and plasticity at grain scale also makes it difficult to propose, implement and validate separated macroscopic constitutive laws for numerical simulations. Here this article presents a hybrid data-driven method designed to capture the multiscale hydro-mechanical coupling effect of porous media with pores of various different sizes. At each scale, data-driven models generated from supervised machine learning are hybridized with classical constitutive laws in a directed graph that represents the numerical models. By using sub-scale simulations to generate database to train material models, an offline homogenization procedure is used to replace the up-scaling procedure to generate cohesive laws for localized physical discontinuities at both grain and specimen scales. Through a proper homogenization procedure that preserves spatial length scales, the proposed method enables field-scale simulations to gather insights from meso-scale and grain-scale micro-structural attributes. This method is proven to be much more computationally efficient than the classical DEM–FEM or FEM2 approach while at the same time more robust and flexible than the classical surrogate modeling approach. Due to the usage of bridging-scale technique, the proposed model may provide multiple opportunities to incorporate different types of simulations and experimental data across different length scales for machine learning. Numerical issues will also be discussed.

42 ENGINEERING↗
NASA NTRS2011

Determining the Optimal Number of Clusters with the Clustergram

Cluster analysis aids research in many different fields, from business to biology to aerospace. It consists of using statistical techniques to group objects in large sets of data into meaningful classes. However, this process of ordering data points presents much uncertainty because it involves several steps, many of which are subject to researcher judgment as well as inconsistencies depending on the specific data type and research goals. These steps include the method used to cluster the data, the variables on which the cluster analysis will be operating, the number of resulting clusters, and parts of the interpretation process. In most cases, the number of clusters must be guessed or estimated before employing the clustering method. Many remedies have been proposed, but none is unassailable and certainly not for all data types. Thus, the aim of current research for better techniques of determining the number of clusters is generally confined to demonstrating that the new technique excels other methods in performance for several disparate data types. Our research makes use of a new cluster-number-determination technique based on the clustergram: a graph that shows how the number of objects in the cluster and the cluster mean (the ordinate) change with the number of clusters (the abscissa). We use the features of the clustergram to make the best determination of the cluster-number.

Fluegemann, Joseph K.↗
DOE OSTI2013

Toward a Data Scalable Solution for Facilitating Discovery of Scientific Data Resources

Science is increasingly motivated by the need to process larger quantities of data. It is facing severe challenges in data collection, management, and processing, so much so that the computational demands of "data scaling" are competing with, and in many fields surpassing, the traditional objective of decreasing processing time. Example domains with large datasets include astronomy, biology, genomic, climate and weather, and material sciences. This paper presents a real-world use case in which we wish to answer queries provided by domain scientists in order to facilitate discovery of relevant science resources. The problem is that the metadata for these science resources is very large and is growing quickly, rapidly increasing the need for a data scaling solution. We propose the use of our SGEM stack -- a system designed for answering graph-based queries over large datasets on cluster architectures -- for answering complex queries over the metadata, and we report early results for our current capability.

Chappell, Alan R.↗
DOE OSTI2017

Addressing Global Data Dependencies in Heterogeneous Asynchronous Runtime Systems on GPUs

Large-scale parallel applications with complex global data dependencies beyond those of reductions pose significant scalability challenges in an asynchronous runtime system. Internodal challenges include identifying the all-to-all communication of data dependencies among the nodes. Intranodal challenges include gathering together these data dependencies into usable data objects while avoiding data duplication. This paper addresses these challenges within the context of a large-scale, industrial coal boiler simulation using the Uintah asynchronous many-task runtime system on GPU architectures. We show significant reduction in time spent analyzing data dependencies through refinements in our dependency search algorithm. Multiple task graphs are used to eliminate subsequent analysis when task graphs change in predictable and repeatable ways. Using a combined data store and task scheduler redesign reduces data dependency duplication ensuring that problems fit within host and GPU memory. Furthermore, these modifications did not require any changes to application code or sweeping changes to the Uintah runtime system. We report results running on the DOE Titan system on 119K CPU cores and 7.5K GPUs simultaneously. Our solutions can be generalized to other task dependency problems with global dependencies among thousands of nodes which must be processed efficiently at large scale.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Q-BEEP: Quantum Bayesian Error Mitigation Employing Poisson Modeling over the Hamming Spectrum

Quantum computing technology has grown rapidly in recent years, with new technologies being explored, error rates being reduced, and quantum processor’s qubit capacity growing. However, near-term quantum algorithms are still unable to be induced without compounding consequential levels of noise, leading to non-trivial erroneous results. Quantum Error Correction (in-situ error mitigation) and Quantum Error Mitigation (post-induction error mitigation) are promising fields of research within the quantum algorithm scene, aiming to alleviate quantum errors, increasing the overall fidelity and hence the overall quality of circuit induction. Earlier this year, a pioneering work, namely HAMMER, published in ASPLOS-22 demonstrated the existence of a latent structure regarding post-circuit induction errors when mapping to the Hamming spectrum. However, they intuitively assumed that errors occur in local clusters, and that at higher average Hamming distances this structure falls away. In this work, we show that such a correlation structure is not only local but extends certain non-local clustering patterns which can be precisely described by a Poisson distribution model taking the input circuit, the device run time status (i.e., calibration statistics) and qubit topology into consideration. Using this quantum error characterizing model, we developed an iterative algorithm over the generated Bayesian network state-graph for post-induction error mitigation. Thanks to more precise modeling of the error distribution latent structure and the new iterative method, our Q-Beep approach provides state of the art performance and can boost circuit execution fidelity by up to 234.6% on Bernstein-Vazirani circuits and on average 71.0% on QAOA solution quality, using 16 practical IBMQ quantum processors. For other benchmarks such as those in QASMBench, the fidelity improvement is up to 17.8%. Q-Beep is a light-weight post-processing technique that can be performed offline and remotely, making it a useful tool for quantum vendors to integrate and provide more reliable circuit induction results.

Stein, Samuel A.↗
DOE OSTI2014

On the Suitability of MPI as a PGAS Runtime

Partitioned Global Address Space (PGAS) models are emerging as a popular alternative to MPI models for designing scalable applications. At the same time, MPI remains a ubiquitous communication subsystem due to its standardization, high performance, and availability on leading platforms. In this paper, we explore the suitability of using MPI as a scalable PGAS communication subsystem. We focus on the Remote Memory Access (RMA) communication in PGAS models which typically includes {\em get, put,} and {\em atomic memory operations}. We perform an in-depth exploration of design alternatives based on MPI. These alternatives include using a semantically-matching interface such as MPI-RMA, as well as not-so-intuitive interfaces such as MPI two-sided with a combination of multi-threading and dynamic process management. With an in-depth exploration of these alternatives and their shortcomings, we propose a novel design which is facilitated by the data-centric view in PGAS models. This design leverages a combination of highly tuned MPI two-sided semantics and an automatic, user-transparent split of MPI communicators to provide asynchronous progress. We implement the asynchronous progress ranks approach and other approaches within the Communication Runtime for Exascale which is a communication subsystem for Global Arrays. Our performance evaluation spans pure communication benchmarks, graph community detection and sparse matrix-vector multiplication kernels, and a computational chemistry application. The utility of our proposed PR-based approach is demonstrated by a 2.17x speed-up on 1008 processors over the other MPI-based designs.

Computational modeling↗
DOE OSTI2025

Optimizing Management of Persistent Data Structures in High-Performance Analytics

Large-scale data analytics workflows ingest massive input data into various data structures, including graphs and key-value datastores. These data structures undergo multiple transformations and computations and are typically reused in incremental and iterative analytics workflows. Persisting in-memory views of these data structures enables reusing them beyond the scope of a single program run while avoiding repetitive raw data ingestion overheads. Memory-mapped I/O enables persisting in-memory data structures without data serialization and deserialization overheads. However, memory-mapped I/O lacks the key feature of persisting consistent snapshots of these data structures for incremental ingestion and processing. The obstacles to efficient virtual memory snapshots using memory-mapped I/O include background writebacks outside the application’s control, and the significantly high storage footprint of such snapshots. To address these limitations, we present Privateer, a memory and storage management tool that enables storage-efficient virtual memory snapshotting while also optimizing snapshot I/O performance. Here, we integrated Privateer into Metall, a state-of-the-art persistent memory allocator for C++, and the Lightning Memory-Mapped Database (LMDB), a widely-used key-value datastore in data analytics and machine learning. Privateer optimized application performance by 1.22× when storing data structure snapshots to node-local storage, and up to 16.7× when storing snapshots to a parallel file system. Privateer also optimizes storage efficiency of incremental data structure snapshots by up to 11× using data deduplication and compression.

Computer science↗
NASA NTRS1988

Observed variability of the solar luminosity

Data on solar luminosity variations are compiled in graphs and analyzed, with a focus on the SMM satellite bolometric measurements and other observations made since 1983. Consideration is given to total irradiance measurements in integrated light; emission-line variability in the visible, UV, and IR; mechanisms contributing to the solar variability (sunspots, faculae, rotation, oscillations, granulation, and active networks); the role of the solar magnetic cycle; and other observable parameters indicating variability. Also discussed are the effects of solar variations on the earth climate and the implications of the observations for theoretical models of convective-envelope processes (thermal diffusion near the surface and magnetic pressure in the interior).

Hudson, H. S.↗
DOE OSTI2021

Nickel(ii) complexes based on dithiolate–polyamine binary ligand systems: crystal structures, hirshfeld surface analysis, theoretical study, and catalytic activity study on photocatalytic hydrogen generation

To ascertain the influence of binary ligand systems [1,1-dicyanoethylene-2,2-dithiolate (i-mnt -2 ) and polyamine {tetraen = tris(2-aminoethyl)amine, tren = diethylene triamine and opda = o-phenylenediamine}] on the coordination modes of the Ni(II) metal center and resulting supramolecular architectures, a series of nickel(II) thiolate complexes [Ni(tetraen)(i-mnt)](DMSO) (1), [Ni 2 (tren) 2 (i-mnt) 2 ] (2), and [Ni 2 (i-mnt) 2 (opda) 2 ] n (3) have been synthesized in high yield in one step in water and structurally characterized by single crystal X-ray crystallography and spectroscopic techniques. X-ray diffraction studies disclose the diverse i-mnt -2 coordination to the Ni +2 center in the presence of active polyamine ligands, forming a slightly distorted octahedral geometry (NiN 4 S 2 ) in 1, square planar (NiS 4 ) and distorted octahedral geometries (NiN 6 ) in the bimetallic co-crystallized aggregate of cationic [Ni(tren) 2 ] +2 and anionic [Ni(i-mnt) 2 ] -2 in 2, and a one dimensional (1D) polymeric chain along the [100] axis in 3, having consecutive square planar (NiS 4 ) and octahedral (NiN 6 ) coordination kernels. The N–H···O, N–H···S, N–H···N, N–H···S, N–H···N, and N–H···O type hydrogen bonds stabilize the supramolecular assemblies in 1, 2, and 3 respectively imparting interesting graph-set-motifs. The molecular Hirshfeld surface analyses (HS) and 2D fingerprint plots were utilized for decoding all types of non-covalent contacts in the crystal networks. Atomic HS analysis of the Ni +2 centers reveals significant Ni–N metal–ligand interactions compared to Ni–S interactions. We have also studied the unorthodox interactions observed in the solid state structures of 1–3 by QTAIM and NBO analyses. Moreover, all the complexes proved to be highly active water reduction co-catalysts (WRC) in a photo-catalytic hydrogen evolution process involving iridium photosensitizers, wherein 2 and 3 having a square planar arrangement around the nickel center(s) – were found to be the most active ones, achieving 1000 and 1119 turnover numbers (TON), respectively.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2016

Hyperspectral target detection using manifold learning and multiple target spectra

Imagery collected from satellites and airborne platforms provides an important tool for remotely analyzing the content of a scene. In particular, the ability to remotely detect a specific material within a scene is of critical importance in nonproliferation and other applications. The sensor systems that process hyperspectral images collect the high-dimensional spectral information necessary to perform these detection analyses. For a d-dimensional hyperspectral image, however, where d is the number of spectral bands, it is common for the data to inherently occupy an m-dimensional space with m << d. In the remote sensing community, this has led to recent interest in the use of manifold learning, which seeks to characterize the embedded lower-dimensional, nonlinear manifold that the data discretely approximate. The research presented in this paper focuses on a graph theory and manifold learning approach to target detection, using an adaptive version of locally linear embedding that is biased to separate target pixels from background pixels. Finally, this approach incorporates multiple target signatures for a particular material, accounting for the spectral variability that is often present within a solid material of interest.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Scoreboard

Emerging HPC machines have given rise to enhanced compute power that far outstrips the machine's ability to save large scale results for post-processing. To combat this, in situ data analysis techniques are slowly being adopted. With in situ data management favoring workflows composed of multiple simulations and analyses connected in transit on heterogeneous machines, scientists and engineers need a tool that enables them to create data extracts, visualizations, and interactively monitor and steer their simulations. Scoreboard Phase II is a next generation analysis software that supports composite in transit workflows on heterogeneous architectures and restores interactivity to in situ data analysis through simulation monitoring and computational steering. Scoreboard provides a simulation dashboard with graphs of metrics over time, controls for setting custom simulation steering parameters, controls for managing the set of data extracts being produced in the simulation, as well as the ability to explore data extracts, all from a web browser. Realizing the vision outlined in this project required research into making a system that integrates end to end from simulations all the way to the user. In situ tools generally suffer from complexity and excessive software dependencies. Scoreboard, by contrast, is easy to build and integrate into simulation codes and it provides first class FORTRAN support. The Scoreboard library is capable of in situ and in transit data analysis that can produce data extracts commonly needed for Computational Fluid Dynamics (CFD) analysis. Simulations can transparently stage data in transit to a Scoreboard Endpoint program, which can accept their data and produce the requested data extracts. This lets simulations return to their work while the Endpoint works on the analysis. Efficiently staging the data at scale was a topic of this research. Scoreboard provides the means to let the user manage data extracts and monitor/steer many simulations from a web browser. This area of the research focused on discovery of in transit network components to expose and control their steering parameters within an interactive browser-based user interface that includes: system topology, gathered metrics, notifications, dynamically-generated steering controls, and exploration of visualization data products.

Whitlock, BradJoseph [Intelligent Light] (00000001↗
NASA NTRS1983

High sensitivity high resolution double scatter 1-30 MeV gamma ray telescope

The design and performance of the double-scatter 1-30-MeV gamma-ray telescope developed at the University of California for balloon-borne observations are reported and illustrated with drawings, diagrams, and graphs. The telescope comprises an upper array of 16 6.03 x 6.03 x 100-cm plastic scintillator bars viewed by photomultiplier tubes (PMTs) at both ends, a lower array (at 100-cm distance) of 16 4.83 x 4.83 x 100-cm NaI(Tl) bars viewed at both ends by PMTs, and four 122 x 122 x 0.64-cm plastic sheets viewed by four PMTs each as charged-particle anticoincidence shielding. The operating principle of the telescope, the tests conducted on the NaI(Tl) bars, the on-board data-processing electronics, and the overall performance are characterized. Performance parameters include area 1 sq m, field of view pi sr, 4-sigma sensitivity above 1 MeV 0.00004 gamma/sq cm sec over 25 h, angular resolution about 3.5 deg FWHM, and energy resolution 10 percent at 1 MeV and 6 percent at 6 MeV.

Zych, A. D.↗
DOE OSTI2026

iDDS: intelligent distributed dispatch and scheduling for workflow orchestration

The intelligent distributed dispatch and scheduling (iDDS) service is a versatile workflow orchestration system designed for large-scale, distributed scientific computing. iDDS extends traditional workload and data management by integrating data-aware execution, conditional logic, and programmable workflows, enabling automation of complex and dynamic processing pipelines. Originally developed for the ATLAS experiment at the large hadron collider, iDDS has evolved into an experiment-agnostic platform that supports both template-driven workflows and a Function-as-a-Task model for Python-based orchestration. This paper presents the architecture and core components of iDDS, highlighting its scalability, modular message-driven design, and integration with systems such as PanDA and Rucio. We demonstrate its versatility through real-world use cases: fine-grained tape resource optimization for ATLAS, orchestration of large Directed Acyclic Graph (DAG) workflows for the Rubin Observatory, distributed hyperparameter optimization for machine learning applications, active learning for physics analyses, and AI-assisted detector design at the electron–ion collider. By unifying workload scheduling, data movement, and adaptive decision-making, iDDS reduces operational overhead and enables reproducible, high-throughput workflows across heterogeneous infrastructures. We conclude with current challenges and future directions, including interactive, cloud-native, and serverless workflow support.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Quantitative Risk Management - Risk Informed Decision Making on High-Risk Waste Remediation Projects - 19022

For every $1 billion invested in projects, $97 million is wasted due to poor project performance. That means 9.7% of money spent on projects is wasted due to poor risk and uncertainty management. High-risk projects, similar to those in waste remediation, have the most to lose. A solid quantitative risk management program is one of the keys to increasing the likelihood that a project will succeed. A quantitative risk program solves two major problems that challenge the effective management of risk and uncertainty: 1. Risks arise at all levels of an organization; however, not all organizational levels have an effective way to identify and resolve them. This can lead to risks that, even if identified, are not reported. 2. Decision-makers are often forced to operate solely on intuition when implementing mitigation strategies, not fully understanding the effects that certain decisions have on the program's cost and schedule. A quantitative risk program uses a risk register and builds it into the project/program integrated schedule to assess the impact of these risks, typically at the 80% confidence contingency level for specific milestones and/or project completion. Implementing a quantitative risk program drives a forward-thinking, proactive mentality to problem solving into all levels of the organization. A healthy risk program for regular reporting and communication ensures that all members of an organization are thinking about risks and using their communication channels to resolve them. It gives a method of communication and incentive for all levels in the organization and helps to ensure that risks will be reported and resolved. Pacific Northwest National Laboratory (PNNL) has developed tools and methods that provide a level of realism for analyzing different scenarios and estimating contingency that does not exist in the standard industry risk tool kit. These tools and methods provide decision-makers with actionable, quantitatively based information to guide their decision making. The first tool is a probabilistic branching risk model for mitigation strategies. Current quantitative risk analysis tools in the industry simulate two cases: fully mitigated and un-mitigated. In the fully mitigated case, the user assumes that every planned mitigation activity is successful. In the un-mitigated case, the user assumes that none are successful. The truth lies between these two extremes. The PNNL team has developed a probabilistic risk model for mitigation scenarios that includes a probability that a mitigation activity will succeed and a probability that a mitigation action will be funded. These probabilities are elicited from subject matter experts and ensure that the contingency calculations at the specified confidence are approaching reality. The second tool developed at PNNL is the Risk Box, which is a tool that integrates with different scenarios in the risk register. For example, say the user is faced with two different and mutually exclusive mitigation actions to reduce a risk. The Risk Box methodology performs the contingency calculations and then displays them on a time-phased graph that shows how each scenario reduces the risk contingency needed on the project/program. This graph then provides quantitative information to show decision-makers which mitigation action provides the greatest reduction (cost and schedule) in risk contingency for the program. Implementing these and other tools within a structured risk program increases the likelihood that a project will come in on schedule and budget, ensures that all personnel are proactively thinking about problems and resolutions, creates communication channels for all members to submit and resolve potential problems, produces a prioritized list of risk rankings to focus leadership on issues that may cause the biggest problems, and implements processes that drive updates to the risk program as the risk profile changes. PNNL has been developing these methods and using them on projects for over 20 years. They have been successfully applied on waste management projects, high-tech research and development projects, and construction projects, and they continue to be used on a number of U.S. Department of Energy projects today. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2020

DeepClone: Lightweight State Replication of Deep Learning Models for Data Parallel Training

Training modern deep neural network (DNN) models involves complex workflows triggered by model exploration, sensitivity analysis, explainability, etc. A key primitive in this context is the ability to clone a model training instance, i.e. “fork” the training process in a potentially different direction, which enables comparisons of different evolution paths using variations of training data and model parameters. However, in a quest improve the training throughput, a mix of data parallel, model parallel, pipeline parallel and layer-wise parallel approaches are making the problem of cloning highly complex. In this paper, we explore the problem of efficient cloning under such circumstances. To this end, we leverage several properties of data-parallel training and layer-wise parallelism to design DeepClone, a cloning approach based on augmenting the execution graph to gain direct access to tensors, which are then sharded and reconstructed asynchronously in order to minimize runtime overhead, standby duration, readiness duration. Compared with state-of-art approaches, DeepClone shows orders of magnitude improvement for several classes of DNN models.

data-parallel training↗
DOE OSTI2023

Lorentz group equivariant autoencoders

Abstract There has been significant work recently in developing machine learning (ML) models in high energy physics (HEP) for tasks such as classification, simulation, and anomaly detection. Often these models are adapted from those designed for datasets in computer vision or natural language processing, which lack inductive biases suited to HEP data, such as equivariance to its inherent symmetries. Such biases have been shown to make models more performant and interpretable, and reduce the amount of training data needed. To that end, we develop the Lorentz group autoencoder (LGAE), an autoencoder model equivariant with respect to the proper, orthochronous Lorentz group $$\textrm{SO}^+(3,1)$$ SO + ( 3 , 1 ) , with a latent space living in the representations of the group. We present our architecture and several experimental results on jets at the LHC and find it outperforms graph and convolutional neural network baseline models on several compression, reconstruction, and anomaly detection metrics. We also demonstrate the advantage of such an equivariant model in analyzing the latent space of the autoencoder, which can improve the explainability of potential anomalies discovered by such ML models.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2026

Intelligent experiments through real-time AI: Fast Data Processing and Autonomous Detector Control for sPHENIX and future EIC detectors (Phase-I)

With an ever increasing demand for high precision data from modern detectors for discovery science and precision measurements, all major high energy nuclear and particle experiments, current and future, are facing the challenge on how to deal with the large volume of raw data generated from sophisticated state-of-the-art detectors in high rate collisions. These goals need to be balanced with available hardware and cost limits on DAQ (Data AcQuisition system) bandwidth and offline computing resources to capture, store and process the signal events. Two prototypical examples are the upcoming sPHENIX experiment, the DOE next generation heavy ion physics experiment at the Relativistic Heavy Ion Collider at BNL, and the future EIC experiments that are planned to be online circa 2030.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗