Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “distributed algorithms”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 523 records · Page 29
All sourcesNASA NTRSDOE OSTI
DOE OSTI2013

Primordial power spectrum: a complete analysis with the WMAP nine-year data

We have improved further the error sensitive Richardson-Lucy deconvolution algorithm making it applicable directly on the un-binned measured angular power spectrum of Cosmic Microwave Background observations to reconstruct the form of the primordial power spectrum. This improvement makes the application of the method significantly more straight forward by removing some intermediate stages of analysis allowing a reconstruction of the primordial spectrum with higher efficiency and precision and with lower computational expenses. Applying the modified algorithm we fit the WMAP 9 year data using the optimized reconstructed form of the primordial spectrum with more than 300 improvement in χ{sup 2}{sub eff} with respect to the best fit power-law. This is clearly beyond the reach of other alternative approaches and reflects the efficiency of the proposed method in the reconstruction process and allow us to look for any possible feature in the primordial spectrum projected in the CMB data. Though the proposed method allow us to look at various possibilities for the form of the primordial spectrum, all having good fit to the data, proper error-analysis is needed to test for consistency of theoretical models since, along with possible physical artefacts, most of the features in the reconstructed spectrum might be arising from fitting noises in the CMB data. Reconstructed error-band for the form of the primordial spectrum using many realizations of the data, all bootstrapped and based on WMAP 9 year data, shows proper consistency of power-law form of the primordial spectrum with the WMAP 9 data at all wave numbers. Including WMAP polarization data in to the analysis have not improved much our results due to its low quality but we expect Planck data will allow us to make a full analysis on CMB observations on both temperature and polarization separately and in combination.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

Non-parametric projections of the net-income distribution for all U.S. states for the Shared Socioeconomic Pathways

Abstract Income distributions are a growing area of interest in the examination of equity impacts brought on by climate change and its responses. Such impacts are especially important at subnational levels, but projections of income distributions at these levels are scarce. Here, we project U.S. state-level income distributions for the Shared Socioeconomic Pathways (SSPs). We apply a non-parametric approach, specifically a recently developed principal components algorithm to generate net income distributions for deciles across 50 U.S. states and the District of Columbia. We produce these projections to 2100 for three SSP scenarios in combination with varying projections of GDP per capita to represent a wide range of possible futures and uncertainties. In the generation of these scenarios, we also generated tax adjusted historical deciles by U.S. states, which we used for validating model performance. Our method thus produces income distributions by decile for each state, reflecting the variability in state income, population, and tax regimes. Our net income projections by decile can be used in both emissions- and impact-related research to understand distributional effects at various income levels and identify economically vulnerable populations.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2023

Robust sampling for weak lensing and clustering analyses with the Dark Energy Survey

Recent cosmological analyses rely on the ability to accurately sample from high-dimensional posterior distributions. A variety of algorithms have been applied in the field, but justification of the particular sampler choice and settings is often lacking. Here, we investigate three such samplers to motivate and validate the algorithm and settings used for the Dark Energy Survey (DES) analyses of the first 3 yr (Y3) of data from combined measurements of weak lensing and galaxy clustering. We employ the full DES Year 1 likelihood alongside a much faster approximate likelihood, which enables us to assess the outcomes from each sampler choice and demonstrate the robustness of our full results. We find that the ellipsoidal nested sampling algorithm multinest reports inconsistent estimates of the Bayesian evidence and somewhat narrower parameter credible intervals than the sliced nested sampling implemented in polychord. We compare the findings from multinest and polychord with parameter inference from the Metropolis–Hastings algorithm, finding good agreement. We determine that polychord provides a good balance of speed and robustness for posterior and evidence estimation, and recommend different settings for testing purposes and final chains for analyses with DES Y3 data. Our methodology can readily be reproduced to obtain suitable sampler settings for future surveys.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2023

Fast and efficient identification of anomalous galaxy spectra with neural density estimation

ABSTRACT Current large-scale astrophysical experiments produce unprecedented amounts of rich and diverse data. This creates a growing need for fast and flexible automated data inspection methods. Deep learning algorithms can capture and pick up subtle variations in rich data sets and are fast to apply once trained. Here, we study the applicability of an unsupervised and probabilistic deep learning framework, the probabilistic auto-encoder, to the detection of peculiar objects in galaxy spectra from the SDSS survey. Different to supervised algorithms, this algorithm is not trained to detect a specific feature or type of anomaly, instead it learns the complex and diverse distribution of galaxy spectra from training data and identifies outliers with respect to the learned distribution. We find that the algorithm assigns consistently lower probabilities (higher anomaly score) to spectra that exhibit unusual features. For example, the majority of outliers among quiescent galaxies are E+A galaxies, whose spectra combine features from old and young stellar population. Other identified outliers include LINERs, supernovae, and overlapping objects. Conditional modelling further allows us to incorporate additional information. Namely, we evaluate the probability of an object being anomalous given a certain spectral class, but other information such as metrics of data quality or estimated redshift could be incorporated as well. We make our code publicly available.

Böhm, Vanessa↗
DOE OSTI2016

Fault Tolerant Frequent Pattern Mining

FP-Growth algorithm is a Frequent Pattern Mining (FPM) algorithm that has been extensively used to study correlations and patterns in large scale datasets. While several researchers have designed distributed memory FP-Growth algorithms, it is pivotal to consider fault tolerant FP-Growth, which can address the increasing fault rates in large scale systems. In this work, we propose a novel parallel, algorithm-level fault-tolerant FP-Growth algorithm. We leverage algorithmic properties and MPI advanced features to guarantee an O(1) space complexity, achieved by using the dataset memory space itself for checkpointing. We also propose a recovery algorithm that can use in-memory and disk-based checkpointing, though in many cases the recovery can be completed without any disk access, and incurring no memory overhead for checkpointing. We evaluate our FT algorithm on a large scale InfiniBand cluster with several large datasets using up to 2K cores. Our evaluation demonstrates excellent efficiency for checkpointing and recovery in comparison to the disk-based approach. We have also observed 20x average speed-up in comparison to Spark, establishing that a well designed algorithm can easily outperform a solution based on a general fault-tolerant programming model.

Shohdy, Sameh↗
DOE OSTI2015

A Scenario Generation Method for Wind Power Ramp Events Forecasting

Wind power ramp events (WPREs) have received increasing attention in recent years due to their significant impact on the reliability of power grid operations. In this paper, a novel WPRE forecasting method is proposed which is able to estimate the probability distributions of three important properties of the WPREs. To do so, a neural network (NN) is first proposed to model the wind power generation (WPG) as a stochastic process so that a number of scenarios of the future WPG can be generated (or predicted). Each possible scenario of the future WPG generated in this manner contains the ramping information, and the distributions of the designated WPRE properties can be stochastically derived based on the possible scenarios. Actual data from a wind power plant in the Bonneville Power Administration (BPA) was selected for testing the proposed ramp forecasting method. Results showed that the proposed method effectively forecasted the probability of ramp events.

17 WIND ENERGY↗
DOE OSTI2017

Dispatch Control with PEV Charging and Renewables for Multiplayer Game Application

This paper presents a demand response model for a hypothetical microgrid that integrates renewable resources and plug-in electric vehicle (PEV) charging systems. It is assumed that the microgrid has black start capability and that external generation is available for purchase while grid connected to satisfy additional demand. The microgrid is developed such that in addition to renewable, non-dispatchable generation from solar, wind and run of the river hydroelectric resources, local dispatchable generation is available in the form of small hydroelectric and moderately sized gas and coal fired facilities. To accurately model demand, the load model is separated into independent residential, commercial, industrial, and PEV charging systems. These are dispatched and committed based on a mixed integer linear program developed to minimize the cost of generation and load shedding while satisfying constraints associated with line limits, conservation of energy, and ramp rates of the generation units. The model extends a research tool to longer time frames intended for policy setting and educational environments and provides a realistic and intuitive understanding of beneficial and challenging aspects of electrification of vehicles combined with integration of green electricity production.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2014

Design and Stability of Load-Side Primary Frequency Control in Power Systems

We present a systematic method to design ubiquitous continuous fast-acting distributed load control for primary frequency regulation in power networks, by formulating an optimal load control (OLC) problem where the objective is to minimize the aggregate cost of tracking an operating point subject to power balance over the network. We prove that the swing dynamics and the branch power flows, coupled with frequency-based load control, serve as a distributed primal-dual algorithm to solve OLC. We establish the global asymptotic stability of a multimachine network under such type of load-side primary frequency control. These results imply that the local frequency deviations on each bus convey exactly the right information about the global power imbalance for the loads to make individual decisions that turn out to be globally optimal. Simulations confirm that the proposed algorithm can rebalance power and resynchronize bus frequencies after a disturbance with significantly improved transient performance.

Zhao, CH↗
DOE OSTI2022

MetallData

MetallData is an HPC platform for interactive data science applications at HPC-scales. It provides an ecosystem for persistent distributed data structures, including algorithms, interactivity and storage.

Pearce, RogerA↗
DOE OSTI2022

Fed-DeepONet: Stochastic Gradient-Based Federated Training of Deep Operator Networks

The Deep Operator Network (DeepONet) framework is a different class of neural network architecture that one trains to learn nonlinear operators, i.e., mappings between infinite-dimensional spaces. Traditionally, DeepONets are trained using a centralized strategy that requires transferring the training data to a centralized location. Such a strategy, however, limits our ability to secure data privacy or use high-performance distributed/parallel computing platforms. To alleviate such limitations, in this paper, we study the federated training of DeepONets for the first time. That is, we develop a framework, which we refer to as Fed-DeepONet, that allows multiple clients to train DeepONets collaboratively under the coordination of a centralized server. To achieve Fed-DeepONets, we propose an efficient stochastic gradient-based algorithm that enables the distributed optimization of the DeepONet parameters by averaging first-order estimates of the DeepONet loss gradient. Then, to accelerate the training convergence of Fed-DeepONets, we propose a moment-enhanced (i.e., adaptive) stochastic gradient-based strategy. Finally, we verify the performance of Fed-DeepONet by learning, for different configurations of the number of clients and fractions of available clients, (i) the solution operator of a gravity pendulum and (ii) the dynamic response of a parametric library of pendulums.

Moya, Christian↗
DOE OSTI2020

Protein Conformational States—A First Principles Bayesian Method

Automated identification of protein conformational states from simulation of an ensemble of structures is a hard problem because it requires teaching a computer to recognize shapes. We adapt the naïve Bayes classifier from the machine learning community for use on atom-to-atom pairwise contacts. The result is an unsupervised learning algorithm that samples a ‘distribution’ over potential classification schemes. We apply the classifier to a series of test structures and one real protein, showing that it identifies the conformational transition with >95% accuracy in most cases. A nontrivial feature of our adaptation is a new connection to information entropy that allows us to vary the level of structural detail without spoiling the categorization. This is confirmed by comparing results as the number of atoms and time-samples are varied over 1.5 orders of magnitude. Further, the method’s derivation from Bayesian analysis on the set of inter-atomic contacts makes it easy to understand and extend to more complex cases.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Dark Energy Survey Year 1 Results: Cross-Correlation Redshifts in the DES -- Calibration of the Weak Lensing Source Redshift Distributions

We present the calibration of the Dark Energy Survey Year 1 (DES Y1) weak lensing source galaxy redshift distributions from clustering measurements. By cross-correlating the positions of source galaxies with luminous red galaxies selected by the redMaGiC algorithm we measure the redshift distributions of the source galaxies as placed into different tomographic bins. These measurements constrain any such shifts to an accuracy of $\sim0.02$ and can be computed even when the clustering measurements do not span the full redshift range. The highest-redshift source bin is not constrained by the clustering measurements because of the minimal redshift overlap with the redMaGiC galaxies. We compare our constraints with those obtained from $\texttt{COSMOS}$ 30-band photometry and find that our two very different methods produce consistent constraints.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2014

High-Performance Computation of Distributed-Memory Parallel 3D Voronoi and Delaunay Tessellation

Computing a Voronoi or Delaunay tessellation from a set of points is a core part of the analysis of many simulated and measured datasets: N-body simulations, molecular dynamics codes, and LIDAR point clouds are just a few examples. Such computational geometry methods are common in data analysis and visualization; but as the scale of simulations and observations surpasses billions of particles, the existing serial and shared-memory algorithms no longer suffice. A distributed-memory scalable parallel algorithm is the only feasible approach. The primary contribution of this paper is a new parallel Delaunay and Voronoi tessellation algorithm that automatically determines which neighbor points need to be exchanged among the subdomains of a spatial decomposition. Other contributions include periodic and wall boundary conditions, comparison of our method using two popular serial libraries, and application to numerous science datasets.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Bayesian calibration of terrestrial ecosystem models: A study of advanced Markov chain Monte Carlo methods

Calibration of terrestrial ecosystem models is important but challenging. Bayesian inference implemented by Markov chain Monte Carlo (MCMC) sampling provides a comprehensive framework to estimate model parameters and associated uncertainties using their posterior distributions. The effectiveness and efficiency of the method strongly depend on the MCMC algorithm used. In this study, a Differential Evolution Adaptive Metropolis (DREAM) algorithm was used to estimate posterior distributions of 21 parameters for the data assimilation linked ecosystem carbon (DALEC) model using 14 years of daily net ecosystem exchange data collected at the Harvard Forest Environmental Measurement Site eddy-flux tower. The DREAM is a multi-chain method and uses differential evolution technique for chain movement, allowing it to be efficiently applied to high-dimensional problems, and can reliably estimate heavy-tailed and multimodal distributions that are difficult for single-chain schemes using a Gaussian proposal distribution. The results were evaluated against the popular Adaptive Metropolis (AM) scheme. DREAM indicated that two parameters controlling autumn phenology have multiple modes in their posterior distributions while AM only identified one mode. The calibration of DREAM resulted in a better model fit and predictive performance compared to the AM. DREAM provides means for a good exploration of the posterior distributions of model parameters. Lastly, it reduces the risk of false convergence to a local optimum and potentially improves the predictive performance of the calibrated model.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Exploring the Landscape of Distributed Graph Clustering on Leadership Supercomputers

The rapid growth of large-scale datasets in fields like biology and social networks has driven the need for advanced graph analytics techniques. Community detection, a fundamental task in graph analytics, identifies closely connected groups of nodes within a network, providing valuable insights across various disciplines. This study focuses on two classic community detection methods, the Louvain algorithm and Markov Clustering (MCL), and evaluates the performance of two prominent distributed community detection algorithms: HiPDPL-GPU, our prior implementation, and HipMCL. We conduct experiments on GPU-accelerated heterogeneous HPC systems, Summit and Frontier, to assess their performance under varying conditions. Our objective is to identify the strengths and weaknesses of these algorithms in terms of scalability, and quality of solutions. We evaluate these algorithms on a diverse set of 70+ networks spanning 13 domains, with sizes ranging up to 4.2 billion edges. Our results demonstrate that HiPDPL-GPU consistently outperforms HipMCL, especially for large-scale networks. HiPDPL-GPU achieves significantly faster runtimes (47x to 1439x), higher modularity scores, and improved scalability. These findings highlight HiPDPL-GPU as a promising solution for efficient and effective large-scale graph analytics in diverse application domains, and provide insights into the feasibility of using MCL-based approaches for certain application domains.

Community detection, graph algorithms↗
DOE OSTI2020

Autonomous Intelligent Charging/Discharging of Electric Vehicles using Distributed Multi-Agent ADMM Framework for Grid Ancillary Services

The increasing popularity of Electric Vehicles (EVs) in the distribution grid along with technological advancement in EV electronics such as vehicle to grid (V2G) technique has enabled them to participate in grid ancillary services. To achieve this, the EVs need to establish a contract with third-party aggregators and connect to a charging unit, either residential or commercial. At any time they are connected, the EVs can decide to take part in the ancillary services program offered to them by the aggregators. If agreed, the aggregators will use the EVs as a power source capable of charging/discharging power according to the input signal, and in return, they will be compensated. This inter-temporal nature of charging/discharging is also transforming the traditional optimal power flow (OPF) problem into a dynamic OPF problem. This chapter aims at developing a multi-layer time-dependent optimization algorithm to utilize EV potential and provide ancillary services while maximizing its utilization function. Specifically, in the upper layer, an autonomous distributed ADMM algorithm is developed to optimize the cost for charging/discharging EVs while using them to regulate the voltage at each bus in the distribution grid. The distributed ADMM algorithm is also expanded to the lower layer where the individual EVs active and reactive power is controlled for voltage regulation while maintaining the desired state of the charge of the vehicle at the end of the charging period. Here, the effectiveness and performance improvement of the proposed multi-layer algorithm is illustrated through analytical analysis and simulation results.

Rahman, Towfiq↗
DOE OSTI2020

Autonomous Intelligent Charging/Discharging of Electric Vehicles using Distributed Multi-Agent ADMM Framework for Grid Ancillary Services

The increasing popularity of Electric Vehicles (EVs) in the distribution grid along with technological advancement in EV electronics such as vehicle to grid (V2G) technique has enabled them to participate in grid ancillary services. To achieve this, the EVs need to establish a contract with third-party aggregators and connect to a charging unit, either residential or commercial. At any time they are connected, the EVs can decide to take part in the ancillary services program offered to them by the aggregators. If agreed, the aggregators will use the EVs as a power source capable of charging/discharging power according to the input signal, and in return, they will be compensated. This inter-temporal nature of charging/discharging is also transforming the traditional optimal power flow (OPF) problem into a dynamic OPF problem. This chapter aims at developing a multi-layer time-dependent optimization algorithm to utilize EV potential and provide ancillary services while maximizing its utilization function. Specifically, in the upper layer, an autonomous distributed ADMM algorithm is developed to optimize the cost for charging/discharging EVs while using them to regulate the voltage at each bus in the distribution grid. The distributed ADMM algorithm is also expanded to the lower layer where the individual EVs active and reactive power is controlled for voltage regulation while maintaining the desired state of the charge of the vehicle at the end of the charging period. The effectiveness and performance improvement of the proposed multi-layer algorithm is illustrated through analytical analysis and simulation results.

Rahman, Towfiq↗