Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “distributed algorithm”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 433 records · Page 24
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

IMoFi (Intelligent Model Fidelity): Physics-Based Data-Driven Grid Modeling to Accelerate Accurate PV Integration Updated Accomplishments

This report summarizes the work performed under a project funded by U.S. DOE Solar Energy Technologies Office (SETO), including some updates from the previous report SAND2022-0215, to use grid edge measurements to calibrate distribution system models for improved planning and grid integration of solar PV. Several physics-based data-driven algorithms are developed to identify inaccuracies in models and to bring increased visibility into distribution system planning. This includes phase identification, secondary system topology and parameter estimation, meter-to-transformer pairing, medium-voltage reconfiguration detection, determination of regulator and capacitor settings, PV system detection, PV parameter and setting estimation, PV dynamic models, and improved load modeling. Each of the algorithms is tested using simulation data and demonstrated on real feeders with our utility partners. The final algorithms demonstrate the potential for future planning and operations of the electric power grid to be more automated and data-driven, with more granularity, higher accuracy, and more comprehensive visibility into the system.

14 SOLAR ENERGY↗
DOE OSTI2022

Rain Drop Size Distributions Estimated from NOAA Snow-Level Radar Data

Using NOAA’s S-band High-Power Snow-Level Radar (HPSLR), a technique for estimating the rain drop size distribution (DSD) above the radar is presented. This technique assumes the DSD can be described by a four parameter, generalized gamma distribution (GGD). Using the radar’s measured average Doppler velocity spectrum and a value (assumed, measured, or estimated) of the vertical air motion w, an estimate of the GGD is obtained. Four different methods can be used to obtain w. One method that estimates a mean mass-weighted raindrop diameter D m from the measured reflectivity Z produces realistic DSDs compared to prior literature examples. These estimated DSDs provide evidence that the radar can retrieve the smaller drop sizes constituting the “drizzle” mode part of the DSD. Here, this estimation technique was applied to 19 h of observations from Hankins, North Carolina. Results support the concept that DSDs can be modeled using GGDs with a limited range of parameters. Further work is needed to validate the described technique for estimating DSDs in more varied precipitation types and to verify the vertical air motion estimates.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Situational awareness-enhancing community-level load mapping with opportunistic machine learning

Motivated by present and forthcoming challenges in the adoption and integration of distributed renewable energy, we develop a machine learning (ML) approach that builds short-fuse mappings connecting the occasionally-unobservable true load in one target community with information-rich signals collected from relatively more instrumented reference communities. Our setting is inspired by and tailored to target communities with significant unobservable behind-the-meter solar generation, where true load (a relatively well-behaved quantity of interest to grid operators) is hard to discern during daytime due to insufficient instrumentation and/or privacy reasons, but that can be related to reference communities with low unobservable distributed variable generation or with sufficient instrumentation. The developed mapping, herein realized with Support Vector Machine regression, is built using nighttime data from all communities, when their distributed generation is low or zero. Our ML algorithm opportunistically learns to correlate signals of interest and then is operationally used the next day to shed light into target community load evolution. The mapping is subsequently rebuilt, rolling its short-fuse scope perpetually forward in time. Here, we demonstrate the efficacy of our approach on nine synthetically generated topologies and associated timeseries stemming from real-world data, on which we observe cumulative error performance that yields lower than 10% and 15% daily-averaged mean absolute percentage errors in target community load estimation on more than about 75% and 90% of days, respectively, in multiple yearly evaluations that shed light on long-term performance also under seasonal and one-off effects. The proposed ML-powered methodology can offer grid operators much-improved visibility into a previously obscure space and can also serve as an additional source of information in broader, multi-modal solar disaggregation solutions.

14 SOLAR ENERGY↗
DOE OSTI2024

Joint Management and Optimization of Residential Natural Gas and Electricity Distribution Networks Coupled via Fuel Cells

The interesting properties of natural gas as well as the growing electric power demand worldwide have led to increasing attention to natural-gas-based distributed generation applications in electric distribution systems. This paper goes over the interdependency between a residential natural gas network and an electric distribution network that are coupled via fuel cells. The modeling of the gas network is introduced first, and then the algorithm for gas flow study is presented. The optimal placement and sizing of fuel cell based distributed generation systems are formulated to minimize the losses in both the gas and electric distribution networks, subject to their model constraints. In addition to this, in order to capture the probabilistic nature of the optimization problem under study, the K-means clustering algorithm is applied to the gas and electricity demands to determine hourly load states and their corresponding probabilities. Furthermore, simulation studies are carried out on an integrated system consisting of the IEEE 69-bus distribution feeder and a radial 27-node natural gas network to verify the developed optimization model and the proposed method.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2020

Distributed Berkeley Efficient Long-Read to Long-Read Aligner and Overlapper (DiBELLA) v1.0.0

We present a parallel algorithm and scalable implementation for genome analysis, specifically the problem of finding overlaps and alignments for data from "third generation" long read sequencers. While long sequences of DNA offer enormous advantages for biological analysis and insight, current long read sequencing instruments have high error rates and therefore require different approaches to analysis than their short read counterparts. Our work focuses on an efficient distributed-memory parallelization of an accurate single-node algorithm for overlapping and aligning long reads. We achieve scalability of this irregular algorithm by addressing the competing issues of increasing parallelism, minimizing communication, constraining the memory footprint, and ensuring good load balance. The resulting application, DiBELLA, is the first distributed memory overlapper and aligner specifically designed for long reads and parallel scalability.

Ellis, Marquita↗
DOE OSTI2020

Distribution Feeder-Scale Fast Frequency Response via Optimal Coordination of Net-load Resources Part II: Large-Scale Demonstration

This work is the second of a two-part series in which we develop and experimentally demonstrate a hierarchical control solution for optimally coordinating thousands of deferrable loads and distributed energy resources (DERs) to provide fast frequency response (FFR) from an entire distribution feeder. In Part I, we developed and proved practical algorithms for fast, cost-based optimal dispatch and for determining the optimal amount of headroom to operate solar inverters with to support FFR dispatch while minimizing opportunity cost. Simulation results in Part I demonstrated the advantages of the hierarchical dispatch approach in being able to maintain fast solution times needed for FFR even when the problem size increases. In Part II, we implement the algorithms developed in Part I in a novel, large-scale power hardware-in-the-loop experiment including embedded controllers and more than 100 powered appliance loads and DER connected to a simulated real-world distribution system with more than 10,000 controlled devices. Experimental results from multiple scenarios confirm that the optimal FFR dispatch approach scales well and can optimally coordinate more than 10,000 net-load resources across a distribution network while achieving hardware response times within 500 ms, which is not possible using state-of-the-art optimal coordination approaches.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2024

Efficient First-Order Algorithms for Large-Scale, Non-Smooth Maximum Entropy Models with Application to Wildfire Science

Maximum entropy (MaxEnt) models are a class of statistical models that use the maximum entropy principle to estimate probability distributions from data. Due to the size of modern data sets, MaxEnt models need efficient optimization algorithms to scale well for big data applications. State-of-the-art algorithms for MaxEnt models, however, were not originally designed to handle big data sets; these algorithms either rely on technical devices that may yield unreliable numerical results, scale poorly, or require smoothness assumptions that many practical MaxEnt models lack. In this paper, we present novel optimization algorithms that overcome the shortcomings of state-of-the-art algorithms for training large-scale, non-smooth MaxEnt models. Our proposed first-order algorithms leverage the Kullback–Leibler divergence to train large-scale and non-smooth MaxEnt models efficiently. For MaxEnt models with discrete probability distribution of n elements built from samples, each containing m features, the stepsize parameter estimation and iterations in our algorithms scale on the order of O(mn) operations and can be trivially parallelized. Moreover, the strong ℓ1 convexity of the Kullback–Leibler divergence allows for larger stepsize parameters, thereby speeding up the convergence rate of our algorithms. To illustrate the efficiency of our novel algorithms, we consider the problem of estimating probabilities of fire occurrences as a function of ecological features in the Western US MTBS-Interagency wildfire data set. Our numerical results show that our algorithms outperform the state of the art by one order of magnitude and yield results that agree with physical models of wildfire occurrence and previous statistical analyses of wildfire drivers.

Physics↗
DOE OSTI2025

Intrepid MCMC: Metropolis-Hastings with exploration

In engineering examples, one often encounters the need to sample from unnormalized distributions with complex shapes that may also be implicitly defined through a physical or numerical simulation model, making it computationally expensive to evaluate the associated density function. For such cases, MCMC has proven to be an invaluable tool. Random-walk Metropolis Methods (also known as Metropolis-Hastings (MH)), in particular, are highly popular for their simplicity, flexibility, and ease of implementation. However, most MH algorithms suffer from significant limitations when attempting to sample from distributions with multiple modes (particularly disconnected ones). Here, in this paper, we present Intrepid MCMC - a novel MH scheme that utilizes a simple coordinate transformation to significantly improve the mode-finding ability and convergence rate to the target distribution of random-walk Markov chains while retaining most of the simplicity of the vanilla MH paradigm. Through multiple examples, we showcase the improvement in the performance of Intrepid MCMC over vanilla MH for a wide variety of target distribution shapes. We also provide an analysis of the mixing behavior of the Intrepid Markov chain, as well as the efficiency of our algorithm for increasing dimensions. A thorough discussion is presented on the practical implementation of the Intrepid MCMC algorithm. Finally, its utility is highlighted through a Bayesian parameter inference problem for a two-degree-of-freedom oscillator under free vibration.

97 - MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Structural evolution of two-dimensional silicates using a “bond-switching” algorithm

Silicates are the most abundant materials in the earth's crust. In recent years, two-dimensional (2D) versions of them grown on metal supports (known as bilayer silicates) have allowed their study in detail down to the atomic scale. These structures are self-containing. Furthermore, they are not covalently bound to the metal support but interact with it through van der Waals forces. Like their three-dimensional counterparts, the 2D-silicates can form both crystalline and vitreous structures. Furthermore, the interconversion between vitreous to crystalline structures has been experimentally observed at the nanoscale. While theoretical work has been carried out to try to understand these transformations, a limitation for ab initio methods, and even molecular dynamics methods, is the computational cost of studying large systems and long timescales. In this work, we present a simple and computationally inexpensive approach, that can be used to represent the evolution of bilayer silicates using a bond-switching algorithm. This approach allows reaching equilibrium ring size distributions as a function of a parameter that can be related to the ratio between temperature and the energy required for the bond-switching event. The ring size distributions are compared to experimental data available in the literature.

77 NANOSCIENCE AND NANOTECHNOLOGY↗
DOE OSTI2021

A fast particle-mesh simulation of non-linear cosmological structure formation with massive neutrinos

Quasi-N-body simulations, such as FastPM, provide a fast way to simulate cosmological structure formation, but have yet to adequately include the effects of massive neutrinos. In this work, we present a method to include neutrino particles in FastPM, enabling computation of the CDM and total matter power spectra to percent-level accuracy in the non-linear regime. The CDM-neutrino cross-power can also be computed at a sufficient accuracy to constrain cosmological observables. To avoid the shot noise that typically plagues neutrino particle simulations, we employ a quasi-random algorithm to sample the relevant Fermi-Dirac distribution when setting the initial neutrino thermal velocities. We additionally develop an effective distribution function to describe a set of non-degenerate neutrinos as a single particle to speed up non-degenerate simulations. The simulation is accurate for the full range of physical interest, M ν ≲ 0.6eV, and applicable to redshifts z ≲ 2. Such accuracy can be achieved by initializing particles with the two-fluid approximation transfer functions (using the REPS package). Convergence can be reached in ~ 25 steps, with a starting redshift of z=99. Probing progressively smaller scales only requires an increase in the number of CDM particles being simulated, while the number of neutrino particles can remain fixed at a value less than or similar to the number of CDM particles. In turn, the percentage increase in runtime-per-step due to neutrino particles is between ~ 5-20% for runs with 1024 3 CDM particles, and decreases as the number of CDM particles is increased. The code has been made publicly available, providing an invaluable resource to produce fast predictions for cosmological surveys and studying reconstruction.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

‘Flux+Mutability’: a conditional generative approach to one-class classification and anomaly detection

Abstract Anomaly Detection is becoming increasingly popular within the experimental physics community. At experiments such as the Large Hadron Collider, anomaly detection is growing in interest for finding new physics beyond the Standard Model. This paper details the implementation of a novel Machine Learning architecture, called Flux+Mutability, which combines cutting-edge conditional generative models with clustering algorithms. In the ‘flux’ stage we learn the distribution of a reference class. The ‘mutability’ stage at inference addresses if data significantly deviates from the reference class. We demonstrate the validity of our approach and its connection to multiple problems spanning from one-class classification to anomaly detection. In particular, we apply our method to the isolation of neutral showers in an electromagnetic calorimeter and show its performance in detecting anomalous dijets events from standard QCD background. This approach limits assumptions on the reference sample and remains agnostic to the complementary class of objects of a given problem. We describe the possibility of dynamically generating a reference population and defining selection criteria via quantile cuts. Remarkably this flexible architecture can be deployed for a wide range of problems, and applications like multi-class classification or data quality control are left for further exploration.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2021

Non-Gaussian estimates of tensions in cosmological parameters

We discuss how to efficiently and reliably estimate the level of agreement and disagreement on parameter determinations from different experiments, fully taking into account non-Gaussianities in the parameter posteriors. We develop two families of scalable algorithms that allow us to perform this type of calculations in increasing number of dimensions and for different levels of tensions. One family of algorithms rely on kernel density estimates of posterior distributions while the other relies on machine learning modeling of the posterior distribution with normalizing flows. We showcase their effectiveness and accuracy with a set of benchmark examples and find both methods agree with each other and the true tension within 0.5σ in difficult cases and generally to 0.2σ or better. This allows us to study the level of internal agreement between different measurements of the clustering of cosmological structures from the Dark Energy Survey and their agreement with measurements of the Cosmic Microwave Background from the Planck satellite.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2023

Feeder Power Disaggregation: A Data-Efficient Matrix Completion Approach

This paper presents a data-driven algorithm for the feeder power disaggregation problem in distribution systems. Leveraging spatio-temporal power patterns in residential homes, residential power is discomposed into three components: sparse-switching loads, periodic loads, and photovoltaic (PV) generation, which are characterized through the design of two sparse matrices and a low-rank matrix. The matrix completion process is data-efficient because of the matrix sparsity and low rankness, along with the use of power system models. The proposed approach is tested using real-world residential data set on a 33-bus distribution system, demonstrating accurate power disaggregation with efficient matrix completion.

distribution system↗
DOE OSTI2023

Parallel Memory-Independent Communication Bounds for SYRK

In this paper, we focus on the parallel communication cost of multiplying a matrix with its transpose, known as a symmetric rank-k update (SYRK). SYRK requires half the computation of general matrix multiplication because of the symmetry of the output matrix. Recent work (Beaumont et al., SPAA '22) has demonstrated that the sequential I/O complexity of SYRK is also a constant factor smaller than that of general matrix multiplication. Inspired by this progress, we establish memory-independent parallel communication lower bounds for SYRK with smaller constants than general matrix multiplication, and we show that these constants are tight by presenting communication-optimal algorithms. The crux of the lower bound proof relies on extending a key geometric inequality to symmetric computations and analytically solving a constrained nonlinear optimization problem. Here, the optimal algorithms use a triangular blocking scheme for parallel distribution of the symmetric output matrix and corresponding computation.

Communication costs↗
DOE OSTI2021

Improving probabilistic infectious disease forecasting through coherence

With an estimated $10.4 billion in medical costs and 31.4 million outpatient visits each year, influenza poses a serious burden of disease in the United States. To provide insights and advance warning into the spread of influenza, the U.S. Centers for Disease Control and Prevention (CDC) runs a challenge for forecasting weighted influenza-like illness (wILI) at the national and regional level. Many models produce independent forecasts for each geographical unit, ignoring the constraint that the national wILI is a weighted sum of regional wILI, where the weights correspond to the population size of the region. We propose a novel algorithm that transforms a set of independent forecast distributions to obey this constraint, which we refer to as probabilistically coherent. Enforcing probabilistic coherence led to an increase in forecast skill for 79% of the models we tested over multiple flu seasons, highlighting the importance of respecting the forecasting system’s geographical hierarchy.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

Four Years of Atmospheric Boundary Layer Height Retrievals Using COSMIC-2 Satellite Data

This work aimed to study the atmospheric boundary layer height (ABLH) from COSMIC-2 refractivity data, endeavoring to refine existing ABLH detection algorithms and scrutinize the resulting spatial and seasonal distributions. Through validation analyses involving different ground-based methodologies (involving data from lidar, ceilometer, microwave radiometers, and radiosondes), the optimal ABLH determination relied on identifying the lowest refractivity gradient negative peak with a magnitude at least $τ$% times the minimum refractivity gradient magnitude, where $τ$ is a fitting parameter representing the minimum peak strength relative to the absolute minimum refractivity gradient. Different $τ$ values were derived accounting for the moment of the day (daytime, nighttime, or sunrise/sunset) and the underlying surface (land or sea). Results show discernible relations between ABLH and various features, notably, the land cover and latitude. On average, ABLH is higher over oceans (≈1.5 km), but extreme values (maximums > 2.5 km, and minimums < 1 km) are reached over intertropical lands. Variability is generally subtle over oceans, whereas seasonality and daily evolution are pronounced over continents, with higher ABLHs during daytime and local wintertime (summertime) in intertropical (middle) latitudes.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Feeder Power Disaggregation: A Data-Efficient Matrix Completion Approach: Preprint

This paper presents a data-driven algorithm for the feeder power disaggregation problem in distribution systems. Leveraging spatio-temporal power patterns in residential homes, residential power is discomposed into three components: sparse-switching loads, periodic loads, and photovoltaic generation, using two sparse matrices and a rank-one matrix. The matrix completion process is data-efficient because of the matrix sparsity and low rankness, along with the use of power system models. The proposed approach is tested using real-world residential datasets on a 33-bus distribution system, demonstrating accurate power disaggregation with efficient matrix completion.

distribution system↗