Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Distributed Processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Supervised Learning for Distribution Secondary Systems Modeling: Improving Solar Interconnection Processes

The current interconnection process and hosting capacity analysis for distributed energy resources (DERs), such as photovoltaics (PV) and battery energy storage systems, are based on analyzing grid network constraints (voltage and thermal) using only medium-voltage distribution network models. This is because most utilities do not have secondary low-voltage system models that connect service transformers and residential customers. This is important because in many cases the main impact of interconnecting DERs could occur on the low-voltage distribution systems. This paper proposes a supervised learning method to approximate local secondary models to improve the interconnection process. The proposed supervised learning method includes a decision tree model that predicts the secondary topology and a logistic regression model that predicts conductor types. The case studies demonstrate the benefits of including secondary low-voltage circuits in the interconnection process. We report the proposed modeling methodology is readily scalable and thus can reduce the cost and effort of PV interconnection for the industry and stakeholders.

14 SOLAR ENERGY↗
DOE OSTI2023

State Energy Offices’ Engagement in Electric Distribution Planning to Meet State Policy Goals

NASEO and Berkeley Lab released a new publication on State Energy Offices’ Engagement in Electric Distribution Planning to Meet State Policy Goals. State and Territory Energy Offices develop plans, programs, policies, and projects that have a substantial impact on electric distribution systems. They also can participate in distribution system planning (DSP) processes to help ensure that utilities – consumer- and investor-owned – meet the state’s future energy needs. This paper recognizes the wide spectrum of roles that State Energy Offices can play in DSP processes, including planning for distributed energy resources and grid modernization. It highlights various examples of non-regulatory activities by State Energy Offices including planning, conducting studies, convening stakeholder processes, and implementing programs that inform and contribute to distribution system planning. It also provides examples of State Energy Offices’ engagement in proceedings before their respective public utility commissions. As State Energy Offices face myriad challenges associated with meeting state policy goals, preparing for anticipated rates of distributed energy resource deployment, addressing concerns regarding grid reliability and resilience, and recommending and making long-term investment decisions, the examples provided through this guide can serve as a resource in navigating those challenges.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2023

Cross-Scale Land-Atmosphere Experiment (CSLAEX) (Final Report)

It is now well recognized that land-atmosphere interactions play an important role in both weather and climate. These interactions take place over a wide range of spatial and temporal scales. Most of the current formulations of the momentum, heat and moisture transport laws are informed by observations with a limited temporal or spatial scale, and hence may have limited applicability. The objective of this work is to improve the multi-scale representation of the near-surface heat transfer in the ground and in the atmosphere using a combination of modeling and field observations. Our knowledge and predictive capacity in land-atmosphere coupling is limited by our capacity to observe the full-scale spectrum at play. In order to bridge this fundamental gap in our observation capacity I will work to: create the first long-term Integrated Cross-Scale land-Atmosphere Experiment (CSLAEX) since I will work to: measure the components of the surface energy balance at a spatial scale relevant to remote sensing and meteorological applications (5km), along with observations of the nested subscale processes with Distributed Temperature Sensors (DTS). This proposal aims to fill the gaps in modeling and observing land-atmosphere interactions across scales via a novel Cross-Scale Land-Atmosphere Experiment (CSLAEX) to be implemented as part of the Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) Southern Great Plains (SGP) site near Lamont, Oklahoma. The proposed research effort aims at exploring land-atmosphere processes across multiple spatial scales and to investigate the effect of landscape heterogeneity using Distributed Temperature Sensor measurements. The novel instrumentation of the Surface Energy Balance and simultaneous observations of the atmospheric state (boundary layer, cloud, convection) available at the DOE ARM SGP site will be used to create CSLAEX. This dataset will be used to frame and evaluate new transport laws and parameterizations for the boundary layer and the land surface, which can be implemented into the Weather and Research Forecasting (WRF) model and Community Atmospheric Model (CAM). The result of this study should impact multiple fields: meteorology, climate, and hydrology.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Comprehensive Analysis of the Relative Dispersion of Droplet-Size Distributions and Their Relationships to Key Physical Fog Processes Under Different Aerosol Conditions and Evolutionary Stages

The relative dispersion of cloud and fog droplets has significant impacts on aerosol indirect effects, radiative transfer, and microphysical processes. However, previous studies have been mostly concerned with clouds, with limited studies on fog, particularly those that examine the combined influences of all key physical processes and their roles during fog evolution. As such, this study aims to conduct a comprehensive investigation by examining the relationships between relative dispersion and other microphysical variables, as well as the underlying microphysical and dynamic processes, based on field fog campaigns in polluted and clean conditions. In polluted fog, droplet concentrations are higher, leading to smaller droplets and increased dispersion. The correlation between dispersion and droplet volume-mean radius is positive in the polluted fog, but shifts to negative in clean fog. Here, we attribute the difference to various microphysical processes like aerosol activation, condensation, collision-coalescence, and entrainment-mixing. In polluted fog, high aerosol concentrations, low supersaturations, and strong turbulence (entrainment-mixing) provide suitable conditions for the simultaneous occurrence of droplet condensation and aerosol activation, resulting in a positive correlation between dispersion and volume-mean radius, especially during the fog formation stage. In contrast, during the mature stage in clean fog, condensation is dominant with weak aerosol activation leading to a negative correlation between relative dispersion and volume-mean radius. The collision-coalescence process is more active in the mature stage, increasing radii and leading to the negative correlation between dispersion and volume-mean radius. This result sheds new light on understanding the relative dispersion and mechanisms in fog under different aerosol backgrounds.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Development of Steady-State and Dynamic Mass and Energy Constrained Neural Networks for Distributed Chemical Systems Using Noisy Transient Data

The paper presents the development of algorithms for mass and energy constrained neural network models that can exactly conserve the overall mass and energy of distributed chemical process systems, even though the noisy transient data used for optimal model training violate the same. In contrast to approximately satisfying mass and energy balance constraints of a system by soft penalization of objective function, algorithms have been developed for solving equality-constrained nonlinear optimization problems, thus providing the guarantee of exactly satisfying the system mass and energy conservation laws. For developing dynamic mass-energy constrained network models for distributed systems, hybrid series and parallel dynamic-static neural networks have been leveraged. The developed algorithms for solving both the training and forward problems are validated using both steady-state and dynamic data in the presence of various noise characteristics. The developed data-driven algorithms are flexible to exactly satisfy mass and energy balance constraints for dynamic chemical processes if the system holdup information is available. The proposed network structures and algorithms are applied to the development of data-driven lumped and distributed models of an adiabatic superheater/reheater system, a nonisothermal continuous stirred tank reactor, as well as an electrically heated plug-flow reactor system where one form of energy gets transformed to another. It has been observed that the mass-energy constrained neural networks yield a root mean squared error of <1% with respect to the system truth for the case studies evaluated in this work.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2022

Automated Identification of Characteristic Droplet Size Distributions in Stratocumulus Clouds Utilizing a Data Clustering Algorithm

Abstract Droplet-level interactions in clouds are often parameterized by a modified gamma fitted to a “global” droplet size distribution. Do “local” droplet size distributions of relevance to microphysical processes look like these average distributions? This paper describes an algorithm to search and classify characteristic size distributions within a cloud. The approach combines hypothesis testing, specifically, the Kolmogorov–Smirnov (KS) test, and a widely used class of machine learning algorithms for identifying clusters of samples with similar properties: density-based spatial clustering of applications with noise (DBSCAN) is used as the specific example for illustration. The two-sample KS test does not presume any specific distribution, is parameter free, and avoids biases from binning. Importantly, the number of clusters is not an input parameter of the DBSCAN-type algorithms but is independently determined in an unsupervised fashion. As implemented, it works on an abstract space from the KS test results, and hence spatial correlation is not required for a cluster. The method is explored using data obtained from the Holographic Detector for Clouds (HOLODEC) deployed during the Aerosol and Cloud Experiments in the Eastern North Atlantic (ACE-ENA) field campaign. The algorithm identifies evidence of the existence of clusters of nearly identical local size distributions. It is found that cloud segments have as few as one and as many as seven characteristic size distributions. To validate the algorithm’s robustness, it is tested on a synthetic dataset and successfully identifies the predefined distributions at plausible noise levels. The algorithm is general and is expected to be useful in other applications, such as remote sensing of cloud and rain properties. Significance Statement A typical cloud can have billions of drops spread over tens or hundreds of kilometers in space. Keeping track of the sizes, positions, and interactions of all of these droplets is impractical, and, as such, information about the relative abundance of large and small drops is typically quantified with a “size distribution.” Droplets in a cloud interact locally, however, so this work is motivated by the question of whether the cloud droplet size distribution is different in different parts of a cloud. A new method, based on hypothesis testing and machine learning, determines how many different size distributions are contained in a given cloud. This is important because the size distribution describes processes such as cloud droplet growth and light transmission through clouds.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

An Integrated Modeling Framework for Sediment Dynamics During Urban Flooding: Application to Hurricane Harvey in Houston

Floodwater can mobilize and redistribute large volumes of sediment from upland to downstream urban areas, threatening infrastructure, water quality, and ecosystem health. However, existing modeling approaches often fail to capture sediment dynamics in urban floodplains due to the lack of integration between upland hydrological processes and riverine sediment transport. This study presents the first integrated modeling framework that couples the Energy Exascale Earth System Model (E3SM) land component, which simulates runoff and hillslope erosion, with TELEMAC-GAIA, a two-dimensional hydrodynamic and sediment transport model. This framework enables the fully distributed, process-based simulation of high-resolution (as fine as 30 m) sediment dynamics from hillslopes to floodplains. Applied to a highly urbanized watershed in Houston during Hurricane Harvey, this framework reproduced observed water levels at 16 USGS gauges (median R 2 = 0.83 and KGE = 0.78), key sediment dynamics such as sediment transport and deposition processes, and reproduced spatial deposition patterns consistent with LiDAR-derived data. Based on the simulation, we estimate 8.0 million m 3 of event-scale sediment deposition, including 5.7 million m 3 trapped in the flood-control reservoirs and 2.3 million m 3 deposited along major channels and floodplains. Using a representative unit removal cost, this corresponds to an estimated dredging cost of $581 million for total deposition. These results provide a first-order, physically based quantification of Harvey-scale sediment impacts. This study provides a valuable tool for the holistic analysis of sediment dynamics triggered by extreme urban flooding, supporting flood-resilience planning. More broadly, it highlights the importance of integrating physically based hydrological processes for urban flooding and sediment research.

Hurricane Harvey↗
DOE OSTI2023

State Energy Justice Roundtable Series: Energy Justice Metrics

The National Association of Regulatory Utility Commissioners (NARUC), National Association of State Energy Officials (NASEO), and National Governors Association (NGA) hosted a State Energy Justice Roundtable (Roundtable) in April 2022. Participants included federal and state decision-makers, members of community-based organizations, and subject-matter experts. The Roundtable members explored current state efforts to articulate and incorporate energy justice concerns into energy-related decision-making. Participants established connections with one another to better understand the current landscape of existing resources, learn about emerging efforts, and identify ongoing support opportunities for advancing energy justice.This paper is one of five authored by the host organizations on topics that were the focus of the Roundtable. Each paper summarizes key themes, emerging efforts, and group takeaways that were discussed at the Roundtable and should assist state members in developing and meeting their own state goals around energy justice. The papers all include the same discussions of background, introduction, and reading list so they can be read separately. Each paper is written from the perspective of one association and includes options for its members to take actions that could support more equitable state energy policies and programs. The five papers cover: (1) Participation in decision making (NARUC), (2) Customer affordability and arrearages (NARUC), (3) Energy justice metrics (NARUC), (4) Equity in clean energy research and development (NASEO), and (5) Equitable distributed energy resource (DER) access (NGA). The resources and recommendations listed in these papers are not meant to be exhaustive, as this field of study continues to evolve. Although this brief is focused on electricity, energy justice considerations extend to all energy needs and services, including the impact of energy extraction, processing, and distribution functions.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2023

State Energy Justice Roundtable Series: Customer Affordability and Arrearages

The National Association of Regulatory Utility Commissioners (NARUC), National Association of State Energy Officials (NASEO), and National Governors Association (NGA) hosted a State Energy Justice Roundtable (Roundtable) in April 2022. Participants included federal and state decision-makers, members of community-based organizations, and subject-matter experts. The Roundtable members explored current state efforts to articulate and incorporate energy justice concerns into energy-related decision-making. Participants established connections with one another to better understand the current landscape of existing resources, learn about emerging efforts, and identify ongoing support opportunities for advancing energy justice. This paper is one of five authored by the host organizations on topics that were the focus of the Roundtable. Each paper summarizes key themes, emerging efforts, and group takeaways that were discussed at the Roundtable and should assist state members in developing and meeting their own state goals around energy justice. The papers all include the same discussions of background, introduction, and reading list so they can be read separately. Each paper is written from the perspective of one association and includes options for its members to take actions that could support more equitable state energy policies and programs. The five papers cover: (1) Participation in decision making (NARUC) (2) Customer affordability and arrearages (NARUC) (3) Energy justice metrics (NARUC) (4) Equity in clean energy research and development (NASEO) (5) Equitable distributed energy resource (DER) access (NGA). The resources and recommendations listed in these papers are not meant to be exhaustive, as this field of study continues to evolve. Although this brief is focused on electricity, energy justice considerations extend to all energy needs and services, including the impact of energy extraction, processing, and distribution functions.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2023

State Energy Justice Roundtable Series: Participation in Decision Making

The National Association of Regulatory Utility Commissioners (NARUC), National Association of State Energy Officials (NASEO), and National Governors Association (NGA) hosted a State Energy Justice Roundtable (Roundtable) in April 2022. Participants included federal and state decision-makers, members of community-based organizations, and subject-matter experts. The Roundtable members explored current state efforts to articulate and incorporate energy justice concerns into energy-related decision-making. Participants established connections with one another to better understand the current landscape of existing resources, learn about emerging efforts, and identify ongoing support opportunities for advancing energy justice. This paper is one of five authored by the host organizations on topics that were the focus of the Roundtable. Each paper summarizes key themes, emerging efforts, and group takeaways that were discussed at the Roundtable and should assist state members in developing and meeting their own state goals around energy justice. The papers all include the same discussions of background, introduction, and reading list so they can be read separately. Each paper is written from the perspective of one association and includes options for its members to take actions that could support more equitable state energy policies and programs. The five papers cover: • Participation in decision making (NARUC) • Customer affordability and arrearages (NARUC) • Energy justice metrics (NARUC) • Equity in clean energy research and development (NASEO) • Equitable distributed energy resource (DER) access (NGA). The resources and recommendations listed in these papers are not meant to be exhaustive, as this field of study continues to evolve. Although this brief is focused on electricity, energy justice considerations extend to all energy needs and services, including the impact of energy extraction, processing, and distribution functions.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2022

Thermodynamics of the Inner Heliosheath

We derive annual sky maps of the proton temperature in the inner heliosheath (IHS), and track their temporal evolution over the years 2009–2016 of Interstellar Boundary Explorer observations. Other associated thermodynamic parameters also determined are the density, kappa (the parameter that characterizes kappa distributions), temperature rate, polytropic index, and entropy. We exploit the theory of kappa distributions and their connection with polytropes, to (i) express a new polytropic quantity Π that remains invariant along streamlines where temperature and density may vary, (ii) parameterize the proton flux in terms of the Π invariant and kappa, and (iii) derive the temperature and density, respectively, from the slope and intercept of the linear relationship between kappa and logarithm of Π. We find the following thermodynamic characteristics: (1) temperature sky maps and histograms shifted to their lowest values in 2012 and their highest in 2015; (2) temperature negatively correlated with density, reflecting the subisothermal polytropic behavior; (3) temperature positively correlated with kappa, revealing characteristics of the mechanism responsible for generating kappa distributions; (4) processes in IHS are subisothermal tending toward isobaric, consistent with previously published results; (5) linear relationship between kappa and polytropic indices, revealing characteristics of the particle potential energy; and (6) entropy positively correlated with polytropic index, aligned with the underlying theory that entropy increases toward the isothermal state where the kappa distribution reduces to the Maxwell–Boltzmann description.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Riverine bacterioplankton and phytoplankton assembly along an environmental gradient induced by urbanization

In the Anthropocene, human activities are intensifying stresses on biological communities across ecosystems. Knowledge on how environmental changes influence microbial community dynamics is crucial but currently limited. Here we unraveled ecological processes generating distribution patterns of bacterioplankton and phytoplankton communities in a river draining an urbanizing watershed in southern China. In this ecosystem, longitudinal environmental heterogeneity and directional flow potentially have counteracting effects on planktonic community assembly. Using simultaneously multiple approaches, we showed that the observed compositional patterns result from an interplay of stochastic and deterministic processes, indicating that dispersal via fluvial connectivity could not override local selective pressures. Successional shifts in the relative importance of the above processes were associated with trends in climatic factors. High diversity was observed at the intermediate levels of ecological processes alongside intense inter-species interactions maintained by a few keystone taxa. Altogether, interactions between climatic conditions, landscape and hydrology control plankton community dynamics.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Overview and status of the Long-Baseline Neutrino Facility Far Site cryogenics system

The Sanford Underground Research Facility (SURF) will host the Far Detector of the Deep Underground Neutrino Experiment (DUNE), an international multi-kiloton Long-Baseline neutrino experiment that will be installed about one and a half kilometers underground in Lead, SD. Detectors will be located inside four cryostats filled with almost 70,000 metric tons of ultrapure liquid argon, with a level of impurities lower than 100 parts per trillion of oxygen equivalent contamination. The cryogenics infrastructure supporting this experiment is provided by the Long-Baseline Neutrino Facility (LBNF). This contribution presents modes of operation, layout, and main features of the LBNF Far Site cryogenic system, which is composed of three subsystems: Infrastructure, Proximity, and Internal cryogenics. The Infrastructure cryogenics supports the needs of the cryostat and Proximity cryogenics. It includes the equipment to receive the argon in liquid phase, vaporize it and transfer it underground as a gas, the nitrogen system (composed of the refrigeration system, liquid nitrogen buffer tanks and liquid and gaseous nitrogen distribution), liquid and gaseous argon distribution and process controls. The Proximity cryogenics receives fluids from the Infrastructure cryogenics and delivers them to the Internal cryogenics at the required temperature, pressure, purity and mass flow rate. It includes the argon condensers, liquid and gaseous argon purification and regeneration systems, nitrogen and argon phase separators, piping, valves, and instrumentation. The Internal cryogenics comprises the liquid and gaseous argon distribution inside the cryostats for the commissioning, cool down, fill, and steady state operations of the cryostats and detectors. An international engineering team is designing these systems and will manufacture, install, commission, and qualify them. This contribution describes the main features, performance, functional requirements, and modes of operation of the LBNF Far Site cryogenics system. It also presents the status of the design, along with present and future needs to support the DUNE experiment.

43 PARTICLE ACCELERATORS↗
DOE OSTI2023

VDiSC: An Open Source Framework for Distributed Smart City Vision and Biometric Surveillance Networks

Recent global growth in the interest of smart cities has led to trillions of dollars of investment toward research and development. These connected cities have the potential to create a symbiosis of technology and society and revolutionize the cost of living, safety, ecological sustainability, and quality of life of societies on a world-wide scale. Some key components of the smart city construct are connected smart grids, self-driving cars, federated learning systems, smart utilities, large-scale public transit, and proactive surveillance systems. While exciting in prospect, these technologies and their subsequent integration cannot be attempted without addressing the potential societal impacts of such a high degree of automation and data sharing. Additionally, the feasibility of coordinating so many disparate tasks will require a fast, extensible, unifying framework. To that end, we propose the Distributed Smart City framework for Vision, or VDiSC. VDiSC serves as a unified biometric API harness that allows for seamless evaluation, deployment, and simple pipeline creation for heterogeneous biometric software. VDiSC additionally provides a fully declarative capability for defining and coordinating custom machine learning and sensor pipelines, allowing the distribution of processes across otherwise incompatible hardware and networks. VDiSC ultimately provides a way to quickly configure, hot-swap, and expand large coordinated or federated systems online without interruptions for maintenance. Because much of the data collected in a smart city contains Personally Identifying Information (PII), VDiSC also provides built-in tools and layers to ensure secure and encrypted streaming, storage, and access of PII data across distributed systems.

Brogan, Joel↗
DOE OSTI2025

Hybrid Quantum Networks with Discrete Polarizations and Continuous Quadrature Variables

Final Scientific/Technical Report for DOE Project DE-SC0022069. Quantum communication and computation systems have evolved along two largely independent paradigms: discrete-variable (DV) systems that encode information in qubits such as photon polarizations or photon-number states, and continuous-variable (CV) systems that encode information in optical field quadratures. Each approach offers distinct advantages—DVs provide low error rates and compatibility with single-photon platforms, while CVs support deterministic operations and efficient quantum state manipulation. A fundamental challenge for building a scalable Quantum Internet lies in interfacing these two regimes into a unified hybrid architecture that can coherently distribute and process quantum information across heterogeneous quantum nodes. This project aims to develop and demonstrate a hybrid optical quantum network that seamlessly integrates DV and CV systems. Specifically, we investigate a new class of hybrid entanglement between the discrete polarizations of single photons and the continuous quadrature variables of optical cat states, overcoming incompatibilities in existing DV–CV demonstrations. Using this new entanglement resource, the team investigates a multi-node hybrid quantum local area network (Q-LAN) testbed capable of DV–CV entanglement generation, swapping, and distribution across fiber links.

74 ATOMIC AND MOLECULAR PHYSICS↗
DOE OSTI2025

Harnessing distributed GPU computing for generalizable graph convolutional networks in power grid reliability assessments

Although machine learning (ML) has emerged as a powerful tool for rapidly assessing grid contingencies, prior studies have largely considered a static grid topology in their analyses. This limits their application, since they need to be re-trained for every new topology. Here, this paper explores the development of generalizable graph convolutional network (GCN) models by pre-training them across a range of grid topologies and contingency types. We found that a GCN model with auto-regressive moving average (ARMA) layers with a line graph representation of the grid offered the best predictive performance in predicting voltage magnitudes (VM) and voltage angles (VA). We introduced the concept of phantom nodes to consider disparate grid topologies with a varying number of nodes and lines. For pre-training the GCN ARMA model across a variety of topologies, distributed graphics processing unit (GPU) computing afforded us significant training scalability. The predictive performance of this model on grid topologies that were part of the training data is substantially better than the direct current (DC) approximation. Although direct application of the pre-trained model to topologies that are not part of the grid is not particularly satisfactory, fine-tuning with small amounts of data from a specific topology of interest significantly improves predictive performance. In general, this paper highlights the feasibility of training large-scale GNN models to assess the reliability of power grids by considering a wide variety of grid topologies and contingency types. With the advent of foundational models in ML and the exponential increase in GPU computing clusters, generalizable ML models will significantly enhance how utilities manage power systems and make decisions in real-time or near-real-time.

24 - POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2020

Distributed non-negative matrix factorization with determination of the number of latent features

The holistic analysis and understanding of the latent (that is, not directly observable) variables and patterns buried in large datasets is crucial for data-driven science, decision making and emergency response. Such exploratory analyses require devising unsupervised learning methods for data mining and extraction of the latent features, and non-negative matrix factorization (NMF) is one of the prominent such methods. NMF is based on compute-intense non-convex constrained minimization, which, for large datasets requires fast and distributed algorithms. However, current parallel implementations of NMF fail to estimate the number of latent features. In practice, identifying these features is both difficult and significant for pattern recognition and latent feature analysis, especially for large dense matrices. Here, we introduce a distributed NMF algorithm coupled with distributed custom clustering followed by a stability analysis on dense data, which we call DnMFk, to determine the number of latent variables. The results on synthetic data and the classical Swimmer data set demonstrate the accuracy of model determination while scaling nearly linearly across multiple processors for large data. Further, we employ DnMFk to determine the number of hidden features from a terabyte matrix.

97 MATHEMATICS AND COMPUTING↗