SEARCH · Engineering Papers

Results for “data clustering”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

All sources NASA NTRS DOE OSTI

DOE OSTI2021

Scalable, In-situ Data Clustering Data Analysis for Extreme Scale Scientific Computing (Final Report)

The objective of this project is to address challenges in the design and development of scalable in-situ data clustering and analytics algorithms and software. Our goal is to develop parallel software consisting of a set of spatio-temporal data clustering and anomaly detection functions, both of which are very important for large-scale analysis and have wide applicability for in-situ runs as well as post-processing analysis. Our design principles for in-situ analysis consider the following: (1) identify parts of the computation can be done close to the data within the nodes, while it is still in memory; (2) extract analysis components can (and should) be performed in remote staging and analysis nodes; (3) develop error-bound approximation methods for applications tolerable for small errors; (4) identify the type of derived distributions and statistics, for spatio-temporal data, that can be kept locally in order to both accelerate computations and meet energy constraints in subsequent iterations and phases; (5) use a self-describing data format so that data can be consistent and understood among local storage (memory and SSDs) and at staging and analysis nodes, thereby providing portability and flexibility; (6) develop service-oriented functions that can schedule in-situ and post-hoc analysis tasks based on the dynamic requirements of applications. Our development focus is to produce the parallel data analysis software/library that will be scalable, reusable, extensible, and generic for applications in different disciplines. The software will be able to run in-situ with the simulations as well as post-hoc analysis. This approach will satisfy many synergistic requirements for data intensive applications executed on data coming from instruments and experiments. In particular, the proposed multilevel approach is directly applicable to perform design tradeoffs for running part of the algorithms near the instruments and the rest on remote (analysis) systems.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2015

Cluster-to-cluster data transfer with data compression over wide-area networks

The recent emergence of ultra high-speed networks up to 100 Gb/s has posed numerous challenges and has led to many investigations on efficient protocols to saturate 100 Gb/s links. However, end-to-end data transfers involve many components, not only protocols, affecting overall transfer performance. These components include disk I/O subsystem, additional computation associated with data streams, and network adapters. For example, achievable bandwidth by TCP may not be implementable if disk I/O or CPU becomes a bottleneck in end-to-end data transfer. In this paper, we first model all the system components involved in end-to-end data transfer as a graph. We then formulate the problem whose goal is to achieve maximum data transfer throughput using parallel data flows. We also propose a variable data flow GridFFP XIO stack to improve data transfer with data compression. Our contributions lie in how to optimize data transfers considering all the system components involved rather than in accurately modeling all the system components involved. Our proposed formulations and solutions are evaluated through experiments on the ESnet 100G testbed and a wide-area cluster-to-cluster testbed. The experimental results on the ESnet 100G testbed show that our approach is several times faster than Globus Online-8 x faster for datasets with many 10 MB files and 3-4 x faster for other datasets of larger size files. The experimental results on the cluster-to-cluster testbed show that our variable data flow approach is up to 4 x faster than a normal cluster data transfer.

Disk-to-Disk Transfer↗

DOE OSTI2019

Community Energy by Design: A Simulation-Based Design Workflow Using Measured Data Clustering to Calibrate Urban Building Energy Models (UBEMs)

This paper presents a workflow that informs urban design decisions using measured data clustering to calibrate urban building energy models. The method's goal is to support urban design in terms of form, building systems configurations, as well as influencing user behavior aspects in the built environment through a systemic analysis of measured data to develop reliable future-case design scenario energy models. Detailed data on appliance-level electricity use are employed via data clustering to calibrate a urban building energy model for the Mueller community in Austin, TX, USA. The data were collected by the Pecan Street Institute for a year in 2014 from consumers in Austin and other surrounding cities. First, collected energy data were restructured and cleaned from corrupt and/or missing information. Second, in order to identify common energy use patterns, a model-based clustering algorithm for functional data was applied. Behavioral/usage profiles are determined through clustering and translated into usage schedules and behaviors. As a result, an urban building energy model built in the urban modeling interface (umi) was calibrated, with fully calibrated and semi-calibrated buildings, within a maximum calibration error margin of 14% for daily-scale averages. Finally, an illustration of calibrated-urban building energy model design case scenarios is presented, and implications on community energy potential effects are discussed.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

DOE OSTI2023

Informed Feature Selection for Data Clustering of CSP Plant Production

To make concentrating solar power (CSP) more cost competitive, rigourous optimizations must be run to improve plant design and operations. However, these optimizaitons rely on time consuming annual simulations that solve an electricity dispatch scheduling problem to maximize plant revenue. To reduce the runtime of annual dispatch simulations of CSP plants, a data clustering approach is utilized. This approach assumes that like days of revenue and electricity generation can be identified using weather and price data. Although weather and price are important factors for electricity production, this work investigates how thermal energy storage (TES) inventory at the beginning of a day, denoted as Si, can be used as a supplemental feature to group like days. A framework for creating and training a deep neural network to predict Si is proposed. This model is validated and assessed using eleven sets of testing data that were not used during training. Then, the data clustering approach is performed three seperate times with features of weather and price along with either Si from the neural network, Si from the full annual simulation, or no Si. Ultimately, the results suggest that using Si as an additional clustering feature improves the data clustering simulation accuracy by 1.4%.

Tuman, Matthew J. (ORCID:000900038772051X)↗

DOE OSTI2015

A STATISTICAL STUDY OF THE SOLAR WIND TURBULENCE AT ION KINETIC SCALES USING THE K-FILTERING TECHNIQUE AND CLUSTER DATA

Plasma turbulence at ion kinetic scales in the solar wind is investigated using the multi-point magnetometer data from the Cluster spacecraft. By applying the k-filtering method, we are able to estimate the full three-dimensional power spectral density P(ω{sub sc}, k) at a certain spacecraft frequency ω{sub sc} in wavevector (k) space. By using the wavevector at the maximum power in P(ω{sub sc}, k) at each sampling frequency ω{sub sc} and the Doppler shifted frequency ω{sub pla} in the solar wind frame, the dispersion plot ω{sub pla} = ω{sub pla}(k) is found. Previous studies have been limited to very few intervals and have been hampered by large errors, which motivates a statistical study of 52 intervals of solar wind. We find that the turbulence is predominantly highly oblique to the magnetic field k >> k {sub ∥}, and propagates slowly in the plasma frame with most points having frequencies smaller than the proton gyrofrequency ω{sub pla} < Ω{sub p}. Weak agreement is found that turbulence at the ion kinetic scales consists of kinetic Alfvén waves and coherent structures advected with plasma bulk velocity plus some minor more compressible components. The results suggest that anti-sunward and sunward propagating magnetic fluctuations are of similar nature in both the fast and slow solar wind at ion kinetic scales. The fast wind has significantly more anti-sunward flux than sunward flux and the slow wind appears to be more balanced.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2021

Efficient prediction of concentrating solar power plant productivity using data clustering

Concentrating solar power (CSP) plants convert solar energy to electricity and can be deployed with a thermal storage capability to shift electricity generation from time periods with available solar resource to those with high electricity demand or electricity price. Rigorous optimization of plant design and operational strategies can improve the market-competitiveness and commercial viability; however, such optimization may require hundreds of annual performance simulations, each of which can be computationally expensive when including considerations such as optimization of dispatch scheduling, sub-hourly time resolution, and stochastic effects due to uncertain weather or electricity price forecasts. This paper proposes a methodology to reduce the computational burden associated with simulation of electricity yield and revenue for CSP plants over a single- or multi-year period. Data-clustering techniques are employed to select a small number of limited-duration time blocks for simulation that, when appropriately weighted, can reproduce generation and revenue over a single year or within each year of a multi-year period. After selection of appropriate data features and weighting factors defining similarity between time-series profiles, the methodology captured annual revenue within 2.3%, 1.7%, or 1.2% using simulation of 10, 30, or 50 three-day exemplar time blocks, respectively, for each of three single-year location/weather/market scenarios and five plant configurations ranging from low to high solar multiple and storage capacity. When applied to multi-year datasets, the proposed methodology can capture inter-year variability that is unavailable from typical meteorological year (TMY) datasets while simultaneously requiring simulation of less than a single year of data.

14 SOLAR ENERGY↗

DOE OSTI2023

Crowd cluster data in the USA for analysis of human response to COVID-19 events and policies

We provide data on daily social contact intensity of clusters of people at different types of Points of Interest (POI) by zip code in Florida and California. This data is obtained by aggregating fine-scaled details of interactions of people at the spatial resolution of 10 m, which is then normalized as a social contact index. We also provide the distribution of cluster sizes and average time spent in a cluster by POI type. This data will help researchers perform fine-scaled, privacy-preserving analysis of human interaction patterns to understand the drivers of the COVID-19 epidemic spread and mitigation. Current mobility datasets either provide coarse-level metrics of social distancing, such as radius of gyration at the county or province level, or traffic at a finer scale, neither of which is a direct measure of contacts between people. We use anonymized, de-identified, and privacy-enhanced location-based services (LBS) data from opted-in cell phone apps, suitably reweighted to correct for geographic heterogeneities, and identify clusters of people at non-sensitive public areas to estimate fine-scaled contacts.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2022

Automated Identification of Characteristic Droplet Size Distributions in Stratocumulus Clouds Utilizing a Data Clustering Algorithm

Abstract Droplet-level interactions in clouds are often parameterized by a modified gamma fitted to a “global” droplet size distribution. Do “local” droplet size distributions of relevance to microphysical processes look like these average distributions? This paper describes an algorithm to search and classify characteristic size distributions within a cloud. The approach combines hypothesis testing, specifically, the Kolmogorov–Smirnov (KS) test, and a widely used class of machine learning algorithms for identifying clusters of samples with similar properties: density-based spatial clustering of applications with noise (DBSCAN) is used as the specific example for illustration. The two-sample KS test does not presume any specific distribution, is parameter free, and avoids biases from binning. Importantly, the number of clusters is not an input parameter of the DBSCAN-type algorithms but is independently determined in an unsupervised fashion. As implemented, it works on an abstract space from the KS test results, and hence spatial correlation is not required for a cluster. The method is explored using data obtained from the Holographic Detector for Clouds (HOLODEC) deployed during the Aerosol and Cloud Experiments in the Eastern North Atlantic (ACE-ENA) field campaign. The algorithm identifies evidence of the existence of clusters of nearly identical local size distributions. It is found that cloud segments have as few as one and as many as seven characteristic size distributions. To validate the algorithm’s robustness, it is tested on a synthetic dataset and successfully identifies the predefined distributions at plausible noise levels. The algorithm is general and is expected to be useful in other applications, such as remote sensing of cloud and rain properties. Significance Statement A typical cloud can have billions of drops spread over tens or hundreds of kilometers in space. Keeping track of the sizes, positions, and interactions of all of these droplets is impractical, and, as such, information about the relative abundance of large and small drops is typically quantified with a “size distribution.” Droplets in a cloud interact locally, however, so this work is motivated by the question of whether the cloud droplet size distribution is different in different parts of a cloud. A new method, based on hypothesis testing and machine learning, determines how many different size distributions are contained in a given cloud. This is important because the size distribution describes processes such as cloud droplet growth and light transmission through clouds.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2015

HUBBLE FRONTIER FIELDS FIRST COMPLETE CLUSTER DATA: FAINT GALAXIES AT z ∼ 5-10 FOR UV LUMINOSITY FUNCTIONS AND COSMIC REIONIZATION

We present comprehensive analyses of faint dropout galaxies up to z ∼ 10 with the first full-depth data set of the A2744 lensing cluster and parallel fields observed by the Hubble Frontier Fields (HFF) program. We identify 54 dropouts at z ∼ 5-10 in the HFF fields and enlarge the size of the z ∼ 9 galaxy sample obtained to date. Although the number of highly magnified (μ ∼ 10) galaxies is small because of the tiny survey volume of strong lensing, our study reaches the galaxies' intrinsic luminosities comparable to the deepest-field HUDF studies. We derive UV luminosity functions with these faint dropouts, carefully evaluating by intensive simulations the combination of observational incompleteness and lensing effects in the image plane, including magnification, distortion, and multiplication of images, with the evaluation of mass model dependencies. Our results confirm that the faint-end slope, α, is as steep as –2 at z ∼ 6-8 and strengthen the evidence for the rapid decrease of UV luminosity densities, ρ{sub UV}, at z > 8 from the large z ∼ 9 sample. We examine whether the rapid ρ{sub UV} decrease trend can be reconciled with the large Thomson scattering optical depth, τ{sub e}, measured by cosmic microwave background experiments, allowing a large space of free parameters, such as an average ionizing photon escape fraction and a stellar-population-dependent conversion factor. No parameter set can reproduce both the rapid ρ{sub UV} decrease and the large τ {sub e}. It is possible that the ρ{sub UV} decrease moderates at z ≳ 11, that the free parameters significantly evolve toward high z, or that there exist additional sources of reionization such as X-ray binaries and faint active galactic nuclei.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2019

Data clustering for the high-resolution alignment of microstructure and strain fields

The emergence of small-scale deformation mapping by a combination of scanning electron microscopy and digital image correlation (SEM-DIC) has enabled full-field investigations into the complex roles of microstructure on microscale deformation mechanisms. However, these investigations are hindered by errors in alignment between the microstructure data, generally acquired by electron backscatter diffraction (EBSD), and the microscale strain data obtained by SEM-DIC. Distortions, stitching artifacts, and spatial resolution differences between microstructure and strain fields can lead to misalignments on the order of μms. This alignment uncertainty introduces error into microstructure-strain localization analyses and precludes the examination of deformation mechanisms near grain boundaries. To improve alignment between EBSD-obtained grain boundaries and SEM-DIC strain data, an approach was created wherein a mantle was first established around the EBSD-acquired grain boundaries. Strain data was then clustered within this mantle using a k-means algorithm to identify grain boundary strain localization, and these boundary points were fit to obtain a continuous curve for each individual boundary. Clustered point outliers, such as those due to grain boundary-local dislocation slip, were statistically identified by using an absolute error threshold and removed from the grain boundary fit. The ending identification of grain boundaries in the microscale strain data is significantly improved from EBSD-identified boundaries.

36 MATERIALS SCIENCE↗

DOE OSTI2016

Constraints on a possible variation of the fine structure constant from galaxy cluster data

We propose a new method to probe a possible time evolution of the fine structure constant α from X-ray and Sunyaev-Zel'dovich measurements of the gas mass fraction ( f {sub gas}) in galaxy clusters. Taking into account a direct relation between variations of α and violations of the distance-duality relation, we discuss constraints on α for a class of dilaton runaway models. Although not yet competitive with bounds from high- z quasar absorption systems, our constraints, considering a sample of 29 measurements of f {sub gas}, in the redshift interval 0.14 < z < 0.89, provide an independent estimate of α variation at low and intermediate redshifts. Furthermore, current and planned surveys will provide a larger amount of data and thus allow to improve the limits on α variation obtained in the present analysis.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2017

Systematic design of active spaces for multi-reference calculations of singlet–triplet gaps of organic diradicals, with benchmarks against doubly electron-attached coupled-cluster data

Singlet–triplet gaps in diradical organic π-systems are of interest in many applications. In this work, we calculate them in a series of molecules, including cyclobutadiene and its derivatives and cyclopentadienyl cation, by using correlated participating orbitals within the complete active space (CAS) and restricted active space (RAS) self-consistent field frameworks, followed by second-order perturbation theory (CASPT2 and RASPT2). These calculations are evaluated by comparison with the results of doubly electron-attached (DEA) equation-of-motion (EOM) coupled-cluster (CC) calculations with up to 4-particle–2-hole (4p-2h) excitations. Finally, we find active spaces that can accurately reproduce the DEA-EOMCC(4p-2h) data while being small enough to be applicable to larger organic diradicals.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2023

A Two-Step Time-Series Data Clustering Method for Building-Level Load Profile

Residential and commercial buildings have huge potential to contribute value to improve grid resilience by participating grid services. To reveal the significant value, it is critical to estimate the grid service capability from these buildings. Unlike the large-scale distributed energy resources such as wind and solar farms, those buildings need to participate grid services in aggregation, not by individual. Therefore, it is important to appropriately group buildings for aggregation. The load profiles in the same group will have similar characteristics at the same time step, so grid operators can send the grid service signal to the customer group with a higher chance to respond at that time step. In this paper, we develop a load profile clustering method to classify the building-level load profiles for grid service capability estimation. In our two-step clustering approach, we first calculate the total load consumption for each building, clustering the load profiles based on energy consumption level. Then, we further cluster the load profiles in each energy cluster based on the load shape. The parameter selection for each clustering step is discussed. The proposed method is applied on actual building-level load profiles, and the results have proved the effectiveness of this method.

advanced metering infrastructure (AMI)↗

DOE OSTI2023

A Two-Step Time-Series Data Clustering Method for Building-Level Load Profile

AMI↗

DOE OSTI2023

A Two-Step Time-Series Data Clustering Method for Building-Level Load Profile: Preprint

Residential and commercial buildings have huge potential to contribute value to improve grid resilience by participating grid services. To reveal the significant value, it is critical to estimate the grid service capability from these buildings. Unlike the large-scale distributed energy resources such as wind and solar farms, those buildings need to participate grid services in aggregation, not by individual. Therefore, it is important to appropriately group buildings for aggregation. In this paper, we develop a load profile clustering method to classify the building-level load profiles for grid service capability estimation. In our two-step clustering approach, we first calculate the total load consumption for each building, clustering the load profiles based on energy consumption level. Then, we further cluster the load profiles in each energy cluster based on the load shape. The parameter selection for each clustering step is discussed. The proposed method is applied on actual building-level load profiles, and the results have proved the effectiveness of this method.

advanced metering infrastructure (AMI)↗

DOE OSTI2014

Method for discovering relationships in data by dynamic quantum clustering

Data clustering is provided according to a dynamical framework based on quantum mechanical time evolution of states corresponding to data points. To expedite computations, we can approximate the time-dependent Hamiltonian formalism by a truncated calculation within a set of Gaussian wave-functions (coherent states) centered around the original points. This allows for analytic evaluation of the time evolution of all such states, opening up the possibility of exploration of relationships among data-points through observation of varying dynamical-distances among points and convergence of points into clusters. This formalism may be further supplemented by preprocessing, such as dimensional reduction through singular value decomposition and/or feature filtering.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2017

Method for discovering relationships in data by dynamic quantum clustering

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2015

A Measurement of Gravitational Lensing of the Cosmic Microwave Background by Galaxy Clusters Using Data from the South Pole Telescope

Clusters of galaxies are expected to gravitationally lens the cosmic microwave background (CMB) and thereby generate a distinct signal in the CMB on arcminute scales. Measurements of this effect can be used to constrain the masses of galaxy clusters with CMB data alone. Here we present a measurement of lensing of the CMB by galaxy clusters using data from the South Pole Telescope (SPT). We also develop a maximum likelihood approach to extract the CMB cluster lensing signal and validate the method on mock data. We quantify the effects on our analysis of several potential sources of systematic error and find that they generally act to reduce the best-fit cluster mass. It is estimated that this bias to lower cluster mass is roughly 0.85σ in units of the statistical error bar, although this estimate should be viewed as an upper limit. Furthermore, we apply our maximum likelihood technique to 513 clusters selected via their Sunyaev–Zeldovich (SZ) signatures in SPT data, and rule out the null hypothesis of no lensing at 3.1σ. The lensing-derived mass estimate for the full cluster sample is consistent with that inferred from the SZ flux: M 200,lens = 0.83 +0.38 -0.37 M 200,SZ (68% C.L., statistical error only).

79 ASTRONOMY AND ASTROPHYSICS↗