Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data clustering”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 199 records · Page 11
All sourcesNASA NTRSDOE OSTI
DOE OSTI2017

XRootD popularity on hadoop clusters

Performance data and metadata of the computing operations at the CMS experiment are collected through a distributed monitoring infrastructure, currently relying on a traditional Oracle database system. This paper shows how to harness Big Data architectures in order to improve the throughput and the efficiency of such monitoring. A large set of operational data - user activities, job submissions, resources, file transfers, site efficiencies, software releases, network traffic, machine logs - is being injected into a readily available Hadoop cluster, via several data streamers. The collected metadata is further organized running fast arbitrary queries; this offers the ability to test several Map&Reduce-based frameworks and measure the system speed-up when compared to the original database infrastructure. By leveraging a quality Hadoop data store and enabling an analytics framework on top, it is possible to design a mining platform to predict dataset popularity and discover patterns and correlations.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

Harnessing Data Movement in Virtual Clusters for In-Situ Execution

As a result of increasing data volume and velocity, Big Data science at exascale has shifted towards the in-situ paradigm, where large scale simulations run concurrently alongside data analytics. With in-situ, data generated from simulations can be processed while still in memory, thereby avoiding the slow storage bottleneck. However, running simulations and analytics together on shared resources will likely result in substantial contention if left unmanaged, as demonstrated in this work, leading to much reduced efficiency of simulations and analytics. Recently, virtualization technologies such as Linux containers have been widely applied to data centers and physical clusters to provide highly efficient and elastic resource provisioning for consolidated workloads including scientific simulations and data analytics. In this paper, we investigate to facilitate network traffic manipulation and reduce mutual interference on the network for in-situ applications in virtual clusters. In order to dynamically allocate the network bandwidth when it is needed, we adopt SARIMA-based techniques to analyze and predict MPI traffic issued from simulations. Although this can be an effective technique, the naïve usage of network virtualization can lead to performance degradation for bursty asynchronous transmissions within an MPI job. Here, we analyze and resolve this performance degradation in virtual clusters.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Nontargeted virus sequence discovery pipeline and virus clustering for metagenomic data

The analysis of large microbiome data sets holds great promise for the delineation of the biological and metabolic functioning of living organisms and their role in the environment. In the midst of this genomic puzzle, viruses, especially those that infect microbial communities, represent a major reservoir of genetic diversity with great impact on biogeochemical cycles and organismal health. Overcoming the limitations associated with virus detection directly from microbiomes can provide key insights into how ecosystem dynamics are modulated. Here, we present a computational protocol for accurate detection and grouping of viral sequences from microbiome samples. Our approach relies on an expanded and curated set of viral protein families used as bait to identify viral sequences directly from metagenomic assemblies. This protocol describes how to use the viral protein families catalog (~7 h) and recommended filters for the detection of viral contigs in metagenomic samples (~6 h), and it describes the specific parameters for a nucleotide-sequence-identity-based method of organizing the viral sequences into quasi-species taxonomic-level groups (~10 min).

Paez-Espino, David↗
DOE OSTI2022

Deprojection of X-ray data in galaxy clusters: confronting simulations with observations

ABSTRACT Numerical simulations with varying realism indicate an emergent principle−multiphase condensation and large cavity power occur when the ratio of the cooling time to the free-fall time (tcool/tff) falls below a threshold value close to 10. Observations indeed show cool-core signatures when this ratio falls below 20–30, but the prevalence of cores with tcool/tff ratio below 10 is rare as compared to simulations. In X-ray observations, we obtain projected spectra from which we have to infer radial gas density and temperature profiles. Using idealized models of X-ray cavities and multiphase gas in the core and 3D hydro jet-ICM simulations, we quantify the biases introduced by deprojection based on the assumption of spherical symmetry in determining tcool/tff. We show that while the used methods are able to recover the tcool/tff ratio for relaxed clusters, they have an uncertainty of a factor of 2−3 in systems containing large cavities (≳ 20 kpc). We also show that the mass estimates from these methods, in the absence of X-ray spectra close to the virial radius, suffer from a degeneracy between the virial mass (M200) and the concentration parameter (c) in the form of M200c2 ≈ constant. Additionally, the lack of soft-X-ray (≲ 0.5 keV) coverage and poor spatial resolution makes us overestimate min(tcool/tff) by a factor of few in clusters with min(tcool/tff) ≲ 5. This bias can largely explain the lack of cool-core clusters with min(tcool/tff) ≲ 5.

Sarkar, Kartick C. (ORCID:0000000277678472)↗
DOE OSTI2021

Quantum cluster algorithm for data classification

Abstract We present a quantum algorithm for data classification based on the nearest-neighbor learning algorithm. The classification algorithm is divided into two steps: Firstly, data in the same class is divided into smaller groups with sublabels assisting building boundaries between data with different labels. Secondly we construct a quantum circuit for classification that contains multi control gates. The algorithm is easy to implement and efficient in predicting the labels of test data. To illustrate the power and efficiency of this approach, we construct the phase transition diagram for the metal-insulator transition of VO 2 , using limited trained experimental data, where VO 2 is a typical strongly correlated electron materials, and the metallic-insulating phase transition has drawn much attention in condensed matter physics. Moreover, we demonstrate our algorithm on the classification of randomly generated data and the classification of entanglement for various Werner states, where the training sets can not be divided by a single curve, instead, more than one curves are required to separate them apart perfectly. Our preliminary result shows considerable potential for various classification problems, particularly for constructing different phases in materials.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2014

Precision growth index using the clustering of cosmic structures and growth data

We use the clustering properties of Luminous Red Galaxies (LRGs) and the growth rate data provided by the various galaxy surveys in order to constrain the growth index γ) of the linear matter fluctuations. We perform a standard χ{sup 2}-minimization procedure between theoretical expectations and data, followed by a joint likelihood analysis and we find a value of γ=0.56± 0.05, perfectly consistent with the expectations of the ΛCDM model, and Ω{sub m0} =0.29± 0.01, in very good agreement with the latest Planck results. Our analysis provides significantly more stringent growth index constraints with respect to previous studies, as indicated by the fact that the corresponding uncertainty is only ∼ 0.09 γ. Finally, allowing γ to vary with redshift in two manners (Taylor expansion around z=0, and Taylor expansion around the scale factor), we find that the combined statistical analysis between our clustering and literature growth data alleviates the degeneracy and obtain more stringent constraints with respect to other recent studies.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2018

HiSpatialCluster: A novel high‐performance software tool for clustering massive spatial points

Abstract In the era of big data, spatial clustering is a very important means for geo‐data analysis. When clustering big geo‐data such as social media check‐in data, geotagged photos, and taxi trajectory points, traditional spatial clustering algorithms are facing more challenges. On the one hand, existing spatial clustering tools cannot support the clustering of massive point sets; on the other hand, there is no perfect solution for self‐adaptive spatial clustering. In order to achieve clustering of millions or even billions of points adaptively, a new spatial clustering tool—HiSpatialCluster—was proposed, in which the CFSFDP (clustering by fast search and finding density peaks) idea to find cluster centers and the DBSCAN (density‐based spatial clustering of applications with noise) idea of density‐connect filtering for classification are introduced. The tool’s source codes and other resources have been released on Github, and experimental evaluation was performed through clustering massive taxi trajectory points and Flickr geotagged photos in Beijing, China. The spatial clustering results were compared with those through K‐means and DBSCAN as well. As a spatial clustering tool, HiSpatialCluster is expected to play a fundamental role in big geo‐data research. First, this tool enables clustering adaptively on massive point datasets with uneven spatial density distribution. Second, the density‐connect filter method is applied to generate homogeneous analysis units from geotagged data. Third, the tool is accelerated by both parallel CPU and GPU computing so that millions or even billions of points can be clustered efficiently.

Chen, Yiran↗
DOE OSTI2017

Blind Millimeter Line Emitter Search using ALMA Data Toward Gravitational Lensing Clusters

We present the results of a blind millimeter line emitter search using ALMA Band 6 data with a single-frequency tuning toward four gravitational lensing clusters (RXJ1347.5−1145, Abell S0592, MACS J0416.1−2403, and Abell 2744). We construct 3D signal-to-noise ratio (S/N) cubes with 60 and 100 MHz binning, and search for millimeter line emitters. We do not detect any line emitters with a peak S/N > 5, although we do find a line emitter candidate with a peak S/N ≃ 4.5. These results provide upper limits to the CO(3−2), CO(4−3), CO(5−4), and [C ii] luminosity functions at z ≃ 0.3, 0.7, 1.2, and 6, respectively. Because of the magnification effect of gravitational lensing clusters, the new data provide the first constraints on the CO and [C ii] luminosity functions at unprecedentedly low luminosity levels, i.e., down to ≲10{sup −3} – 10{sup −1} Mpc{sup −3} dex{sup −1} at L{sub CO}{sup ′}∼10{sup 8}--10{sup 10} K km s{sup −1} pc{sup 2} and ≲10{sup −3}–10{sup −2} Mpc{sup −3} dex{sup −1} at L{sub [CII]}∼10{sup 8}--10{sup 10} L{sub ⊙}, respectively. Although the constraints to date are not yet stringent, we find that the evolution of the CO and [C ii] luminosity functions are broadly consistent with the predictions of semi-analytical models. This study demonstrates that the wide observations with a single-frequency tuning toward gravitational lensing clusters are promising for constraining the CO and [C ii] luminosity functions.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2019

Weak Lensing Analysis of SPT selected Galaxy Clusters using Dark Energy Survey Science Verification Data

We present weak-lensing (WL) mass constraints for a sample of massive galaxy clusters detected by the South Pole Telescope (SPT) via the Sunyaev-Zel'dovich effect (SZE). We use griz imaging data obtained from the Science Verification (SV) phase of the Dark Energy Survey (DES) to fit the WL shear signal of 33 clusters in the redshift range 0.25 <= z <= 0.8 with NFW profiles and to constrain a four-parameter SPT mass-observable relation. To account for biases in WL masses, we introduce a WL mass to true mass scaling relation described by a mean bias and an intrinsic, lognormal scatter. We allow for correlated scatter within the WL and SZE mass-observable relations and use simulations to constrain priors on nuisance parameters related to bias and scatter from WL. We constrain the normalization of the zeta-M-500 relation, A(SZ) = 12.0(-6.7)(+2.6) when using a prior on the mass slope B-SZ from the latest SPT cluster cosmology analysis. Without this prior, we recover A(SZ) = 10.8(-5.2)(+2.3) and B-SZ = 1.30(-0.44)(+0.22). Results in both cases imply lower cluster masses than measured in previous work with and without WL, although the uncertainties are large. The WL derived value of B-SZ is approximate to 20 per cent lower than the value preferred by the most recent SPT cluster cosmology analysis. The method demonstrated in this work is designed to constrain cluster masses and cosmological parameters simultaneously and will form the basis for subsequent studies that employ the full SPT cluster sample together with the DES data.

cosmology: observations↗
NASA NTRS1992

Clustering of noisy image data using an adaptive neuro-fuzzy system

Identification of outliers or noise in a real data set is often quite difficult. A recently developed adaptive fuzzy leader clustering (AFLC) algorithm has been modified to separate the outliers from real data sets while finding the clusters within the data sets. The capability of this modified AFLC algorithm to identify the outliers in a number of real data sets indicates the potential strength of this algorithm in correct classification of noisy real data.

Pemmaraju, Surya↗
NASA NTRS2001

Enhancement of Tropical Land Cover Mapping with Wavelet-Based Fusion and Unsupervised Clustering of SAR and Landsat Image Data

The characterization and the mapping of land cover/land use of forest areas, such as the Central African rainforest, is a very complex task. This complexity is mainly due to the extent of such areas and, as a consequence, to the lack of full and continuous cloud-free coverage of those large regions by one single remote sensing instrument, In order to provide improved vegetation maps of Central Africa and to develop forest monitoring techniques for applications at the local and regional scales, we propose to utilize multi-sensor remote sensing observations coupled with in-situ data. Fusion and clustering of multi-sensor data are the first steps towards the development of such a forest monitoring system. In this paper, we will describe some preliminary experiments involving the fusion of SAR and Landsat image data of the Lope Reserve in Gabon. Similarly to previous fusion studies, our fusion method is wavelet-based. The fusion provides a new image data set which contains more detailed texture features and preserves the large homogeneous regions that are observed by the Thematic Mapper sensor. The fusion step is followed by unsupervised clustering and provides a vegetation map of the area.

LeMoigne, Jacqueline↗
DOE OSTI2018

Preliminary Results on Applying Nonparametric Clustering and Bayesian Consensus Clustering Methods to Multimodal Data

In this report, we present preliminary research into nonparametric clustering methods for multi-source imagery data and quantifying the performance of these models. In many domain areas, data sets do not necessarily follow well-defined and well-known probability distributions, such as the normal, gamma, and exponential. This is especially true when combining data from multiple sources describing a common set of objects (which we call multimodal analysis), where the data in each source can follow different distributions and need to be analyzed in conjunction with one another. This necessitates nonparametric density estimation methods, which allow the data to better dictate the distribution of the data. One prominent example of multimodal analysis is multimodal image analysis, when we analyze multiple images taken using different radar systems of the same scene of interest. We develop uncertainty analysis methods, which are inherent in the use of probabilistic models but often not taken advance of, to assess the performance of probabilistic clustering methods used for analyzing multimodal images. This added information helps assess model performance and how much trust decision-makers should have in the obtained analysis results. The developed methods illustrate some ways in which uncertainty can inform decisions that arise when designing and using machine learning models.

42 ENGINEERING↗