Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data analysis cluster”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

An Integrated Platform for Collaborative Data Analytics

While collaboration among data scientists is a key to organizational productivity, data analysts face significant barriers to achieving this end, including data sharing, accessing and configuring the required computational environment, and a unified method of sharing knowledge. Each of these barriers to collaboration is related to the fundamental question of knowledge management “how can organizations use knowledge more effectively?”. In this paper, we consider the problem of knowledge management in collaborative data analytics and present ShareAL, an integrated knowledge management platform, as a solution to that problem. The ShareAL platform consists of three core components: a full stack web application, a dashboard for analyzing streaming data and a High Performance Computing (HPC) cluster for performing real time analysis. Prior research has not applied knowledge management to collaborative analytics or developed a platform with the same capabilities as ShareAL. ShareAL overcomes the barriers data scientists face to collaboration by providing intuitive sharing of data and analytics via the web application, a shared computing environment via the HPC cluster and knowledge sharing and collaboration via a real time messaging application.

Oesch, T↗
DOE OSTI2020

Bi-cross validation of spectral clustering hyperparameters

One challenge impeding the analysis of terabyte scale X-ray scattering data from the Linac Coherent Light Source (LCLS) is determining the number of clusters required for the execution of traditional clustering algorithms. Here, we demonstrate that the previous work using bi-cross validation to determine the number of singular vectors directly maps to the spectral clustering problem of estimating both the number of clusters and hyperparameter values. Here by applying this method to LCLS X-ray scattering data enables the identification of dropped shots without manually setting boundaries on detector fluence and provides a path toward identifying rare and anomalous events.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2025

Erratum: Centrality-Dependent Modification of Jet-Production Rates in Deuteron-Gold Collisions at $\sqrt{𝑠_{𝑁⁢𝑁}}$ = 200 GeV [Phys. Rev. Lett. 116, 122301 (2016)]

We previously reported jet-production rates in 𝑝 + 𝑝 and 𝑑 + Au collisions at $\sqrt{𝑠_{𝑁⁢𝑁}}$ = 200 GeV recorded in 2008 with the PHENIX detector at the Relativistic Heavy Ion Collider. Subsequently, we found that electronic noise in the electromagnetic calorimeter was not completely removed by the tower-timing cuts used in the analysis. We have conducted an independent re-analysis of the data that eliminated noisy towers by removing them completely from the analysis. These cuts remove any contributions from instrumental noise, but reduce the overall acceptance for electromagnetic clusters by ≈16% and hence the overall statistical power of the data. Finally, this re-analysis found that instrumental noise in the electromagnetic calorimeter led to an artificial increase in the jet yield for both the 𝑝 + 𝑝 and 𝑑 + Au collisions.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2020

Metabolic Pathway Engineering

Modern microbial and enzyme engineering and their advancement are increasingly dependent on the marriage of a wide range of sophisticated technologies. For students entering the field of biotechnology, the outlook is indeed daunting. Expertise at levels beyond that of simple familiarity will be needed to conduct competitive research. It goes without saying that to be competitive, all new researchers in this field will need basic preparation in molecular biology, biochemistry, and genetics. Further, experience working with concepts and experimental tools in enzyme biochemistry and kinetics, gene editing, computational metabolic pathway modeling, experimental pathway flux analysis, and computational clustering tools to process complex data sets will be vital for success. We speculate that most biotechnology researchers in early career at this time will build teams of collaborators to address these disparate science fields rather than attempt to become experts in one lab.

BASIC BIOLOGICAL SCIENCES,BIOMASS FUELS↗
DOE OSTI2020

A Satellite-Based Estimate of Convective Vertical Velocity and Convective Mass Flux: Global Survey and Comparison with Radar Wind Profiler Observations

Convective vertical velocity (w c ) and convective mass flux (M c ) lie at the heart of GCM cumulus parameterizations, but few observations of these critical parameters are available. In this paper, we develop and evaluate a novel, satellite-based method for estimating profiles of w c and M c . Here, comparisons with collocated ground-based radar wind profiler (RWP) observations show that satellite estimated median w c is slightly greater than the RWP estimates, but they show solid agreement when compared at the 95th percentiles (intense updrafts). RWP-derived and satellite estimated M c are broadly comparable in the lower and middle troposphere, with some differences in the upper troposphere due to differences in convective core sampling. A k-means cluster analysis of multiple years of w c data shows that convective characteristics are distinctly different among extratropical convection, tropical land convection, and tropical oceanic convection. Tropical land convection is significantly more intense and more variable than the oceanic counterpart.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

ENSO Impact on Winter Precipitation in the Southeast United States through a Synoptic Climate Approach

The ENSO impact on winter precipitation in the Southeast United States was analyzed from the perspective of daily weather types (WTs). We calculated the dynamic contribution associated with the change in frequency of the WTs and the thermodynamic contribution due to changes in the spatial patterns of the environmental fields of the WTs. Six WTs were obtained using a k-means clustering analysis of 850 hPa winds in reanalysis data from November to February of 1948–2022. All the WTs can only persist for a few days. The most frequent winter weather type is WT1 (shallow trough in Eastern U.S.), which can persist or likely transfer to WT4 (Mississippi River Valley ridge). WT1 becomes less frequent in El Niño years, while the frequency of WT4 does not change much. WTs 2–6 correspond to a loop of eastward propagating waves with troughs and ridges in the mid-latitude westerlies. Three WTs with a deep trough in the Southeast U.S., which are WT2 (east coast trough), WT3 (off east coast trough) and WT6 (plains trough), become more frequent in El Niño years. The more frequent deep troughs (WTs 2, 3 and 6) and less frequent shallow trough (WT1) result in above-normal precipitation in the coastal Southeast U.S. in the winter of El Niño years. WT5 (off coast Carolina High), with maximum precipitation extending from Mississippi Valley to the Great Lakes, becomes less frequent in El Niño years, which corresponds to the below-normal precipitation from the Great Lakes to Upper Mississippi and Ohio River Valley in El Niño years, and vice versa in La Niña years. The relative contribution of the thermodynamic and dynamic contribution is location dependent. On the east coast, the two contributions are similar in magnitude.

Qian, Jian-Hua↗
DOE OSTI2020

Constraining the masses of high-redshift clusters with weak lensing: Revised shape calibration testing for the impact of stronger shears and increased blending

Weak lensing measurements suffer from well-known shear estimation biases, which can be partially corrected for with the use of image simulations. Here we present an analysis of simulated images that mimic Hubble Space Telescope/Advance Camera for Surveys observations of high-redshift galaxy clusters, including cluster specific issues such as non-weak shear and increased blending. Our synthetic galaxies have been generated to have similar observed properties as the background-selected source samples studied in the real images. First, we used simulations with galaxies placed on a grid to determine a revised signal-to-noise-dependent (S/N KSB ) correction for multiplicative shear measurement bias, and to quantify the sensitivity of our KSB+ bias calibration to mismatches of galaxy or PSF properties between the real data and the simulations. Next, we studied the impact of increased blending and light contamination from cluster and foreground galaxies, finding it to be negligible for high-redshift (z > 0.7) clusters, whereas shear measurements can be affected at the ~1% level for lower redshift clusters given their brighter member galaxies. Finally, we studied the impact of fainter neighbours and selection bias using a set of simulated images that mimic the positions and magnitudes of galaxies in Cosmic Assembly Near-IR Deep Extragalactic Legacy Survey (CANDELS) data, thereby including realistic clustering. While the initial SExtractor object detection causes a multiplicative shear selection bias of –0.028 ± 0.002, this is reduced to –0.016 ± 0.002 by further cuts applied in our pipeline. Given the limited depth of the CANDELS data, we compared our CANDELS-based estimate for the impact of faint neighbours on the multiplicative shear measurement bias to a grid-based analysis, to which we added clustered galaxies to even fainter magnitudes based on Hubble Ultra Deep Field data, yielding a refined estimate of ~ –0.013. Our sensitivity analysis suggests that our pipeline is calibrated to an accuracy of ~0.015 once all corrections are applied, which is fully sufficient for current and near-future weak lensing studies of high-redshift clusters. As an application, we used it for a refined analysis of three highly relaxed clusters from the South Pole Telescope Sunyaev-Zeldovich survey, where we now included measurements down to the cluster core (r > 200 kpc) as enabled by our work. Compared to previously employed scales (r > 500 kpc), this tightens the cluster mass constraints by a factor 1.38 on average.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Identifying Sample Provenance From SEM/EDS Automated Particle Analysis via Few-Shot Learning Coupled With Similarity Graph Clustering

Automated particle analysis (APA) provides a vast amount of compositional data via energy-dispersive X-ray spectroscopy along with size and shape data via scanning electron microscopy for individual particles in a sample. In many instances, APA data are leveraged to support identification of the source of a sample based on the detection of particles of a specific composition. Often, the particles that provide context make up a minuscule portion of the sample. Additionally, the interpretation of complex samples can be difficult due to the diversity of compositions both in the mixture and within a particle. In this work, we demonstrate a method to compute and cluster similarity graphs that describe inter-particle relationships within a sample using a multi-modal few-shot learning neural network. Here, as a proof-of-concept, we show that samples known to have been exposed to gunshot residue can be distinguished from samples occasionally mistaken for gunshot residue. Our workflow builds upon standard APA techniques and data processing methods to unveil additional information in a readily interpretable and quantitatively comparable format.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2020

Blinding multiprobe cosmological experiments

ABSTRACT The goal of blinding is to hide an experiment’s critical results – here the inferred cosmological parameters – until all decisions affecting its analysis have been finalized. This is especially important in the current era of precision cosmology, when the results of any new experiment are closely scrutinized for consistency or tension with previous results. In analyses that combine multiple observational probes, like the combination of galaxy clustering and weak lensing in the Dark Energy Survey (DES), it is challenging to blind the results while retaining the ability to check for (in)consistency between different parts of the data. We propose a simple new blinding transformation, which works by modifying the summary statistics that are input to parameter estimation, such as two-point correlation functions. The transformation shifts the measured statistics to new values that are consistent with (blindly) shifted cosmological parameters while preserving internal (in)consistency. We apply the blinding transformation to simulated data for the projected DES Year 3 galaxy clustering and weak lensing analysis, demonstrating that practical blinding is achieved without significant perturbation of internal-consistency checks, as measured here by degradation of the χ2 between the data and best-fitting model. Our blinding method’s performance is expected to improve as experiments evolve to higher precision and accuracy.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2025

Enabling Innovative Analysis on Heterogeneous Clusters through HTCdaskgateway

High energy particle (HEP) physics research is going through fundamental changes as we move to collect larger amounts of data from the Large Hadron Collider (LHC). Analysis facilities and distributed computing, through HTCs, have come together to create the next pythonic generation of analysis by utilizing HTCdaskgateway, a Dask gateway extension, allowing users to spawn workers compatible with both their analysis and heterogeneous clusters in line with authentication requirements. This is enabling physicists to engage with scientific python in ways they had not before because of domain specific C++ tools. An example of HTCdaskgateway’s use is Fermilab’s Elastic Analysis Facility.

Chavez, Elise [U. Wisconsin, Madison (main)]↗
DOE OSTI2026

Investigating the Determinants of Household Capabilities Burden During Power Outages: The Case of Winter Storm Uri

Existing research primarily uses census data to identify the vulnerability of communities to hazards. These vulnerability indices provide aggregated data and are not hazard-specific nor well-validated with post-event data. In contrast, our study uses household survey data (n=1065) to understand which Texan households suffered the greatest loss of their capabilities due to power outages and other utility service disruptions during Winter Storm Uri. Inspired by the Capabilities Approach, our measures of burden include the number of household capability types disrupted during the outages (e.g., cooking, heating, refrigeration), the severity of impact for each disrupted capability, and the additional time and financial costs of coping with these disruptions. We perform a clustering analysis, and find two distinct groups in our data, consisting of ‘lesser burden' and ‘heavier burden' households. Results indicate that the households experiencing the heaviest capabilities burden were most likely to experience longer power outages and the loss of water services. They were also more likely to have a Hispanic-Latino household member, lack access to a generator, live in a rented home, have larger households with more young children, fewer adults over 65, lower household incomes, been impacted by the COVID-19 pandemic, and more family characteristics that made life harder. We also fit a logistic regression model to assess the role of outage, household, and community characteristics in predicting differences in capabilities burden. Our results offer insights into enumerating the consequences of utility service disruptions on households, which can inform more targeted and equitable resilience strategies.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2023

Data-driven analysis and prediction of wastewater treatment plant performance: Insights and forecasting for sustainable operations

Here this study presents a comprehensive performance and forecasting analysis of the As-Samra wastewater treatment plant (WWTP) in Jordan, with two main objectives. Firstly, a thorough evaluation of the plant's performance is conducted. The analysis involves independently assessing historical operational conditions, plant production, and their statistical correlations using various statistical techniques. The second objective focuses on developing a data-driven forecasting approach to predict the plant's production one month in advance, using multiple machine learning models. The results highlight the effectiveness of principal component analysis (PCA) in simplifying operational data, revealing distinct operational clusters, and identifying seasonal production patterns while showing correlations between operational conditions and overall power production. The support vector machine (SVM) forecasting model emerged as the top performer, showcasing the potential of a hybrid forecasting approach. The findings offer valuable perspectives for enhancing operational efficiency, refining production planning, and ultimately improving the environmental impact of the plant.

42 ENGINEERING↗
DOE OSTI2022

Poisson hurdle model-based method for clustering microbiome features

Abstract Motivation High-throughput sequencing technologies have greatly facilitated microbiome research and have generated a large volume of microbiome data with the potential to answer key questions regarding microbiome assembly, structure and function. Cluster analysis aims to group features that behave similarly across treatments, and such grouping helps to highlight the functional relationships among features and may provide biological insights into microbiome networks. However, clustering microbiome data are challenging due to the sparsity and high dimensionality. Results We propose a model-based clustering method based on Poisson hurdle models for sparse microbiome count data. We describe an expectation–maximization algorithm and a modified version using simulated annealing to conduct the cluster analysis. Moreover, we provide algorithms for initialization and choosing the number of clusters. Simulation results demonstrate that our proposed methods provide better clustering results than alternative methods under a variety of settings. We also apply the proposed method to a sorghum rhizosphere microbiome dataset that results in interesting biological findings. Availability and implementation R package is freely available for download at https://cran.r-project.org/package=PHclust. Supplementary information Supplementary data are available at Bioinformatics online.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2024

Unveiling Temporal Performance Deviation: Leveraging Clustering in Microservices Performance Analysis

As the market for cloud computing continues to grow, an increasing number of users are deploying applications as microservices. The shift introduces unique challenges in identifying and addressing performance issues, particularly within large and complex infrastructures. To address this challenge, we propose a methodology that unveils temporal performance deviations in microservices by clustering containers based on their performance characteristics at different time intervals. Showcasing our methodology on the Alibaba dataset, we found both stable and dynamic performance patterns, providing a valuable tool for enhancing overall performance and reliability in modern application landscapes.

Clustering↗
DOE OSTI2020

Machine learning analysis of perovskite oxides grown by molecular beam epitaxy

Reflection high-energy electron diffraction (RHEED) is a ubiquitous in situ molecular beam epitaxial (MBE) characterization tool. Although RHEED can be a powerful means for crystal surface structure determination, it is often used as a static qualitative surface characterization method at discrete intervals during a growth. A full analysis of RHEED data collected during the entirety of MBE growths is made possible using principle component analysis (PCA) and $\textit{k}$-means clustering to examine significant boundaries that occur in the temporal clusters grouped from RHEED data and identify statistically significant patterns. This process is applied to data from homoepitaxial SrTiO 3 growths, heteroepitaxial SrTiO 3 grown on scandate substrates, BaSnO 3 films grown on SrTiO 3 substrates, and LaNiO 3 films grown on SrTiO 3 substrates. We report this analysis may provide additional insights into the surface evolution and transitions in growth modes at precise times and depths during growth, and that video archival of an entire RHEED image sequence may be able to provide more insight and control overgrowth processes and film quality.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Molecular diversity and evolution of far-red light-acclimated photosystem I

The need to acclimate to different environmental conditions is central to the evolution of cyanobacteria. Far-red light (FRL) photoacclimation, or FaRLiP, is an acclimation mechanism that enables certain cyanobacteria to use FRL to drive photosynthesis. During this process, a well-defined gene cluster is upregulated, resulting in changes to the photosystems that allow them to absorb FRL to perform photochemistry. Because FaRLiP is widespread, and because it exemplifies cyanobacterial adaptation mechanisms in nature, it is of interest to understand its molecular evolution. Here, we performed a phylogenetic analysis of the photosystem I subunits encoded in the FaRLiP gene cluster and analyzed the available structural data to predict ancestral characteristics of FRL-absorbing photosystem I. The analysis suggests that FRL-specific photosystem I subunits arose relatively late during the evolution of cyanobacteria when compared with some of the FRL-specific subunits of photosystem II, and that the order Nodosilineales, which include strains like Halomicronema hongdechloris and Synechococcus sp. PCC 7335, could have obtained FaRLiP via horizontal gene transfer. We show that the ancestral form of FRL-absorbing photosystem I contained three chlorophyll f-binding sites in the PsaB2 subunit, and a rotated chlorophyll a molecule in the A0B site of the electron transfer chain. Along with our previous study of photosystem II expressed during FaRLiP, these studies describe the molecular evolution of the photosystem complexes encoded by the FaRLiP gene cluster.

ancestral sequence reconstruction↗
DOE OSTI2025

Galaxy cluster matter profiles - I. Self-similarity, mass calibration, and observable-mass relation validation employing cluster mass posteriors

We present a study of the weak lensing inferred matter profiles ΔΣ(R) of 698 South Pole Telescope (SPT) thermal Sunyaev-Zel’dovich effect (tSZE) selected and MCMF optically confirmed galaxy clusters in the redshift range 0.25 < z < 0.94 that have associated weak gravitational lensing shear profiles from the Dark Energy Survey (DES). Rescaling these profiles to account for the mass dependent size and the redshift dependent density produces average rescaled matter profiles ΔΣ(R/R200c)/(ρcritR200c) with a lower dispersion than the unscaled ΔΣ(R) versions, indicating a significant degree of self-similarity. Galaxy clusters from hydrodynamical simulations also exhibit matter profiles that suggest a high degree of self-similarity, with RMS variation among the average rescaled matter profiles with redshift and mass falling by a factor of approximately six and 23, respectively, compared to the unscaled average matter profiles. We employed this regularity in a new Bayesian method for weak lensing mass calibration that employs the so-called cluster mass posterior P(M200|ζ̂, λ̂, z), which describes the individual cluster masses given their tSZE (ζ̂) and optical (λ̂, z) observables. This method enables simultaneous constraints on richness λ-mass and tSZE detection significance ζ-mass relations using average rescaled cluster matter profiles. We validated the method using realistic mock datasets and present observable-mass relation constraints for the SPT×DES sample, where we constrained the amplitude, mass trend, redshift trend, and intrinsic scatter. Our observable-mass relation results are in agreement with the mass calibration derived from the recent cosmological analysis of the SPT×DES data based on a cluster-by-cluster lensing calibration. Our new mass calibration technique offers a higher efficiency when compared to the single cluster calibration technique. We present new validation tests of the observable-mass relation that indicate the underlying power-law form and scatter are adequate to describe the real cluster sample but that also suggest a redshift variation in the intrinsic scatter of the λ-mass relation may offer a better description. In addition, the average rescaled matter profiles offer high signal-to-noise ratio (S/N) constraints on the shape of real cluster matter profiles, which are in good agreement with available hydrodynamical ΛCDM simulations. This high S/N profile contains information about baryon feedback, the collisional nature of dark matter, and potential deviations from general relativity.Key words: gravitational lensing: weak / galaxies: clusters: general / large-scale structure of Universe

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Assessment of Outliers in Alloy Datasets Using Unsupervised Techniques

We report advancements in data analytics techniques have enabled complex, disparate datasets to be leveraged for alloy design. Identifying outliers in a dataset can reduce noise, identify erroneous and/or anomalous records, prevent overfitting, and improve model assessment and optimization. In this work, two alloy datasets (9-12% Cr ferritic martensitic steels, and austenitic stainless steels) have been assessed for outliers using unsupervised techniques and supplemented with domain knowledge. Principal component analysis and k-means clustering were applied to the data, and points were assessed as outliers based on their distance away from other points in the cluster and from other points in the dataset. The outlier characteristics were investigated to determine both cluster-specific and overall trends in the properties of the outlier points. The approach demonstrated here is extensible to other alloy datasets for outlier identification and evaluation to improve the reliability of machine learning and modeling predictions for advanced alloy design.

36 MATERIALS SCIENCE↗