Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Self Organizing Map”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Predicting Rare Earth Element Potential in Produced and Geothermal Waters of the United States via Emergent Self-Organizing Maps

This work applies emergent self-organizing map (ESOM) techniques, a form of machine learning, in the multidimensional interpretation and prediction of rare earth element (REE) abundance in produced and geothermal waters in the United States. Visualization of the variables in the ESOM trained using the input data shows that each REE, with the exception of Eu, follows the same distribution patterns and that no single parameter appears to control their distribution. Cross-validation, using a random subsample of the starting data and only using major ions, shows that predictions are generally accurate to within an order of magnitude. Using the same approach, an abridged version of the U.S. Geological Survey Produced Waters Database, Version 2.3 (which includes both data from produced and geothermal waters) was mapped to the ESOM and predicted values were generated for samples that contained enough variables to be effectively mapped. Results show that in general, produced and geothermal waters are predicted to be enriched in REEs by an order of magnitude or more relative to seawater, with maximum predicted enrichments in excess of 1000-fold. Cartographic mapping of the resulting predictions indicates that maximum REE concentrations exceed values in seawater across the majority of geologic basins investigated and that REEs are typically spatially co-associated. The factors causing this co-association were not determined from ESOM analysis, but based on the information currently available, REE content in produced and geothermal waters is not directly controlled by lithology, reservoir temperature, or salinity.

Engle, Mark A. (ORCID:0000000152587374)↗

Enhancing weak lensing redshift distribution characterization by optimizing the Dark Energy Survey Self-Organizing Map Photo-z method

Characterization of the redshift distribution of ensembles of galaxies is pivotal for large scale structure cosmological studies. In this work, we focus on improving the Self-Organizing Map (SOM) methodology for photometric redshift estimation (SOMPZ), specifically in anticipation of the Dark Energy Survey Year 6 (DES Y6) data. This data set, featuring deeper and fainter galaxies than DES Year 3 (DES Y3), demands adapted techniques to ensure accurate recovery of the underlying redshift distribution. We investigate three strategies for enhancing the existing SOM-based approach used in DES Y3: 1) Replacing the Y3 SOM algorithm with one tailored for redshift estimation challenges; 2) Incorporating $\textit{g}$-band flux information to refine redshift estimates (i.e. using $\textit{griz}$ fluxes as opposed to only $\textit{riz}$); 3) Augmenting redshift data for galaxies where available. These methods are applied to DES Y3 data, and results are compared to the Y3 fiducial ones. Our analysis indicates significant improvements with the first two strategies, notably reducing the overlap between redshift bins. By combining strategies 1 and 2, we have successfully managed to reduce redshift bin overlap in DES Y3 by up to 66$\%$. Conversely, the third strategy, involving the addition of redshift data for selected galaxies as an additional feature in the method, yields inferior results and is abandoned. Our findings contribute to the advancement of weak lensing redshift characterization and lay the groundwork for better redshift characterization in DES Year 6 and future stage IV surveys, like the Rubin Observatory.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Dark Energy Survey Year 6 results: Clustering redshifts and importance sampling of self-organized-maps 𝑛⁡(𝑧) realizations for 3 × 2 ⁢pt samples

This work is part of a series establishing the redshift framework for the 3 × 2 ⁢pt analysis of the Dark Energy Survey Year 6 (DES Y6). For DES Y6, photometric redshift distributions are estimated using self-organizing maps (SOMs), calibrated with spectroscopic and many-band photometric data. To overcome limitations from color-redshift degeneracies and incomplete spectroscopic coverage, we enhance this approach by incorporating clustering-based redshift constraints (clustering-z, or WZ) from angular cross-correlations with BOSS and eBOSS galaxies and eBOSS quasar samples. We define a WZ likelihood and apply importance sampling to a large ensemble of SOM-derived 𝑛⁡(𝑧) realizations, selecting those consistent with the clustering measurements to produce a posterior sample for each lens and source bin. The analysis uses angular scales corresponding to 1.5–5 Mpc to optimize signal-to-noise ratio while mitigating modeling uncertainties and marginalizes over redshift-dependent galaxy bias and other systematics informed by the N-body simulation CARDINAL . While a sparser spectroscopic reference sample limits WZ constraining power at 𝑧 >1.1, particularly for source bins, we demonstrate that combining SOM with WZ improves redshift accuracy and enhances the overall cosmological constraining power of DES Y6. As a result, we estimate an improvement in 𝑆 8 of approximately 10% for cosmic shear and 3 ×2⁢pt analysis, primarily due to the WZ calibration of the source samples.

Cosmological parameters↗

Connecting Large‐Scale Meteorological Patterns to Extratropical Cyclones in CMIP6 Climate Models Using Self‐Organizing Maps

Abstract Extratropical cyclones (ETCs) are responsible for the majority of cool‐season extreme events in the northeastern United States (NEUS), often leading to high‐impact weather conditions that can have wide‐ranging socioeconomic impacts. Evaluating the ability of climate models to adequately simulate ETC dynamics is essential for improving model performance and increasing confidence in future projections used by stakeholders and policymakers. ETCs are traditionally studied using techniques such as case studies and synoptic typing, however, these approaches can be time‐consuming, require subjective analysis, and do not necessarily identify the coincident large‐scale meteorological patterns (LSMPs). Here, we apply self‐organizing maps (SOMs) as an automated machine‐learning approach to characterize the LSMPs and associated frequency and intensity of discrete ETC events over NEUS. The dominant patterns of geopotential height variability are identified through SOM analysis of five reanalysis products during the last four decades. ETC events are tracked using TempestExtremes and are integrated with SOMs to classify the accumulated cyclone activity (ACA) associated with each pattern. We then evaluate the skill of CMIP6 historical experiments in simulating the LSMPs and ETC events identified in the SOM. Our results identify a robust bias toward more zonal patterns, with models struggling to reproduce the more amplified patterns typically associated with the highest cyclone activity. While model resolution has some impact on simulation credibility, model configuration appears to be more important in LSMP representation. The vast majority of CMIP6 models produce too few ETCs, although model errors are distributed around historical reanalyses when ACA is normalized by storm frequency.

54 ENVIRONMENTAL SCIENCES↗

Efficient Clustering of Software Vulnerabilities using Self Organizing Map (SOM)

The common vulnerabilities and exposures (CVE) database was created with a mission to ``identify, define, and catalog publicly disclosed cybersecurity vulnerabilities''. This rich body of information can be used to enable rapid and efficient response to secure and defend cyber operations and protect critical cyber infrastructure. The main goal of this paper is to develop a visual analytics tool to enable deep analysis of CVEs using unsupervised clustering techniques. We enhance our analysis by first mapping CVEs to hierarchical-classes in Common Weakness Enumeration (CWE) using information in the National Vulnerability Database (NVD). Both the mapping and the numerical representation of CVEs are enabled by V2W-BERT, which uses natural language processing of the extensive information in NVD to generate a large tabular database of 137,226 CVE entries from 1999 to 2020, where each CVE is represented by a vector of 768 numerical features. The vectorized data is processed by Self-Organizing Maps (SOM), which is an unsupervised machine learning technique for dimensionality reduction, visual representation and clustering. Using a Torus map of 6417 units, we achieve ~10-fold data compression of ~140k CVEs using SOM. The trained map is further clustered using standard K-means clustering into 138 clusters of CVEs. We conducted a brief investigation of the rich mapping of CVEs to best-matching-units to K-means clusters, as well as CVEs to CWEs. For example, this novel mapping provided insight into the role of CWE-59 and CWE-264 in several CVEs that is otherwise hard to explore in the original data. We conclude that our this novel approach will not only enable deep analysis of the complex relationships between CVEs and CWEs, but also a mechanism to quickly respond to and design mitigation actions for rapidly evolving vulnerabilities that have not been mapped to existing CWEs.

Panchal, Khyati↗

Lagrangian Characterization of Surface Transport From the Equatorial Atlantic to the Caribbean Sea Using Climatological Lagrangian Coherent Structures and Self‐Organizing Maps

Abstract This study presents an assessment of the transport of suspended material by surface ocean currents, which have a critical role in determining the connectivity and distribution of living and non‐living material. Lagrangian experiments reveal pathways from the Equatorial Atlantic to 10 strategic regions within the Caribbean Sea, determined by considering the space‐time variability of climatological Lagrangian Coherent Structures, which act as recurrent attracting pathways and transport barriers. Due to windage or Stokes drift, wind forcing is a significant factor in determining the spatial locations where particles cluster and the time needed to reach the Caribbean from the Equatorial Atlantic. Pathways shift westward within the Caribbean and take less time to arrive with increasing wind influence. Depending on the wind effect, the particles show higher confluence in different areas of the Caribbean. A case study is presented for the Mexican Caribbean nearshore area, isolated from ocean‐current trajectories. Here, wind weakens the transport barrier responsible for this isolation and causes particle confluence toward that region. Spatial patterns of the Eulerian velocity identified through Self‐Organizing Maps, with time dependence given by their best matching units, can reproduce the characteristic Lagrangian patterns of surface current climate variability. Our study demonstrates the application of tools from dynamical systems and unsupervised neural networks to understand Lagrangian patterns and identify the processes that drive them. These findings improve our understanding of transport mechanisms of suspended material by surface ocean currents in the Western Atlantic and the Caribbean Sea, which is essential for managing and conserving marine ecosystems.

Allende‐Arandía, Ma. Eugenia↗

Detection of Anomalies in Gamma Background Radiation Data with K-Means and Self-Organizing Map Clustering Algorithms (Consortium on Nuclear Security Technologies (CONNECT) Q1 Report)

Environmental screening of gamma radiation consists of detecting weak nuisance and anomaly signal in the presence of strong and highly varying background. In a typical scenario, a mobile detector-spectrometer continuously measures gamma radiation spectra in short, e.g., one-second, signal acquisition intervals. The measurement data is a 2D matrix, where one dimension is gamma ray energy, and the other dimension is the number of measurements or total time. In principle, gamma radiation sources can be detected and identified from the measured data by their unique spectral lines. Detecting sources from data measured in a search scenario is difficult due to the highly varying background because of naturally occurring radioactive material (NORM), and low signal-to-noise ratio (S/N) of spectral signal measured during one-second acquisition intervals. The objective of this work is to explore unsupervised machine learning (ML) algorithms for detection and identification of weak nuisances and anomalies events in the presence of highly fluctuating background. The challenge is that spectral lines of isotopes are difficult to observe in one-second measurements. Averaging over the entire measurement campaign data set reveals spectral lines of most common background isotopes. Spectral lines of orphan sources, which might appear only in a few measurements during the campaign, will be washed out if averaging is performed over the entire measurement data set. The approach we have explored consists of extracting one-second measurements containing weak spectral features through data clustering. Averaging one-second spectra in a cluster should reveal the presence of anomaly sources. We created two ML models using K-means clustering and Neural Network Self-organizing Map (SOM). Performance of these ML models was benchmarked using search data. One data set contained 137 Cs source, and another dataset contained 131 I source.

61 RADIATION PROTECTION AND DOSIMETRY↗

Self-Organizing Map-Based Resilience Quantification and Resilient Control of Distribution Systems Under Extreme Events

Due to climate change, extreme weather events are occurring more frequently and with increasing impact. This trend poses a significant challenge for distribution system operators (DSO) to ensure that there is uninterrupted power supply to critical loads in their networks. To embed resilience into DSO's decision-making, resilience needs to be first quantified and then integrated into the system-level optimization. Therefore, this paper first develops a novel self-organizing map (SOM) based method (called SomRes) to quantify the time-varying resilience index of a system that can leverage the powerful classification property of SOMs and removes some of the disadvantages of subjective weight assignment methods. Using SomRes, a resilient resource allocation and operational dispatch algorithm is further developed to enhance system resilience against extreme events by considering the SomRes resilience index directly as the feedback. Here, the proposed resilience quantification approach is benchmarked with a state-of-the-art approach and the efficacy of the proposed resilient dispatch algorithm is demonstrated through several deterministic and statistical case studies on the IEEE 123-bus distribution system. Simulation studies show that the proposed SomRes quantification method is an appropriate indicator of system resilience, and the resilient resource allocation and dispatch strategy can significantly reduce critical load shedding under varying event propagation scenarios.

24 POWER TRANSMISSION AND DISTRIBUTION↗

A procedure for rule extraction from a Self-Organising plasma disruption predictor for JET

In a previous paper, a Self-Organizing Map had proven to be able to identify the regions of the plasma operative space characterizing the pre-disruptive phase at JET without relying on any a priori information. One of the strengths of this disruption predictor lies in its inherent self-organization capability. The Self-Organizing Map discovers non-trivial relationships and captures the complicated interplay of device diagnostics on the internal plasma states directly from the experimental data. Moreover, the provided model allows the visualization of high-dimensional plasma parameters and facilitates easy interrogation of the model to understand the reasons behind its correlations. In this paper, an additional step is taken towards the interpretability of models for predicting disruptions by training a Decision Tree to classify the plasma states according to the interpretation provided by the Self-Organizing Map (stable or at high risk of disruptions). The Decision tree provides a set of rules which describe the transition of the plasma towards the pre-disruptive phase as visualized in the Self-Organizing Map. The obtained rules for the database explored in the study identify four regions in the map, two of which are at risk of disruption. These regions correspond to partitions of a 3D space based on the peaking factors of the core and divertor radiation, as well as the Locked Mode. The agreement between the Self-Organizing Map answers and the rules supplied by the Decision Tree is confirmed by the comparison of the performance exhibited by the two models in the prediction of disruptions.

Setzu, Samuele [Univ. of Cagliari, Monserrato, Cag↗

Characterizing vertical upper ocean temperature structures in the European Arctic through unsupervised machine learning

In-situ observations of subsurface ocean temperatures are, in many regions, inconsistently distributed in time and space. These spatio-temporal inconsistencies in the observational network lead to difficulties in utilizing those observations effectively for ocean model evaluation or understanding larger-scale ocean characteristics. Model accuracy of subsurface ocean characteristics is especially important within regions that contain complex ocean structures. One such region is the European Arctic which not only contains several types of water masses with unique characteristics, but also wintertime sea ice coverage and complex bathymetry. This study presents an unsupervised neural networking technique that can be used in combination with traditional ocean model evaluation techniques to provide additional information on the accuracy of modeled vertical ocean temperature profiles. Self-organizing maps is an unsupervised machine learning technique that we apply to approximately twenty thousand Argo and CTD temperature profiles from 2012 to 2020 in the European Arctic to categorize the observed vertical ocean temperature structures in the top 150 m. The observed ocean profile categories, or neurons, defined by the self-organizing map show strong spatial and temporal dependencies. We then use the neuron weights, or the learned temperature profile structure of each neuron, to validate the spatial and temporal variability of modeled vertical temperature structures. This analysis gives us new insights about the model’s capabilities to reproduce specific vertical structures of the top-most ocean layer within different regions and seasons. Mapping modeled ocean temperature profiles onto the neuron-space of the observationally-defined self organized map highlights the potential of this method to advance our understanding of model deficiencies in that region.

54 ENVIRONMENTAL SCIENCES↗

Characterizing Mesoscale Cellular Convection in Marine Cold Air Outbreaks With a Machine Learning Approach

Abstract During marine cold‐air outbreaks (MCAOs), when cold polar air moves over warmer ocean, a well‐recognized cloud pattern develops, with open or closed mesoscale cellular convection (MCC) at larger fetch over open water. The Cold‐Air Outbreaks in the Marine Boundary Layer Experiment provided a comprehensive set of ground‐based in situ and remote sensing observations of MCAOs at a coastal location in northern Norway. MCAO periods that unambiguously exhibit open or closed MCC are determined. Individual cells observed with a profiling Ka‐band radar are identified using a watershed segmentation method. Using self‐organizing maps (SOMs), these cells are then objectively classified based on the variability in their vertical structure. The SOM nodes contain some information about the location of the cell transect relative to the center of the MCC. This adds classification noise, requiring numerous cell transects to isolate cell dynamical information. The SOM‐based classification shows that comparatively intense convection occurs only in open MCC. This convection undergoes an apparent lifecycle. Developing cells are associated with stronger updrafts, large spectrum width, larger amounts of liquid water, lower surface precipitation rates, and lower cloud tops than mature and weakening cells. The weakening of these cells is associated with the development of precipitation‐induced cold pools. The SOM classification also reveals less intense convection, with a similar lifecycle. More stratiform vertical cloud structures with weak vertical motions are common during closed MCC periods and are separated into precipitating and non‐precipitating stratiform cores. Convection is observed only occasionally in the closed MCC environment.

Meteorology & Atmospheric Sciences↗

Severe Convective Storms in Limited Instability Organized by Pattern and Distribution

Abstract Severe convection occurring in high-shear, low-CAPE (HSLC) environments is a common cool-season threat in the southeastern United States. Previous studies of HSLC convection document the increased operational challenges that these environments present compared to their high-CAPE counterparts, corresponding to higher false-alarm ratios and lower probability of detection for severe watches and warnings. These environments can exhibit rapid destabilization in the hours prior to convection, sometimes associated with the release of potential instability. Here, we use self-organizing maps (SOMs) to objectively identify environmental patterns accompanying HSLC cool-season severe events and associate them with variations in severe weather frequency and distribution. Large-scale patterns exhibit modest variation within the HSLC subclass, featuring strong surface cyclones accompanied by vigorous upper-tropospheric troughs and northward-extending regions of instability, consistent with prior studies. In most patterns, severe weather occurs immediately ahead of a cold front. Other convective ingredients, such as lower-tropospheric vertical wind shear, near-surface equivalent potential temperature (θ e ) advection, and the release of potential instability, varied more significantly across patterns. No single variable used to train SOMs consistently demonstrated differences in the distribution of severe weather occurrence across patterns. Comparison of SOMs based on upper and lower quartiles of severe occurrence demonstrated that the release of potential instability was most consistently associated with higher-impact events in comparison to other convective ingredients. Overall, we find that previously developed HSLC composite parameters reasonably identify high-impact HSLC events. Significance Statement Even when atmospheric instability is not optimal for severe convective storms, in some situations they can still occur, presenting increased challenges to forecasters. These marginal environments may occur at night or during the cool season, when people are less attuned to severe weather threats. Here, we use a sorting algorithm to classify different weather patterns accompanying such storms, and we distinguish which specific patterns and weather system features are most strongly associated with severe storms. Our goals are to increase situational awareness for forecasters and to improve understanding of the processes leading to severe convection in marginal environments.

Meteorology & Atmospheric Sciences↗

Influence of Carbon-Nitride Dot-Emitting Species and Evolution on Fluorescence-Based Sensing and Differentiation

Carbon dots have attracted widespread interest for sensing applications based on their low cost, ease of synthesis, and robust optical properties. We investigate structure–function evolution on multiemitter fluorescence patterns for model carbon-nitride dots (CNDs) and their implications on trace-level sensing. Hydrothermally synthesized CNDs with different reaction times were used to determine how specific functionalities and their corresponding fluorescence signatures respond upon the addition of trace-level analytes. Archetype explosives molecules were chosen as a testbed due to similarities in substituent groups or inductive properties (i.e., electron withdrawing), and solution-based assays were performed using ratiometric fluorescence excitation–emission mapping (EEM). Analyte-specific quenching and enhancement responses were observed in EEM landscapes that varied with the CND reaction time. We then used self-organizing map models to examine EEM feature clustering with specific analytes. Finally, the results reveal that interactions between carbon-nitride frameworks and molecular-like species dictate response characteristics that may be harnessed to tailor sensor development for specific applications.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Programs and Code for Geothermal Exploration Artificial Intelligence

The scripts below are used to run the Geothermal Exploration Artificial Intelligence developed within the "Detection of Potential Geothermal Exploration Sites from Hyperspectral Images via Deep Learning" project. It includes all scripts for pre-processing and processing, including: - Land Surface Temperature K-Means classifier - Labeling AI using Self Organizing Maps (SOM) - Post-processing for Permanent Scatterer InSAR (PSInSAR) analysis with SOM - Mineral marker summarizing - Artificial Intelligence (AI) Data splitting: creates data set from a single raster file - Artificial Intelligence Model: creates AI from a single data set, after splitting in Train, Validation and Test subsets - AI Mapper: creates a classification map based on a raster file

15 GEOTHERMAL ENERGY↗

Predicting Cislunar Orbit Lifetimes from Initial Orbital Elements

The volume of space between Earth’s geosynchronous orbit out to the Moon’s sphere of influence, including the lunar Lagrange points, is crucial for the successful planning and execution of space missions, but not fully understood dynamically. This region is a part of cislunar space. Trajectories through cislunar space are influenced by the gravitational forces of the Sun, Earth, Moon, and other Solar System planets leading to typically unpredictable and chaotic trajectory behavior. It is therefore difficult to predict the stability of an trajectory through cislunar space from a set of initial conditions or orbital elements. We simulate one million cislunar orbits to train a self-organizing map (SOM) to cluster the trajectories and orbits into families based on how long they remain stable within the cislunar space. Using the trained SOM, we are able to predict the stable lifetime of a trajectory through cislunar space from a set of initial orbital elements to within an accuracy of 10% for 8% of simulated trajectories and within 50% for 43% of the simulated trajectories. Clustering in the SOM suggests that a variety of trajectory morphologies have similar lifetimes. Once trained, the SOM can predict the stable lifetime of a given cislunar trajectory within milliseconds. The methods developed in this work enable the rapid identification of stable cislunar orbits and trajectories that could be used for future space exploration. Moreover, the developed SOM method can generate orbital and trajectory lifetime estimates from minimal observational data, such as a single two line element, making it useful for early warning systems and large-scale sensor network operations.

79 ASTRONOMY AND ASTROPHYSICS↗

Linking Synoptic Patterns to Cloud Properties and Local Circulations Over Southeastern Texas

This report classifies meteorological regimes in the southeastern Texas region to identify environmental conditions that favor sea-breeze induced convection. The classification is accomplished using a Self-Organizing Map (SOM) approach. We applied SOM to 10 years of 700-hPa geopotential height anomalies during the summer months from reanalysis data to distinguish three dominant synoptic regimes, with a continuum of transitional states between those. The primary regimes include: (a) a pre-trough regime associated with a synoptic trough, (b) a post-trough regime with upper-level northerly flow, and (c) an anticyclonic regime within the westward extent of the Bermuda High. We project the data from the Geostationary Operational Environmental Satellite and the Next Generation Weather Radar system onto each SOM node to investigate the characteristics of cloud and precipitation properties in different regimes. When southeastern Texas is positioned to the southwest quadrant of a maritime high pressure system, an increased cloud frequency is observed over the region during the afternoon hours due to significant moisture advection. A confluence of synoptic southerly flow and sea-breeze circulation commonly occurs in this regime. When a high pressure system is over southeastern Texas, the area is dominated by large-scale subsidence with weak pressure gradients and moderate precipitable water vapor. This weak synoptic forcing is favorable for the formation of a sea-breeze circulation. This is confirmed by an enhanced onshore flow and a decreased temperature at the surface in the early afternoon, as well as a sharp increase in radar echo top height.

54 ENVIRONMENTAL SCIENCES↗

Dark Energy Survey Year 3 results: redshift calibration of the weak lensing source galaxies

ABSTRACT Determining the distribution of redshifts of galaxies observed by wide-field photometric experiments like the Dark Energy Survey (DES) is an essential component to mapping the matter density field with gravitational lensing. In this work we describe the methods used to assign individual weak lensing source galaxies from the DES Year 3 Weak Lensing Source Catalogue to four tomographic bins and to estimate the redshift distributions in these bins. As the first application of these methods to data, we validate that the assumptions made apply to the DES Y3 weak lensing source galaxies and develop a full treatment of systematic uncertainties. Our method consists of combining information from three independent likelihood functions: self-organizing map p(z) (sompz), a method for constraining redshifts from galaxy photometry; clustering redshifts (WZ), constraints on redshifts from cross-correlations of galaxy density functions; and shear ratios (SRs), which provide constraints on redshifts from the ratios of the galaxy-shear correlation functions at small scales. Finally, we describe how these independent probes are combined to yield an ensemble of redshift distributions encapsulating our full uncertainty. We calibrate redshifts with combined effective uncertainties of σ〈z〉 ∼ 0.01 on the mean redshift in each tomographic bin.

79 ASTRONOMY AND ASTROPHYSICS↗