Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Dynamic clustering algorithm”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Substructure in the stellar halo near the Sun: II. Characterisation of independent structures

In an accompanying paper, we present a data-driven method for clustering in ‘integrals of motion’ space and apply it to a large sample of nearby halo stars with 6D phase-space information. The algorithm identified a large number of clusters, many of which could tentatively be merged into larger groups. The goal here is to establish the reality of the clusters and groups through a combined study of their stellar populations (average age, metallicity, and chemical and dynamical properties) to gain more insights into the accretion history of the Milky Way. To this end, we developed a procedure that quantifies the similarity of clusters based on the Kolmogorov–Smirnov test using their metallicity distribution functions, and an isochrone fitting method to determine their average age, which is also used to compare the distribution of stars in the colour–absolute magnitude diagram. Also taking into consideration how the clusters are distributed in integrals of motion space allows us to group clusters into substructures and to compare substructures with one another. We find that the 67 clusters identified by our algorithm can be merged into 12 extended substructures and 8 small clusters that remain as such. The large substructures include the previously known Gaia-Enceladus, Helmi streams, Sequoia, and Thamnos 1 and 2. We identify a few over-densities that can be associated with the hot thick disc and host a small metal-poor population. Especially notable is the largest (by number of member stars) substructure in our sample which, although peaking at the metallicity characteristic of the thick disc, has a very well populated metal-poor component, and dynamics intermediate between the hot thick disc and the halo. We also identify additional debris in the region occupied by Sequoia with clearly distinct kinematics, likely remnants of three different accretion events with progenitors of similar masses. Although only a small subset of the stars in our sample have chemical abundance information, we are able to identify different trends of [Mg/Fe] versus [Fe/H] for the various substructures, confirming our dissection of the nearby halo. We find that at least 20% of the halo near the Sun is associated to substructures. When comparing their global properties, we note that those substructures on retrograde orbits are not only more metal-poor on average but are also older. We provide a table summarising the properties of the substructures, as well as a membership list that can be used for follow-up chemical abundance studies for example.

79 ASTRONOMY AND ASTROPHYSICS↗
NASA NTRS2011

Defining Dynamic Route Structure

This poster describes a method for defining route structure from flight tracks. Dynamically generated route structures could be useful in guiding dynamic airspace configuration and helping controllers retain situational awareness under dynamically changing traffic conditions. Individual merge and diverge intersections between pairs of flights are identified, clustered, and grouped into nodes of a route structure network. Links are placed between nodes to represent major traffic flows. A parametric analysis determined the algorithm input parameters producing route structures of current day flight plans that are closest to todays airway structure. These parameters are then used to define and analyze the dynamic route structure over the course of a day for current day flight paths. Route structures are also compared between current day flight paths and more user preferred paths such as great circle and weather avoidance routing.

Zelinski, Shannon↗
DOE OSTI2025

Web-based wide-area monitoring platform for ringdown and clustering analytics in power systems

This paper introduces an open-source research platform for monitoring the Mexican interconnected power grid, allowing real-time processing and information extraction of the grid’s dynamic condition. Moreover, the platform is a Python-based development that embeds different ringdown and clustering analytics tools. In the case of ringdown analysis, the modal information can be extracted using some of the most known algorithms, i.e., Prony analysis, eigensystem realization algorithm (ERA), and matrix pencil (MP). For clustering analysis, the coherent behaviour of generator and non-generator buses is provided by applying recent state-of-the-art techniques such as affinity propagation, K-means, hierarchical agglomerative clustering, and typicality data analysis. The results of up to 93 PMUs show that this open-source platform suits researchers’ and engineers’ power system dynamic analysis requirements.

Clustering↗
DOE OSTI2024

Identifying Climate Patterns Using Clustering Autoencoder Techniques

Abstract The complexity of growing spatiotemporal resolution of climate simulations produces a variety of climate patterns under different projection scenarios. This paper proposes a new data-driven climate classification workflow via an unsupervised deep learning technique that can dimensionally reduce the vast volume of spatiotemporal numerical climate projection data into a compact representation. We aim to identify distinct zones that capture multiple climate variables as well as their future changes under different climate change scenarios. Our approach leverages convolutional autoencoders combined with k -means clustering (standard autoencoder) and online clustering based on the Sinkhorn–Knopp algorithm (clustering autoencoder) across the conterminous United States (CONUS) to capture unique climate patterns in a data-driven fashion from the Geophysical Fluid Dynamics Laboratory Earth System Model with GOLD component (GFDL-ESM2G). The developed approach compresses 70 years of GFDL-ESM2G simulation at 0.125° spatial resolution across the CONUS under multiple warming scenarios to a lower-dimensional space by a factor of 660 000 and then tested on 150 years of GFDL-ESM2G simulation data. The results show that five climate clusters capture physically reasonable and spatially stable climatological patterns matched to known climate classes defined by human experts. Results also show that using a clustering autoencoder can reduce the computational time for clustering by up to 9.2 times when compared to using a standard autoencoder. Our five unique climate patterns resulting from the deep learning–based clustering of the lower-dimensional space thereby enable us to provide insights on hydrometeorology and its spatial heterogeneity across the conterminous United States immediately without downloading large climate datasets. Significance Statement This paper presents a data-driven climate classification approach using unsupervised deep learning to dimensionally reduce climate model outputs and to identify distinct climate regions for their future changes. Our approach compresses climate information for 70 years of Geophysical Fluid Dynamics Laboratory Earth System Model data across the conterminous United States (CONUS) at 0.125° spatial resolution. The results reveal that five climate clusters capture reasonable and stable climatological patterns matched to known climate patterns. The embedded clustering process in deep learning provides ×9.2 times faster execution than the k -means clustering technique. These results give us insight about climate spatial patterns and heterogeneity of hydrological patterns across the conterminous United States without downloading large climate datasets.

Kurihana, Takuya↗
DOE OSTI2022

Velocity dispersions of clusters in the Dark Energy Survey Y3 redMaPPer catalogue

ABSTRACT We measure the velocity dispersions of clusters of galaxies selected by the red-sequence Matched-filter Probabilistic Percolation (redMaPPer) algorithm in the first three years of data from the Dark Energy Survey (DES), allowing us to probe cluster selection and richness estimation, λ, in light of cluster dynamics. Our sample consists of 126 clusters with sufficient spectroscopy for individual velocity dispersion estimates. We examine the correlations between cluster velocity dispersion, richness, X-ray temperature, and luminosity, as well as central galaxy velocity offsets. The velocity dispersion–richness relation exhibits a bimodal distribution. The majority of clusters follow scaling relations between velocity dispersion, richness, and X-ray properties similar to those found for previous samples; however, there is a significant population of clusters with velocity dispersions that are high for their richness. These clusters account for roughly 22 per cent of the λ < 70 systems in our sample, but more than half (55 per cent) of λ < 70 clusters at z > 0.5. A couple of these systems are hot and X-ray bright as expected for massive clusters with richnesses that appear to have been underestimated, but most appear to have high velocity dispersions for their X-ray properties likely due to line-of-sight structure. These results suggest that projection effects contribute significantly to redMaPPer selection, particularly at higher redshifts and lower richnesses. The redMaPPer determined richnesses for the velocity dispersion outliers are consistent with their X-ray properties, but several are X-ray undetected and deeper data are needed to understand their nature.

79 ASTRONOMY AND ASTROPHYSICS↗
NASA NTRS2024

Characterizing the California Current System through Sea Surface Temperature and Salinity

Characterizing temperature and salinity (T-S) conditions is a standard framework in oceanography to identify and describe deep water masses and their dynamics. At the surface, this practice is hindered by multiple air–sea–land processes impacting T-S properties at shorter time scales than can easily be monitored. Now, however, the unsurpassed spatial and temporal coverage and resolution achieved with satellite sea surface temperature (SST) and salinity (SSS) allow us to use these variables to investigate the variability of surface processes at climate-relevant scales. In this work, we use SSS and SST data, aggregated into domains using a cluster algorithm over a T-S diagram, to describe the surface characteristics of the California Current System (CCS), validating them with in situ data from uncrewed Saildrone vessels. Despite biases and uncertainties in SSS and SST values in highly dynamic coastal areas, this T-S framework has proven useful in describing CCS regional surface properties and their variability in the past and in real time, at novel scales. This analysis also shows the capacity of remote sensing data for investigating variability in land–air–sea interactions not previously possible due to limited in situ data.

Marisol García-Reyes↗
NASA NTRS2013

Global Weather States and Their Properties from Passive and Active Satellite Cloud Retrievals

In this study, the authors apply a clustering algorithm to International Satellite Cloud Climatology Project (ISCCP) cloud optical thickness-cloud top pressure histograms in order to derive weather states (WSs) for the global domain. The cloud property distribution within each WS is examined and the geographical variability of each WS is mapped. Once the global WSs are derived, a combination of CloudSat and Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observations (CALIPSO) vertical cloud structure retrievals is used to derive the vertical distribution of the cloud field within each WS. Finally, the dynamic environment and the radiative signature of the WSs are derived and their variability is examined. The cluster analysis produces a comprehensive description of global atmospheric conditions through the derivation of 11 WSs, each representing a distinct cloud structure characterized by the horizontal distribution of cloud optical depth and cloud top pressure. Matching those distinct WSs with cloud vertical profiles derived from CloudSat and CALIPSO retrievals shows that the ISCCP WSs exhibit unique distributions of vertical layering that correspond well to the horizontal structure of cloud properties. Matching the derived WSs with vertical velocity measurements shows a normal progression in dynamic regime when moving from the most convective to the least convective WS. Time trend analysis of the WSs shows a sharp increase of the fair-weather WS in the 1990s and a flattening of that increase in the 2000s. The fact that the fair-weather WS is the one with the lowest cloud radiative cooling capability implies that this behavior has contributed excess radiative warming to the global radiative budget during the 1990s.

histograms↗
DOE OSTI2021

Dynamic Mode Decomposition of Random Pressure Fields over Bluff Bodies

Fluctuating surface pressures on a bluff body exposed to a boundary layer flow generally are characterized as a spatiotemporally varying random field. In this paper, a dynamic mode decomposition (DMD) was applied to extract dominant features embedded in these random pressure fields. Utilizing an unsupervised machine learning algorithm, spatial modes and their temporal variations were grouped into different clusters at scales, e.g., macro, meso, and micro. A proper orthogonal decomposition (POD) of the experimental data was carried out to observe commonalities and distinctive perspectives each decomposition offers. Here, a comprehensive examination of the DMD/POD for their convergence criteria, data sufficiency, and modal components analysis was conducted. The physical interpretation of the spatiotemporal pressure field based on these decomposition schemes was discussed. At different scales, the DMD modes can capture the evolution of aerodynamic features, e.g., convection of vortices (or vortex tubes) and other structures. The distribution of energy among these three broad scales also reflects an energy cascade in pressure fluctuations akin to turbulence.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

SMALE: Enhancing Scalability of Machine Learning Algorithms on Extreme-Scale Computing Platforms

Deployment and execution of machine learning tasks on extreme-scale computing platforms face several significant technical challenges: 1) High computing cost incurred by dense networks – The computing workload of deep networks with densely-connected topology increases rapidly with the network size, imposing a non-scalable computing model of extreme-scale computing platforms; 2) Non-optimized workload distribution – Many advanced deep learning algorithms, e.g., sparsification and irregular net-work topology, produce very unbalanced workload distribution on extreme-scale computing platforms. The computation efficiency is greatly hindered by the incurred data and computation redundancies as well as long tails of the node with extensive workload; 3) Constraints in data movement and I/O bottle-neck – Inter-node data movement in extreme-scale computing platforms are associated with high energy and latency costs, and subject to the constraints of I/O bandwidth; and 4) Generalization of algorithm realization and acceleration on computing platforms – The large varieties of machine learning algorithms and structures of extreme-scale computing platforms make the derivation of a generalized algorithm realization and acceleration method very challenging, which, however, is the requirement by domain scientists and interested users. We call the above challenges Smale’s Problems in Machine Learning and Understanding for High-Performance Computing Scientific Discovery. The objective of our three-year research project is to develop a holistic innovation set at structure, assembly, and acceleration layers of machine learning algorithms to address the above challenges in algorithm deployment and execution. Three tasks are particularly performed, including: At the algorithm structure level, we investigate the techniques that can structurally sparsify on the topology of deep networks for computing workload reduction. We also study clustering and pruning techniques that can optimize the workload distributions over the extreme-scale computing platforms; At the algorithm assembly level, we derive a unified learning framework for unsupervised transfer learning and dynamic growing capabilities. Novel training methods are also exploited to enhance the training efficiency of the proposed framework; At the algorithm acceleration level, we will develop a series of techniques that can accelerate the computation of sparse matrix operations, which are one of the core executions in deep learning and optimize memory access of the concerned platforms. Our proposed techniques attack the fundamental problems in machine learning algorithms running on extreme-scale computing platforms by vertically integrating the solutions at three closely entangled layers, paving the long-term scaling path of machine learning applications under DOE context. Three tasks corresponding to the above respective research orientations are performed during the three-year project period with our collaborators at ORNL. The outcome of the proposed project is anticipated to form a holistic solution set of novel algorithms and network topologies, efficient training techniques, and fast acceleration methods to promote the computing scalability of the machine learning applications of particular interest to DOE.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2016

Automated Storm Tracking and the Lightning Jump Algorithm Using GOES-R Geostationary Lightning Mapper (GLM) Proxy Data

This study develops a fully automated lightning jump system encompassing objective storm tracking, Geostationary Lightning Mapper proxy data, and the lightning jump algorithm (LJA), which are important elements in the transition of the LJA concept from a research to an operational based algorithm. Storm cluster tracking is based on a product created from the combination of a radar parameter (vertically integrated liquid, VIL), and lightning information (flash rate density). Evaluations showed that the spatial scale of tracked features or storm clusters had a large impact on the lightning jump system performance, where increasing spatial scale size resulted in decreased dynamic range of the system's performance. This framework will also serve as a means to refine the LJA itself to enhance its operational applicability. Parameters within the system are isolated and the system's performance is evaluated with adjustments to parameter sensitivity. The system's performance is evaluated using the probability of detection (POD) and false alarm ratio (FAR) statistics. Of the algorithm parameters tested, sigma-level (metric of lightning jump strength) and flash rate threshold influenced the system's performance the most. Finally, verification methodologies are investigated. It is discovered that minor changes in verification methodology can dramatically impact the evaluation of the lightning jump system.

lightning jump↗
DOE OSTI2022

Accelerating error correction in tomographic reconstruction

Abstract Spurred by recent advances in detector technology and X-ray optics, upgrades to scanning-probe-based tomographic imaging have led to an exponential growth in the amount and complexity of experimental data and have created a clear opportunity for tomographic imaging to approach single-atom sensitivity. The improved spatial resolution, however, is highly susceptible to systematic and random experimental errors, such as center of rotation drifts, which may lead to imaging artifacts and prevent reliable data extraction. Here, we present a model-based approach that simultaneously optimizes the reconstructed specimen and sinogram alignment as a single optimization problem for tomographic reconstruction with center of rotation error correction. Our algorithm utilizes an adaptive regularizer that is dynamically adjusted at each alternating iteration step. Furthermore, we describe its implementation in a software package targeting high-throughput workflows for execution on distributed-memory clusters. We demonstrate the performance of our solver on large-scale synthetic problems and show that it is robust to a wide range of noise and experimental drifts with near-ideal throughput.

Ali, Sajid (ORCID:0000000321864636)↗
DOE OSTI2024

Subhalos in Galaxy Clusters: Coherent Accretion and Internal Orbits

Subhalo dynamics in galaxy cluster host halos govern the observed distribution and properties of cluster member galaxies. We use the IllustrisTNG simulation to investigate the accretion and orbits of subhalos found in cluster-size halos. We find that the median change in the major axis direction of cluster-size host halos is approximately 80° between a ∼ 0.1 and the present day. We identify coherent regions in the angular distribution of subhalo accretion, and ∼68% of accreted subhalos enter their host halo through ∼38% of the surface area at the virial radius. The majority of galaxy clusters in the sample have ∼2 such coherent regions. We further measure angular orbits of subhalos with respect to the host major axis and use a clustering algorithm to identify distinct orbit modes with varying oscillation timescales. The orbit modes correlate with subhalo accretion conditions. Subhalos in orbit modes with shorter oscillations tend to have lower peak masses and accretion directions somewhat more aligned with the major axis. One orbit mode, exhibiting the least oscillatory behavior, largely consists of subhalos that accrete near the plane perpendicular to the host halo major axis. Our findings are consistent with expectations from inflow from major filament structures and internal dynamical friction: most subhalos accrete through coherent regions, and more massive subhalos experience fewer orbits after accretion. Our work offers a unique quantification of subhalo dynamics that can be connected to how the intracluster medium strips and quenches cluster galaxies.

galaxy clusters↗
NASA NTRS2012

Classification of Ascension Island and Natal Ozonesondes Using Self-Organizing Maps

Ozone profiles from balloon-borne ozonesondes are used for development of satellite algorithms and in chemistry-climate model initialization, assimilation and evaluation. An important issue in the application of these profiles is how best to treat variations where varying photochemical and dynamical influences can cause the ozone mixing ratio in the tropospheric segments of the profile to change by of a factor of 2-3 within a day. Clustering techniques are an ideal way to approach the statistical classification of profile data and we apply self-organizing maps to tropical tropospheric SHADOZ data, hypothesizing that the data will sort according to various influences on ozone, namely anthropogenic sources like biomass burning, meteorological conditions, and stratospheric or extra-tropical intrusions. Self-organizing maps, that use a learning algorithm to reveal the most prominent features of a data set according to a specified number of clusters, have been determined for the 1998-2009 SHADOZ profiles over Ascension Island (512 profiles, 7.98 deg. S, 14.42 deg. W) and Natal, Brazil (425 profiles, 5.42degS, 35.38degW). The 2 × 2 self-organizing map, which creates 4 clusters, reveals that deviations from the average ozone in the free troposphere include both increased ozone resulting from seasonal biomass burning in Africa and locally reduced ozone brought about by convective lifting of unpolluted boundary-layer air. Expanding to a 4 × 4 self-organizing map shows how biomass burning influences the yearly cycle of tropospheric ozone at Ascension Island and captures the seasonality of ozone at both Ascension Island and Natal. Comparing Ascension Island and Natal using a 4 × 4 self-organizing map at each site reveals similarities in mid-tropospheric ozone, but shows differences in lower-tropospheric ozone due to Ascension Island being closer to African biomass burning and more affected by descent from the mean Walker circulation, with less convective activity, than Natal.

algorithms↗
DOE OSTI2021

Adaptive pruning-based optimization of parameterized quantum circuits

Abstract Variational hybrid quantum–classical algorithms are powerful tools to maximize the use of noisy intermediate-scale quantum devices. While past studies have developed powerful and expressive ansatze, their near-term applications have been limited by the difficulty of optimizing in the vast parameter space. In this work, we propose a heuristic optimization strategy for such ansatze used in variational quantum algorithms, which we call ‘parameter-efficient circuit training (PECT)’. Instead of optimizing all of the ansatz parameters at once, PECT launches a sequence of variational algorithms, in which each iteration of the algorithm activates and optimizes a subset of the total parameter set. To update the parameter subset between iterations, we adapt the Dynamic Sparse Reparameterization scheme which was originally proposed for training deep convolutional neural networks. We demonstrate PECT for the Variational Quantum Eigensolver, in which we benchmark unitary coupled-cluster ansatze including UCCSD and k -UpCCGSD, as well as the Low-Depth Circuit Ansatz (LDCA), to estimate ground state energies of molecular systems. We additionally use a layerwise variant of PECT to optimize a hardware-efficient circuit for the Sycamore processor to estimate the ground state energy densities of the one-dimensional Fermi-Hubbard model. From our numerical data, we find that PECT can enable optimizations of certain ansatze that were previously difficult to converge and more generally can improve the performance of variational algorithms by reducing the optimization runtime and/or the depth of circuits that encode the solution candidate(s).

Physics↗
DOE OSTI2022

DefectTrack: a deep learning-based multi-object tracking algorithm for quantitative defect analysis of in-situ TEM videos in real-time

Abstract In-situ irradiation transmission electron microscopy (TEM) offers unique insights into the millisecond-timescale post-cascade process, such as the lifetime and thermal stability of defect clusters, vital to the mechanistic understanding of irradiation damage in nuclear materials. Converting in-situ irradiation TEM video data into meaningful information on defect cluster dynamic properties (e.g., lifetime) has become the major technical bottleneck. Here, we present a solution called the DefectTrack , the first dedicated deep learning-based one-shot multi-object tracking (MOT) model capable of tracking cascade-induced defect clusters in in-situ TEM videos in real-time. DefectTrack has achieved a Multi-Object Tracking Accuracy (MOTA) of 66.43% and a Mostly Tracked (MT) of 67.81% on the test set, which are comparable to state-of-the-art MOT algorithms. We discuss the MOT framework, model selection, training, and evaluation strategies for in-situ TEM applications. Further, we compare the DefectTrack with four human experts in quantifying defect cluster lifetime distributions using statistical tests and discuss the relationship between the material science domain metrics and MOT metrics. Our statistical evaluations on the defect lifetime distribution suggest that the DefectTrack outperforms human experts in accuracy and speed.

42 ENGINEERING↗
DOE OSTI2021

Examination of Radiation Belt Dynamics During Substorm Clusters: Activity Drivers and Dependencies of Trapped Flux Enhancements

Here, dynamical variations of radiation belt trapped electron fluxes are examined to better understand the variability of enhancements linked to substorm clusters. Analysis is undertaken using the Substorm Onsets and Phases from Indices of the Electrojet substorm cluster algorithm for event detection. Observations from low earth orbit are complemented by additional measurements from medium earth orbit to allow a major expansion in the energy range considered, from medium energy energetic electrons up to ultra-relativistic electrons. The number of substorms identified inside a cluster does not depend strongly on solar wind drivers or geomagnetic indices either before, during, or after the cluster start time. Clusters of substorms linked to moderate (100 nT < AE ≤ 300 nT) or strong AE (AE ≥ 300 nT) disturbances are associated with radiation belt flux enhancements, including up to ultra-relativistic energies by the strongest substorms (as measured by strong southward Bz and high AE). These clusters reliably occur during times of high speed solar winds streams with associated increased magnetospheric convection. However, substorm clusters associated with quiet AE disturbances (AE ≤ 100 nT) lead to no significant chorus whistler mode intensity enhancements, or increases in energetic, relativistic, or ultra-relativistic electron flux in the outer radiation belts. In these cases the solar wind speed is low, and the geomagnetic Kp index indicates a lack of magnetospheric convection. Our study clearly indicates that clusters of substorms occurring outside of high speed wind streams are not by themselves sufficient to drive acceleration, which may be due to the lack of pre-cluster convection.

79 ASTRONOMY AND ASTROPHYSICS↗
NASA NTRS2012

Particle Filtering for Model-Based Anomaly Detection in Sensor Networks

A novel technique has been developed for anomaly detection of rocket engine test stand (RETS) data. The objective was to develop a system that postprocesses a csv file containing the sensor readings and activities (time-series) from a rocket engine test, and detects any anomalies that might have occurred during the test. The output consists of the names of the sensors that show anomalous behavior, and the start and end time of each anomaly. In order to reduce the involvement of domain experts significantly, several data-driven approaches have been proposed where models are automatically acquired from the data, thus bypassing the cost and effort of building system models. Many supervised learning methods can efficiently learn operational and fault models, given large amounts of both nominal and fault data. However, for domains such as RETS data, the amount of anomalous data that is actually available is relatively small, making most supervised learning methods rather ineffective, and in general met with limited success in anomaly detection. The fundamental problem with existing approaches is that they assume that the data are iid, i.e., independent and identically distributed, which is violated in typical RETS data. None of these techniques naturally exploit the temporal information inherent in time series data from the sensor networks. There are correlations among the sensor readings, not only at the same time, but also across time. However, these approaches have not explicitly identified and exploited such correlations. Given these limitations of model-free methods, there has been renewed interest in model-based methods, specifically graphical methods that explicitly reason temporally. The Gaussian Mixture Model (GMM) in a Linear Dynamic System approach assumes that the multi-dimensional test data is a mixture of multi-variate Gaussians, and fits a given number of Gaussian clusters with the help of the wellknown Expectation Maximization (EM) algorithm. The parameters thus learned are used for calculating the joint distribution of the observations. However, this GMM assumption is essentially an approximation and signals the potential viability of non-parametric density estimators. This is the key idea underlying the new approach.

Solano, Wanda↗
NASA NTRS2006

Diurnal Variability of Vertical Structure from a TRMM Passive Microwave "Virtual Radar" Retrieval

Robust description of the diurnal cycle from TRMM observations is complicated by the limitations of Low Earth Orbit (LEO) sampling; from a 'climatological' perspective, sufficient sampling must exist to control for both spatial and seasonal variability, before tackling an additional diurnal component (e.g., with 8 additional 3-hourly or 24 1-hourly bins). For documentation of vertical structure, the narrow sample swath of the TRMM Precipitation Radar limits the resolution of any of these components. A neural-network based 'virtual radar" retrieval has been trained and internally validated, using multifrequency / multipolarization passive microwave(TM1) brightness temperatures and textures parameters and lightning (LIS) observations, as inputs, and PR volumetric reflectivity as targets (outputs). By training the algorithms (essentially highly multivariate, nonlinear regressions) on a very large sample of high-quality co-located data from the center of the TRMM swath, 3D radar reflectivity and derived parameters (VIL, IWC, Echo Tops, etc.) can be retrieved across the entire TMI swath, good to 8-9% over the dynamic range of parameters. As a step in the retrieval (and as an output of the process), each TMI multifrequency pixel (at 85 GHz resolution) is classified into one of the 25 archetypal radar profile vertical structure "types", previously identified using cluster analysis. The dynamic range of retrieved vertical structure appears to have higher fidelity than the current (Version 6) experimental GPROF hydrometeor vertical structure retrievals. This is attributable to correct representation of the prior probabilities of vertical structure variability in the neural network training data, unlike the GPROF cloud-resolving model training dataset used in the V6 algorithms. The LIS lightning inputs are supplementary inputs, and a separate offline neural network has been trained to impute (predict) LIS lightning from passive-microwave-only data. The virtual radar retrieval is thus, in principle, extensible to Aqua/AMSR-E and NPOESS/CMIS passive microwave instruments. The virtual radar approach yields a threefold increase in effective sampling from the mission, albeit of lower-quality "retrieved" data, reducing the variance of local estimates by one third (or the standard deviation by-0.57). In this talk, the variance reduction is leveraged to more finely resolve global diurnal variability in both space and time (local hour).

Boccippio, Dennis J.↗