Foundations of Feature Selection and Classification for Non-Gaussian Distributed Targets
This report describes the author’s research toward creating algorithms for selecting useful statistical features in pattern/target classification problems in which the features are non-Gaussian distributed. In engineering practice, it is common to either (1) Not perform any feature selection procedure, or (2) Use a feature selection algorithm that assumes the features are Gaussian distributed. This process can be far from optimal if the data are actually non-Gaussian, as they frequently are in practice. This research has the goal of mitigating that problem by creating a useful algorithm that can be used in practice. The approach is to focus on the performance indices used in common feature selection algorithms, such as Branch and Bound algorithms and the Sequential Forward Selection Algorithm. Ordinarily, the performance index used to measure the class separation in feature space involves assuming the data are Gaussian and deriving tractable performance indices that use closed form equations that represent the probability density functions of the class data. The advantage of this approach is that it produces feature selection algorithms that have low computational complexity and do not require knowledge of the actual data densities. The disadvantage is that these algorithms may not perform reasonably when the data are non-Gaussian, as they commonly are in practice. This research examines the use of information-theoretic class separability measures that can deal with the non-Gaussian case. In particular, this work shows that the Hellinger Distance (a divergence measure), has very desirable mathematical properties and can be useful for feature selection. The work in this paper employs a multivariate kernel density estimator along with the Hellinger Distance and an optimal subset selection algorithm to do feature selection. This report (1) Defines the classification and feature selection problems, (2) Describes the continuous variable Hellinger distance in one dimension and discusses its mathematical properties in comparison with other distance measures, (3) Develops the discrete variable Hellinger distance expression for one and two dimensions, and shows that the discrete Hellinger distance for multiple dimensions is a straightforward extension, and (4) Presents experimental density estimation results using a new MATLAB implementation of a multivariate kernel density estimator. The draft of this report served as a starting point for Ensign Matthew Wilder’s MSEE thesis work at the Naval Postgraduate School (NPS) in 2010 and 2011. Ensign Wilder completed his thesis and graduated from NPS in June 2011 (M. J. Wilder, “Automatic Target Recognition: Statistical Feature Selection of Non-Gaussian Distributed Target Classes,” MSEE Thesis, Naval Postgraduate School, Thesis Advisor: Grace A. Clark, Second Reader: Monique P. Fargues, June 2011). In the thesis, Ensign Wilder created implementations of the Branch and Bound and Sequential Feature Selection algorithms, extended the multivariate kernel density estimator code, and evaluated the performance of Hellinger Distance-based feature selection algorithms using both simulated data and data from a real-world benchmark data set. The new algorithm was shown to be very effective for both Gaussian and non-Gaussian data sets. The advantages gained for non-Gaussian data come at the cost of increased computational complexity. Using a laptop computer for practical problems, we generally use initial feature vectors containing no more than about ten or twelve features. Proposed future research lies in creating faster kernel density estimator algorithms that would allow for the use of higher dimensional data.