6 Kernel methods

Pagina 1

7 Feature Selection and Kernel Methods What to do in the case the model you are considering is not performing well even by tuning properly the parameters (cross-validation) to discriminate between classes? We have two opposite options: simplify the model or increase its complexity. On one hand possibility is to reduce the dimensionality of the data we considered since most of them do not bring any information to the task, thus they only introduce noise in the estimation procedure. On the other hand, we could consider high dimensional space where the classification task is easy, by enhancing the input space with new features. 7.1 Principal Component Analysis Principal Component Analysis (PCA) is a technique to perform dimensionality reduc- tion, i.e., to extract some low dimensional features from a dataset. More specifically, we find a linear transformation of the original data X s.t. the greatest variance lies on the first coordinate, the second greatest variance on the second coordinate and so on. It results that the new coordinates are orthogonal to each others. A procedure for computing these coordinates is the following: • Translate the original data X to ˜X s.t. they have zero mean; • Compute the covariance matrix of ˜X,C = ˜XT ˜X; • The eigenvector e1 corresponding to the largest eigenvalueλ1 is the first principal component; • The eigenvector e2 corresponding to the second largest eigenvalue λ2 is the sec- ond principal component; • etc. We apply this technique to the Iris dataset. At first we load the data: 1 load iris_dataset; 2 [irisTargets, ~] = find(irisTargets == 1); 3 gplotmatrix(irisInputs',[],irisTargets); 4 irisInputs = irisInputs'; 1 7 Feature Selection and Kernel Methods In this case, we do not normalize the data since otherwise we are loosing information…