5 Bias Variance Dilemma

Page 1

5 Bias–Variance Dilemma In this exercise session, we generate a synthetic dataset and examine the phenomenon of bias/variance tradeoff in different models. After that we will analyse some technique in order to manage the tradeoff on real data. The presented procedures allow to decide which model, among a set of given models, is best suited for the problem analysed. For instance, we might want to select the most important features to use, the order of the polynomial in a regression, the coefficient for the regularization term or the K parameter in theK-NN method. 5.1 Bias-Variance Analysis At first, let us generate a dataset, so we are assured to know the real model, in this case assume to have inputx∈ [0, 5] and target: t =t(x) = f (x) +ε = 1 + 1 2x + 1 10x2 +ε, where ε is a Gaussian random variable with E[ε] = 0 and Var (ε) = σ2 = 0.72. To generate a dataset inMATLAB we write: 1 n_points = 1000; 2 eps = 0.7; 3 func = @(x)(1 + 1 / 2 * x + 1 / 10 * x.^2); 4 5 x = 5 * rand(n_points,1); 6 t = func(x); 7 t_noisy = func(x) + eps * randn(n_points,1); After that, we consider two different linear regression models: L1 : y(x) = a +bx L2 : y(x) = a +bx +cx2 If for the former one we do not need to make use of additional features, for the latter one we need to define the basis functionφ1(xi) = xi andφ2(xi) = x2 i : 1 phi = [x x.^2]; 1 5 Bias–Variance Dilemma Once we created the input for both the models we train them: 1 lin_model = fitlm(x, t_noisy); 2 qua_model = fitlm(phi, t_noisy); Let us plot in the parameter space the models we estimated and the optimal ones, where the optimal one in the familyL1 is the model that mina,b ∫ 5 0 (y(x)−a−bx)dx (exercise: compute the best model inL1): 1 real_par = [1 1/2 1/10]; 2 best_lin_par = [7/12 1 0]; 3 4 lin_c =…