Raccolta domande teoriche aggiornate

Page 1

Machine Learning Exam Answers Politecnico di Milano Author Lorenzo Norcini Reviewer Tommaso Scarlatti 1 Linear Models 1.1 Ridge Regression Ridge regression is a particular case of linear regression where we want to find a mapping between an input x and a continuous output t. Our final model has the following form y(x) =wTφ(x) (1) Where φ(x) is a fixed feature-space transformation for x, and w are our model weights. In linear regression the problem mainly consists in minimizing the RSS (residual sum of square) error L(w) = 1 2 N∑ n=1 (tn −wTφ(xn))2 (2) Such problem has closed form solution w = (φTφ)−1φTt (3) the matrix inversion can lead to numerical problems when φTφ is close to singular (i.e. ill conditioned), leading to large weights and in general poor generalization capabilities (over- fitting). Ridge regression still optimizes the RSS error but with the addition of a regularization term that penalizes large weights. L(w) = 1 2 N∑ n=1 (tn −wTφ(xn))2 + λ 2wTw That has solution of the form: w = (φTφ +λI)−1φTt this way φTφ +λI eigenvalues are ”forced” to be at least λ, the larger the λ the larger the regularization. 1 1.2 Lasso Regression Lasso regression is conceptually similar to Ridge regression, in this case the loss function to minimize is L(w) = 1 2 N∑ n=1 (tn −wTφ(xn))2 + λ 2 ||w||1 Since the || · ||1 operation is non-linear there is no closed form solution. Lasso regression yields sparser models since it has the effect of making some of the weights 0. 1.3 Describe the Bias-Variance tradeoff for regression problems. Explain how is it possible to evaluate the bias-variance trade- off by looking at the train error and at the test error. Bias can be described as the difference between the truth and what you expect to learn, meaning the assumptions you make about the…