Complete course notes

Page 1

Formulas and concepts of Machine Learning Samuele Pino July 2020 i This document puts togeheter the M.Restelli lessons (A.A. 2018/2019) of Machine Learning at Politecnico of Milan and the exercises of the teaching assistant F. Trov` o. Some material has been taken online too, from the recap of Polimi DataScience and from the recap by Simone Staffa. This was made only for study purposes, with no intention of selling. This is a best-effort document, therefore it comes without any guarantees of correctness. The author rejects any responsibility related to the usage of the content in this document. Contents 1 Introduction 1 1.1 Definitions and Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Statistics Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 Taxonomy of Supervised Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 I Supervised Learning 6 2 Linear Regression 7 2.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2 Type of approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3 Direct approach: LS Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.3.1 Geometric interpretation of OLS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.4 Discriminative approach: Likelyhood Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.5 Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.6 Bayesian Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.6.1 Predictive…