Complete course notes

Page 1

E d i t e d b y : M a t t i a M a n c a s s o l a N a t u r a l L a n g u a g e P r o c e s s i n g C o u r s e N o t e s These notes have been made thanks to the effort of Polimi Data Scientists staff. Are you interested in Data Science activities? Follow PoliMi Data Scientists on Facebook ! Polimi Data Scientist is a community of students and Alumni of Politecnico di Milano. We organize events and activities related to Artificial Intelligence and Machine Learning, our aim is to create a strong and passionate community about Data Science at Politecnico di Milano. Do you want to learn more? Visit our website and join our Telegram Group ! ! Natural Language Processing Mattia Mancassola 2019 Contents 1 Introduction 5 1.1 Some terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2 Error Correction 6 2.1 Edit Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2 The Noisy Channel Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 3 N-gram Language Models 9 3.1 The Chain Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.2 Evaluating Language Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.3 Perplexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.4 Generalization and Zeros . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.5 Smoothing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.5.1 Laplace Smoothing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 3.5.2 Add-k smoothing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.5.3 Good-Turing smoothing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15…