10 Reinforcement Learning

Page 1

10 Reinforcement Learning 10.1 Practical Session In this exercise session we will analyse problems in which we are not provided com- plete information on the MDP we would like to analyse. In particular we would like to learn the optimal policy to follow by relying on the information coming from the sam- ples given by the MDP . We will implement two of the most common Reinforcement Learning (RL) algorithms: Q-Learning and SARSA. As case study we consider the following MDP modeling an advertising problem: Figure 10.1: The considered MDP . If in the case of Dynamic Programming (DP) we were provided of the model of the MDP in this case we are able to resort only on the transition model functions: r :S ×A → R P :S →S or in MATLAB 1 [s_prime, inst_rew] = transition_model(s, a) 1 10 Reinforcement Learning which can tell you the new state s_prime and the instantaneous reward inst_rew if we provide a current state s and a chosen action a. So the analysis one might per- form will be only based on the generation of some episodes coming from the MDP . Notice that in this case we do not have the expected instantaneous reward, but only a realization of the instantaneous reward. Moreover, we should consider a specific policy to run on our problem in order to select the next action. In this case we rely on theε-greedy one: 1 function a = eps_greedy(s, allowed_actions, Q, eps) 2 3 if rand() <= eps 4 actions = find(allowed_actions(s,:) == 1); 5 a = randsample(actions,1); 6 else 7 curr_Q = Q(s,:); 8 curr_Q(~allowed_actions(s,:)) = -inf; 9 [~, ind_action] = max(curr_Q); 10 if length(ind_action) > 1 11 a = randsample(ind_action,1); 12 else 13 a = ind_action; 14 end 15 end where we need to specify the current state s, the actions which can be chosen in the current state allowed_actions, the…