|
所在平台: Coursera |
课程主页: https://www.coursera.org/learn/sample-based-learning-methods
课程评论：没有评论
课程名称：基于样本的学习方法 概述：本课程将介绍几种算法，这些算法能够通过与环境的试错互动来学习接近最优的策略，即从代理的自身经验中学习。利用实际经验进行学习非常引人注目，因为它不需要环境动态的先前知识，但仍然能够实现最优行为。我们将涵盖直观简单但功能强大的蒙特卡洛方法，以及包括Q-learning在内的时序差分学习方法。课程最后将探讨如何结合基于模型的规划（类似于动态规划）和时序差分更新，以大幅加速学习。 学习目标： - 理解时序差分学习和蒙特卡洛方法作为从样本经验中估计价值函数的两种策略 - 理解在使用样本经验而非动态规划时，探索的重要性 - 理解蒙特卡洛方法与动态规划和时序差分之间的联系 - 实现并应用时序差分算法，用于估计价值函数 - 实现并应用期望Sarsa和Q-learning（两种用于控制的时序差分方法） - 理解基于策略和离策略控制的差异 - 理解利用模拟经验进行规划（与经典规划策略的对比） - 实现一种称为Dyna的基于模型的强化学习方法，使用模拟经验 - 进行实证研究，以观察使用Dyna时样本效率的改善 大纲： 1. 课程介绍：引入讲师，并了解课程内容，鼓励学员在“见面会”部分自我介绍。 2. 用于预测与控制的蒙特卡洛方法：学习如何仅通过环境中的样本经验估计价值函数和最优策略，明确探索问题在强化学习中的意义。 3. 用于预测的时序差分学习方法：深入学习时序差分学习这一强化学习的基础概念，理解其如何在没有模型的情况下进行在线学习。 4. 用于控制的时序差分学习方法：了解使用时序差分学习进行控制的不同算法，包括Sarsa、Q-learning和期望Sarsa，并在Cliff World环境上实现这些算法。 5. 规划、学习与行动：将基于模型的规划与样本学习相结合，学习如何从数据中估计模型并生成假设经验，以提高样本效率，并设计健壮的学习系统。
Name:Welcome to the Course!
Description:Welcome to the second course in the Reinforcement Learning Specialization: Sample-Based Learning Methods, brought to you by the University of Alberta, Onlea, and Coursera. In this pre-course module, you'll be introduced to your instructors, and get a flavour of what the course has in store for you. Make sure to introduce yourself to your classmates in the "Meet and Greet" section!
Name: Monte Carlo Methods for Prediction & Control
Description:This week you will learn how to estimate value functions and optimal policies, using only sampled experience from the environment. This module represents our first step toward incremental learning methods that learn from the agent’s own interaction with the world, rather than a model of the world. You will learn about on-policy and off-policy methods for prediction and control, using Monte Carlo methods---methods that use sampled returns. You will also be reintroduced to the exploration problem, but more generally in RL, beyond bandits.
Name:Temporal Difference Learning Methods for Prediction
Description:This week, you will learn about one of the most fundamental concepts in reinforcement learning: temporal difference (TD) learning. TD learning combines some of the features of both Monte Carlo and Dynamic Programming (DP) methods. TD methods are similar to Monte Carlo methods in that they can learn from the agent’s interaction with the world, and do not require knowledge of the model. TD methods are similar to DP methods in that they bootstrap, and thus can learn online---no waiting until the end of an episode. You will see how TD can learn more efficiently than Monte Carlo, due to bootstrapping. For this module, we first focus on TD for prediction, and discuss TD for control in the next module. This week, you will implement TD to estimate the value function for a fixed policy, in a simulated domain.
Name:Temporal Difference Learning Methods for Control
Description:This week, you will learn about using temporal difference learning for control, as a generalized policy iteration strategy. You will see three different algorithms based on bootstrapping and Bellman equations for control: Sarsa, Q-learning and Expected Sarsa. You will see some of the differences between the methods for on-policy and off-policy control, and that Expected Sarsa is a unified algorithm for both. You will implement Expected Sarsa and Q-learning, on Cliff World.
Name:Planning, Learning & Acting
Description:Up until now, you might think that learning with and without a model are two distinct, and in some ways, competing strategies: planning with Dynamic Programming verses sample-based learning via TD methods. This week we unify these two strategies with the Dyna architecture. You will learn how to estimate the model from data and then use this model to generate hypothetical experience (a bit like dreaming) to dramatically improve sample efficiency compared to sample-based methods like Q-learning. In addition, you will learn how to design learning systems that are robust to inaccurate models.
In this course, you will learn about several algorithms that can learn near optimal policies based on trial and error interaction with the environment---learning from the agent’s own experience. Learning from actual experience is striking because it requires no prior knowledge of the environment’s dynamics, yet can still attain optimal behavior. We will cover intuitively simple but powerful Monte Carlo methods, and temporal difference learning methods including Q-learning. We will wrap up this course investigating how we can get the best of both worlds: algorithms that can combine model-based planning (similar to dynamic programming) and temporal difference updates to radically accelerate learning. By the end of this course you will be able to: - Understand Temporal-Difference learning and Monte Carlo as two strategies for estimating value functions from sampled experience - Understand the importance of exploration, when using sampled experience rather than dynamic programming sweeps within a model - Understand the connections between Monte Carlo and Dynamic Programming and TD. - Implement and apply the TD algorithm, for estimating value functions - Implement and apply Expected Sarsa and Q-learning (two TD methods for control) - Understand the difference between on-policy and off-policy control - Understand planning with simulated experience (as opposed to classic planning strategies) - Implement a model-based approach to RL, called Dyna, which uses simulated experience - Conduct an empirical study to see the improvements in sample efficiency when using Dyna
