|
所在平台: Coursera |
课程主页: https://www.coursera.org/learn/prediction-control-function-approximation
课程评论：没有评论
课程名称：使用函数近似进行预测与控制 概述：在本课程中，您将学习如何解决具有大规模、高维度和可能无限状态空间的问题。您将看到，价值函数的估计可以视为一个监督学习问题——函数近似——使您能够构建精心平衡泛化与区分的智能体，以最大化奖励。我们将开始研究如何将我们的策略评估或预测方法（如蒙特卡罗和时间差分）扩展到函数近似设置。您将学习强化学习中的特征构建技术，以及通过神经网络和反向传播进行表示学习。课程最后将深入探讨策略梯度方法；这是一种直接学习策略而无需学习价值函数的方法。在本课程中，您将解决两个连续状态控制任务，并研究策略梯度方法在连续动作环境中的优势。 先修课程：本课程强烈依赖于课程 1 和 2 的基础，学习者应在开始本课程之前完成这些课程。学习者还应对概率与期望、基础线性代数、基础微积分、Python 3.0（至少 1 年）以及从伪代码实现算法感到熟悉。 完成本课程后，您将能够： - 理解如何使用监督学习方法近似价值函数 - 理解在函数近似下的预测目标（价值估计） - 在无限状态空间的环境中实现具有函数近似的时间差分（状态聚合） - 理解特征构建的固定基础和神经网络方法 - 在连续状态环境中实现具有神经网络函数近似的时间差分 - 理解在转向函数近似时探索的新难度 - 对比控制的折现问题形式与平均回报问题形式 - 在连续状态控制任务上实现具有函数近似的期望 Sarsa 和 Q 学习 - 理解直接估计策略的目标（策略梯度目标） - 在离散状态环境中实现一种策略梯度方法（演员-评论家） 课程大纲： 1. 欢迎参加课程：介绍课程的讲师，并让您了解课程的内容。 2. 使用近似进行on-policy预测：学习如何为给定策略估计价值函数，尤其是在状态数量远大于智能体可用内存时。 3. 预测的特征构建：讨论构建特征的两种基本策略，学习使用神经网络和反向传播来适应特征。 4. 使用近似进行控制：看到经典的 TD 控制方法如何顺利扩展到函数近似设置，并学习如何在无限状态的 MDP 中找到最优策略。 5. 策略梯度：学习直接学习策略参数的方法，以及策略梯度方法在连续状态和动作空间任务中的应用。 此课程将为您提供强化学习中预测和控制的深厚基础，帮助您在相关应用中取得成功。
Name:Welcome to the Course!
Description:Welcome to the third course in the Reinforcement Learning Specialization: Prediction and Control with Function Approximation, brought to you by the University of Alberta, Onlea, and Coursera. In this pre-course module, you'll be introduced to your instructors, and get a flavour of what the course has in store for you. Make sure to introduce yourself to your classmates in the "Meet and Greet" section!
Name:On-policy Prediction with Approximation
Description:This week you will learn how to estimate a value function for a given policy, when the number of states is much larger than the memory available to the agent. You will learn how to specify a parametric form of the value function, how to specify an objective function, and how estimating gradient descent can be used to estimate values from interaction with the world.
Name:Constructing Features for Prediction
Description:The features used to construct the agent’s value estimates are perhaps the most crucial part of a successful learning system. In this module we discuss two basic strategies for constructing features: (1) fixed basis that form an exhaustive partition of the input, and (2) adapting the features while the agent interacts with the world via Neural Networks and Backpropagation. In this week’s graded assessment you will solve a simple but infinite state prediction task with a Neural Network and TD learning.
Name:Control with Approximation
Description:This week, you will see that the concepts and tools introduced in modules two and three allow straightforward extension of classic TD control methods to the function approximation setting. In particular, you will learn how to find the optimal policy in infinite-state MDPs by simply combining semi-gradient TD methods with generalized policy iteration, yielding classic control methods like Q-learning, and Sarsa. We conclude with a discussion of a new problem formulation for RL---average reward---which will undoubtedly be used in many applications of RL in the future.
Name:Policy Gradient
Description:Every algorithm you have learned about so far estimates a value function as an intermediate step towards the goal of finding an optimal policy. An alternative strategy is to directly learn the parameters of the policy. This week you will learn about these policy gradient methods, and their advantages over value-function based methods. You will also learn how policy gradient methods can be used to find the optimal policy in tasks with both continuous state and action spaces.
In this course, you will learn how to solve problems with large, high-dimensional, and potentially infinite state spaces. You will see that estimating value functions can be cast as a supervised learning problem---function approximation---allowing you to build agents that carefully balance generalization and discrimination in order to maximize reward. We will begin this journey by investigating how our policy evaluation or prediction methods like Monte Carlo and TD can be extended to the function approximation setting. You will learn about feature construction techniques for RL, and representation learning via neural networks and backprop. We conclude this course with a deep-dive into policy gradient methods; a way to learn policies directly without learning a value function. In this course you will solve two continuous-state control tasks and investigate the benefits of policy gradient methods in a continuous-action environment. Prerequisites: This course strongly builds on the fundamentals of Courses 1 and 2, and learners should have completed these before starting this course. Learners should also be comfortable with probabilities & expectations, basic linear algebra, basic calculus, Python 3.0 (at least 1 year), and implementing algorithms from pseudocode. By the end of this course, you will be able to: -Understand how to use supervised learning approaches to approximate value functions -Understand objectives for prediction (value estimation) under function approximation -Implement TD with function approximation (state aggregation), on an environment with an infinite state space (continuous state space) -Understand fixed basis and neural network approaches to feature construction -Implement TD with neural network function approximation in a continuous state environment -Understand new difficulties in exploration when moving to function approximation -Contrast discounted problem formulations for control versus an average reward problem formulation -Implement expected Sarsa and Q-learning with function approximation on a continuous state control task -Understand objectives for directly estimating policies (policy gradient objectives) -Implement a policy gradient method (called Actor-Critic) on a discrete state environment
