|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/real-world-hadoop-automating-hadoop-install-with-python/
课程评论：没有评论
**课程名称：** Real World Hadoop - Automating Hadoop install with Python! **课程概述：** 本课程是“Real World Vagrant - Automate a Cloudera Manager Build - Toyin Akin”课程的进阶内容。学员将学习如何使用 Python 自动化部署一个完整的 Hadoop 集群，包括 ZooKeeper、HDFS、YARN 和 Spark。 课程的核心是利用 Cloudera Manager API，通过 Python 代码来驱动 Cloudera Manager 完成 Hadoop 服务的部署和配置。Cloudera Manager API 提供了丰富的功能，包括： * **自动化部署：** 编程化地部署整个 Hadoop 集群，支持 HDFS、MapReduce、YARN、ZooKeeper、HBase、Hive、Oozie、Hue、Flume、Impala、Solr、Sqoop、Spark 和 Accumulo 等服务。 * **配置管理：** 获取和验证 Hadoop 服务的配置。 * **服务生命周期管理：** 对服务和角色执行管理操作，如启动、停止、重启、故障转移等，以及高级工作流，如高可用设置和节点移除。 * **监控与度量：** 监控服务和主机的健康状况，获取服务指标和时间序列度量数据，还可以监控用户作业和集群活动。 * **API 管理：** 管理 Cloudera Manager 本身，例如下载完整的集群部署描述。 * **高级功能（需额外许可证）：** 执行滚动重启和滚动升级，审计用户活动和访问，进行 HDFS 和 Hive 的备份及跨数据中心复制，检索用户 HDFS 用量和 MapReduce 资源使用率报告。 **学习路径建议：** 讲师推荐循序渐进的学习路径，以确保学员掌握 Hadoop 的基础知识和操作： 1. **Hadoop 生态系统入门 (Udemy 课程):** "Big Data Intro for IT Administrators, Devs and Consultants" 2. **Vagrant 虚拟环境实践 (Udemy 课程):** "Real World Vagrant For Distributed Computing" 3. **自动化 Cloudera Manager 构建 (Udemy 课程):** "Real World Vagrant - Automate a Cloudera Manager Build" 4. **通过 Cloudera Manager 部署 Hadoop (Udemy 课程):** "Real World Hadoop - Deploying Hadoop with Cloudera Manager" 5. **HDFS 文件操作实践 (Udemy 课程):** "Real World Hadoop - Hands on Enterprise Distributed Storage" 6. **Python 自动化 Hadoop 安装 (本课程):** "Real World Hadoop - Automating Hadoop install with Python!" 7. **Cloudera 和 Hadoop 升级实践 (Udemy 课程):** "Real World Hadoop - Upgrade Cloudera and Hadoop hands on" **目标学员：** 本课程适合已经具备一定 Hadoop 基础知识，并希望通过 Python 自动化部署和管理 Hadoop 集群的 IT 管理员、开发人员和顾问。在学习本课程前，建议学员先掌握前述的 Vagrant 和 Cloudera Manager 的基础知识。 **课程形式：** 本课程注重实践操作 (Hands on)。
Note: This course is built on top of the "Real World Vagrant - Automate a Cloudera Manager Build - Toyin Akin" course Deploy a Hadoop cluster (Zookeeper, HDFS, YARN, Spark) with Python! Instruct Cloudera Manager to do the work! Hands on. Here we use Python to instruct an already installed Cloudera Manager to deploy your Hadoop Services. .The Cloudera Manager API provides configuration and service lifecycle management, service health information and metrics, and allows you to configure Cloudera Manager itself. The API is served on the same host and port as the Cloudera Manager Admin Console, and does not require an extra process or extra configuration. The API supports HTTP Basic Authentication, accepting the same users and credentials as the Cloudera Manager Admin Console. . Here are some of the cool things you can do with Cloudera Manager via the API: Deploy an entire Hadoop cluster programmatically. Cloudera Manager supports HDFS, MapReduce, YARN, ZooKeeper, HBase, Hive, Oozie, Hue, Flume, Impala, Solr, Sqoop, Spark and Accumulo. Configure various Hadoop services and get config validation. Take admin actions on services and roles, such as start, stop, restart, failover, etc. Also available are the more advanced workflows, such as setting up high availability and decommissioning. Monitor your services and hosts, with intelligent service health checks and metrics. Monitor user jobs and other cluster activities. Retrieve timeseries metric data. Search for events in the Hadoop system. Administer Cloudera Manager itself. Download the entire deployment description of your Hadoop cluster in a json file. Additionally, with the appropriate licenses, the API lets you: Perform rolling restart and rolling upgrade. Audit user activities and accesses in Hadoop. Perform backup and cross data-center replication for HDFS and Hive. Retrieve per-user HDFS usage report and per-user MapReduce resource usage report. . Here I present a curriculum as to the current state of my Cloudera courses. My Hadoop courses are based on Vagrant so that you can practice and destroy your virtual environment before applying the installation onto real servers/VMs. . For those with little or no knowledge of the Hadoop eco systemUdemy course: Big Data Intro for IT Administrators, Devs and Consultants . I would first practice with Vagrant so that you can carve out a virtual environment on your local desktop. You don't want to corrupt your physical servers if you do not understand the steps or make a mistake.Udemy course: Real World Vagrant For Distributed Computing . I would then, on the virtual servers, deploy Cloudera Manager plus agents. Agents are the guys that will sit on all the slave nodes ready to deploy your Hadoop servicesUdemy course: Real World Vagrant - Automate a Cloudera Manager Build . Then deploy the Hadoop services across your cluster (via the installed Cloudera Manager in the previous step). We look at the logic regarding the placement of master and slave services.Udemy course: Real World Hadoop - Deploying Hadoop with Cloudera Manager . If you want to play around with HDFS commands (Hands on distributed file manipulation).Udemy course: Real World Hadoop - Hands on Enterprise Distributed Storage. . You can also automate the deployment of the Hadoop services via Python (using the Cloudera Manager Python API). But this is an advanced step and thus I would make sure that you understand how to manually deploy the Hadoop services first.Udemy course: Real World Hadoop - Automating Hadoop install with Python! . There is also the upgrade step. Once you have a running cluster, how do you upgrade to a newer hadoop cluster (Both for Cloudera Manager and the Hadoop Services).Udemy course: Real World Hadoop - Upgrade Cloudera and Hadoop hands on
