|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/cca-131-cloudera-certified-hadoop-and-spark-administrator/
课程评论：没有评论
**课程名称：** 使用CDH搭建Hadoop和Spark集群实战指南 **课程概述：** 本课程由Cloudera提供，专注于指导学员一步步完成使用Cloudera Distribution of Hadoop (CDH) 搭建Hadoop和Spark集群的过程。课程内容涵盖了集群的安装、配置、管理、安全加固、性能测试以及故障排除。 **核心学习目标：** 1. **安装**: * 掌握Cloudera Manager、CDH及相关生态项目（如HDFS, YARN, Spark, Hive, Impala, HBase, Kafka等）的安装流程。 * 学会搭建本地CDH仓库。 * 进行Hadoop安装所需的操作系统级配置。 * 安装Cloudera Manager服务器和代理。 * 通过Cloudera Manager安装CDH。 * 向现有集群添加新节点。 * 使用Cloudera Manager添加新的集群服务。 2. **配置**: * 掌握Hadoop集群的日常管理所需的基础和高级配置。 * 学习通过Cloudera Manager配置各项服务。 * 创建HDFS用户主目录。 * 配置NameNode和ResourceManager的高可用性 (HA)。 * 配置HiveServer2/Impala的代理。 3. **管理**: * 学习维护和修改集群以支持企业日常运营。 * 执行集群重新平衡操作。 * 设置磁盘空间过载告警。 * 定义并安装机架拓扑脚本。 * 在集群中安装新型I/O压缩库。 * 根据反馈调整YARN资源分配。 * 对节点的启用/禁用进行管理。 4. **安全**: * 学习启用相关服务并根据安全策略配置集群。 * 掌握基础安全实践。 * 配置HDFS访问控制列表 (ACLs)。 * 安装和配置Sentry进行细粒度权限管理。 * 配置Hue用户授权和认证。 * 启用/配置日志和查询的敏感信息 redaction。 * 在HDFS中创建加密区域。 5. **测试**: * 进行集群性能基准测试，评估系统配置的运行效率。 * 通过HTTP执行文件系统命令。 * 学习在集群内部或跨集群高效地复制数据。 * 创建/恢复HDFS目录的快照。 * 获取/设置文件或目录结构的ACLs。 * 对集群进行I/O、CPU和网络性能基准测试。 6. **故障排除**: * 学习查找问题根源、优化低效执行及解决资源争用。 * 解决Cloudera Manager中的错误/警告。 * 解决集群运行时的性能问题/错误。 * 确定应用失败原因。 * 配置Fair Scheduler以解决应用程序延迟。 **课程方法：** 课程将从使用Cloudera QuickStart VM开始（建议内存16GB及以上、四核处理器），以便学员熟悉Cloudera Manager。随后，学员将了解GCP（Google Cloud Platform）概览，并使用模板创建7-8个虚拟机，同时配置外部硬盘以备HDFS使用。之后，将进行服务器自动化设置（Ansible），并搭建本地Cloudera Manager和CDH的软件包仓库。接着，会使用自定义数据库安装Cloudera Manager，并通过其内置的向导完成CDH的安装，包括HDFS、YARN、Spark、Hive、Impala、HBase、Kafka等的配置，并学习HA、调度器、Parcels等概念。
Cloudera is one of the leading vendor for distributions related to Hadoop and Spark. As part of this Practical Guide, you will learn step by step process of setting up Hadoop and Spark Cluster using CDH.Install - Demonstrate an understanding of the installation process for Cloudera Manager, CDH, and the ecosystem projects.Set up a local CDH repositoryPerform OS-level configuration for Hadoop installationInstall Cloudera Manager server and agentsInstall CDH using Cloudera ManagerAdd a new node to an existing clusterAdd a service using Cloudera ManagerConfigure - Perform basic and advanced configuration needed to effectively administer a Hadoop clusterConfigure a service using Cloudera ManagerCreate an HDFS user's home directoryConfigure NameNode HAConfigure ResourceManager HAConfigure proxy for Hiveserver2/ImpalaManage - Maintain and modify the cluster to support day-to-day operations in the enterpriseRebalance the clusterSet up alerting for excessive disk fillDefine and install a rack topology scriptInstall new type of I/O compression library in clusterRevise YARN resource assignment based on user feedbackCommission/decommission a nodeSecure - Enable relevant services and configure the cluster to meet goals defined by security policy; demonstrate knowledge of basic security practicesConfigure HDFS ACLsInstall and configure SentryConfigure Hue user authorization and authenticationEnable/configure log and query redactionCreate encrypted zones in HDFSTest - Benchmark the cluster operational metrics, test system configuration for operation and efficiencyExecute file system commands via HTTPFSEfficiently copy data within a cluster/between clustersCreate/restore a snapshot of an HDFS directoryGet/set ACLs for a file or directory structureBenchmark the cluster (I/O, CPU, network)Troubleshoot - Demonstrate ability to find the root cause of a problem, optimize inefficient execution, and resolve resource contention scenariosResolve errors/warnings in Cloudera ManagerResolve performance problems/errors in cluster operationDetermine reason for application failureConfigure the Fair Scheduler to resolve application delaysOur ApproachYou will start with creating Cloudera QuickStart VM (in case you have laptop with 16 GB RAM with Quad Core). This will facilitate you to get comfortable with Cloudera Manager.You will be able to sign up for GCP and avail credit up to $300 while offer lasts. Credits are valid up to year.You will then understand brief overview about GCP and provision 7 to 8 Virtual Machines using templates. You will also attaching external hard drive to configure for HDFS later.Once servers are provisioned, you will go ahead and set up Ansible for Server Automation.You will take care of local repository for Cloudera Manager and Cloudera Distribution of Hadoop using Packages.You will then setup Cloudera Manager with custom database and then Cloudera Distribution of Hadoop using Wizard that comes as part of Cloudera Manager.As part of setting up of Cloudera Distribution of Hadoop you will setup HDFS, learn HDFS Commands, Setup YARN, Configure HDFS and YARN High Availability, Understand about Schedulers, Setup Spark, Transition to Parcels, Setup Hive and Impala, Setup HBase and Kafka etc.
