|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/learn-by-example-hadoop-mapreduce/
课程评论：没有评论
课程名称：通过实例学习：Hadoop和MapReduce处理大数据问题 课程概述：该课程由一支由四名成员组成的团队教授，其中包括两位斯坦福大学教育背景的前谷歌员工和两位前Flipkart首席分析师。这个团队在使用Java和处理数十亿数据行方面拥有数十年的实际经验。本课程是一个深入浅出的实践训练，围绕Hadoop、MapReduce和并行思维的艺术展开。 课程特色： - **深入浅出**：课程涵盖Hadoop的各个组成部分，深入探讨其工作原理，同时提供高层次的交互视图。 - **实践操作**：课程早期便着重于动手实践，教授如何通过虚拟机和云设置自己的Hadoop集群。 - **并行思维的艺术**：MapReduce彻底改变了人们处理大数据的思维方式，课程通过案例训练学员学会将问题拆分为可并行处理的单元。 课程内容： - 使用MapReduce生成社交网站的朋友推荐：利用协同过滤算法生成前10个朋友推荐。 - 构建搜索引擎的倒排索引：使用MapReduce并行化构建倒排索引的庞大任务。 - 从文本生成二元组：生成二元组并计算其在文本语料中的频率分布。 - 搭建Hadoop集群：在独立模式、伪分布模式和完全分布模式下安装Hadoop；设置基于Linux虚拟机的Hadoop集群；在AWS上使用Cloudera Manager搭建云Hadoop集群。 - 理解HDFS、MapReduce和YARN之间的交互。 - 自定义MapReduce作业：将多个MapReduce作业链式连接，编写自定义分区器。 - 全局排序：通过抽样输入文件对大量数据进行全局排序。 - 二级排序、MR单元测试、使用Hadoop Streaming API与Python集成。 基础内容包括： - MapReduce：Mapper、Reducer、排序/合并、分区、洗牌及排序。 - HDFS与YARN：Namenode、Datanode、资源管理器、节点管理器、MapReduce应用的结构、YARN调度、HDFS和YARN的配置与性能调优。 此课程将为学员提供丰富的技能和实践经验，以应对大数据时代的挑战。
Taught by a 4 person team including 2 Stanford-educated, ex-Googlers and 2 ex-Flipkart Lead Analysts. This team has decades of practical experience in working with Java and with billions of rows of data. This course is a zoom-in, zoom-out, hands-on workout involving Hadoop, MapReduce and the art of thinking parallel. Let's parse that. Zoom-in, Zoom-Out: This course is both broad and deep. It covers the individual components of Hadoop in great detail, and also gives you a higher level picture of how they interact with each other. Hands-on workout involving Hadoop, MapReduce: This course will get you hands-on with Hadoop very early on. You'll learn how to set up your own cluster using both VMs and the Cloud. All the major features of MapReduce are covered - including advanced topics like Total Sort and Secondary Sort. The art of thinking parallel: MapReduce completely changed the way people thought about processing Big Data. Breaking down any problem into parallelizable units is an art. The examples in this course will train you to "think parallel". What's Covered: Lot's of cool stuff.. Using MapReduce to Recommend friends in a Social Networking site: Generate Top 10 friend recommendations using a Collaborative filtering algorithm. Build an Inverted Index for Search Engines: Use MapReduce to parallelize the humongous task of building an inverted index for a search engine. Generate Bigrams from text: Generate bigrams and compute their frequency distribution in a corpus of text. Build your Hadoop cluster: Install Hadoop in Standalone, Pseudo-Distributed and Fully Distributed modes Set up a hadoop cluster using Linux VMs.Set up a cloud Hadoop cluster on AWS with Cloudera Manager.Understand HDFS, MapReduce and YARN and their interaction Customize your MapReduce Jobs: Chain multiple MR jobs togetherWrite your own Customized PartitionerTotal Sort: Globally sort a large amount of data by sampling input filesSecondary sorting Unit tests with MR UnitIntegrate with Python using the Hadoop Streaming API .. and of course all the basics: MapReduce: Mapper, Reducer, Sort/Merge, Partitioning, Shuffle and Sort HDFS & YARN: Namenode, Datanode, Resource manager, Node manager, the anatomy of a MapReduce application, YARN Scheduling, Configuring HDFS and YARN to performance tune your cluster.
