|
所在平台: Coursera |
课程主页: https://www.coursera.org/learn/scala-spark-big-data
课程评论：没有评论
课程名称：使用Scala和Spark进行大数据分析 课程概述：本课程着重于利用功能性概念操作分布在集群上的大数据，这在工业界日益普遍，并且被认为是功能性思想在工业中首次广泛应用的一个例证。通过MapReduce、Hadoop及近期的Apache Spark（一个基于Scala的快速内存分布式集合框架）的流行，显示出这一趋势。在课程中，我们将了解到数据并行范式如何扩展到分布式案例中，并将始终使用Spark。我们将详细讨论Spark的编程模型，仔细理解它在何时和如何与熟悉的编程模型（如共享内存并行集合或顺序Scala集合）有何不同。通过Spark和Scala的实际例子，我们将学习在分布过程中应考虑的重要问题，如延迟和网络通信，以及如何有效解决这些问题以提高性能。 学习成果：完成本课程后，您将能够： - 从持久化存储中读取数据并加载到Apache Spark中； - 使用Spark和Scala操作数据； - 以功能风格表达数据分析算法； - 识别如何在Spark中避免洗牌和重新计算。 推荐背景：您应至少具有一年的编程经验。对Java或C#的熟练掌握是理想的，但对C/C++、Python、Javascript或Ruby等其他语言的经验也是足够的。您应对使用命令行有一定的熟悉度。本课程建议在完成《并行编程》课程后学习。 课程大纲： 1. **开始与Spark基础**：设置Scala环境，完成示例作业，了解数据并行和分布式场景之间的联系，讨论分布式系统中的重要问题（如延迟和故障），学习Spark的基础知识，并动手分析实际数据集。 2. **减少操作与分布式键值对**：学习一种特殊的RDD，即配对RDD，掌握对大数据集的基本操作，如减少和连接。 3. **分区与洗牌**：探讨使用连接操作的性能影响，研究如何通过分区数据来提高数据本地性，从而优化Spark作业。 4. **结构化数据：SQL、数据框和数据集**：利用对数据移动成本的理解，学习如何使用Spark SQL以及其强大的优化器，介绍数据框和数据集，结合RDD与Spark SQL的自动优化功能。
Name:Getting Started + Spark Basics
Description:Get up and running with Scala on your computer. Complete an example assignment to familiarize yourself with our unique way of submitting assignments. In this week, we'll bridge the gap between data parallelism in the shared memory scenario (learned in the Parallel Programming course, prerequisite) and the distributed scenario. We'll look at important concerns that arise in distributed systems, like latency and failure. We'll go on to cover the basics of Spark, a functionally-oriented framework for big data processing in Scala. We'll end the first week by exercising what we learned about Spark by immediately getting our hands dirty analyzing a real-world data set.
Name:Reduction Operations & Distributed Key-Value Pairs
Description:This week, we'll look at a special kind of RDD called pair RDDs. With this specialized kind of RDD in hand, we'll cover essential operations on large data sets, such as reductions and joins.
Name:Partitioning and Shuffling
Description:This week we'll look at some of the performance implications of using operations like joins. Is it possible to get the same result without having to pay for the overhead of moving data over the network? We'll answer this question by delving into how we can partition our data to achieve better data locality, in turn optimizing some of our Spark jobs.
Name:Structured data: SQL, Dataframes, and Datasets
Description:With our newfound understanding of the cost of data movement in a Spark job, and some experience optimizing jobs for data locality last week, this week we'll focus on how we can more easily achieve similar optimizations. Can structured data help us? We'll look at Spark SQL and its powerful optimizer which uses structure to apply impressive optimizations. We'll move on to cover DataFrames and Datasets, which give us a way to mix RDDs with the powerful automatic optimizations behind Spark SQL.
Manipulating big data distributed over a cluster using functional concepts is rampant in industry, and is arguably one of the first widespread industrial uses of functional ideas. This is evidenced by the popularity of MapReduce and Hadoop, and most recently Apache Spark, a fast, in-memory distributed collections framework written in Scala. In this course, we'll see how the data parallel paradigm can be extended to the distributed case, using Spark throughout. We'll cover Spark's programming model in detail, being careful to understand how and when it differs from familiar programming models, like shared-memory parallel collections or sequential Scala collections. Through hands-on examples in Spark and Scala, we'll learn when important issues related to distribution like latency and network communication should be considered and how they can be addressed effectively for improved performance. Learning Outcomes. By the end of this course you will be able to: - read data from persistent storage and load it into Apache Spark, - manipulate data with Spark and Scala, - express algorithms for data analysis in a functional style, - recognize how to avoid shuffles and recomputation in Spark, Recommended background: You should have at least one year programming experience. Proficiency with Java or C# is ideal, but experience with other languages such as C/C++, Python, Javascript or Ruby is also sufficient. You should have some familiarity using the command line. This course is intended to be taken after Parallel Programming: https://www.coursera.org/learn/parprog1.
