|
所在平台: Coursera |
课程主页: https://www.coursera.org/learn/scala2-spark-big-data
课程评论：没有评论
课程名称：使用 Scala 和 Spark 进行大数据分析（Scala 2 版本） 课程概述：在本课程中，您将学习如何使用函数式编程概念处理分布在集群上的大数据。当前，行业中广泛运用的功能思想主要体现在 MapReduce、Hadoop 和 Apache Spark 等工具中，后者是用 Scala 编写的快速内存分布式集合框架。本课程将深入探讨数据并行范式如何扩展到分布式案例，重点介绍 Spark 的编程模型，特别是在与共享内存并行集合或顺序 Scala 集合等常见编程模型的不同之处。通过使用实际示例，您将了解分布式计算中与延迟和网络通信相关的重要问题，并学习如何有效解决这些问题以提高性能。 学习成果：完成本课程后，您将能够： - 从持久存储中读取数据并将其加载到 Apache Spark 中； - 使用 Spark 和 Scala 处理数据； - 以函数式风格表示数据分析的算法； - 识别如何避免 Spark 中的洗牌和重新计算。 推荐背景：您应该至少具备一年的编程经验，熟悉 Java 或 C# 是理想选择，但 C/C++、Python、JavaScript 或 Ruby 等语言的经验也是足够的。此外，建议您有一定的命令行使用经验。本课程建议在完成并行编程课程后进行学习。 课程大纲： 第1部分：入门 + Spark 基础 描述：熟悉 Scala 在您计算机上的运行。完成一个示例任务以了解我们独特的作业提交方式。我们将连接共享内存场景与分布式场景之间的差距，探讨在分布式系统中出现的重要问题，如延迟和故障，最后通过分析真实数据集来实践所学的 Spark 知识。 第2部分：归约操作与分布式键值对 描述：本周，我们将关注一种特定类型的 RDD，称为配对 RDD。掌握这类 RDD 后，我们将涵盖对大数据集的基本操作，如归约和连接。 第3部分：分区和洗牌 描述：本周，我们将探讨使用连接等操作的性能影响。是否可以在不增加网络数据传输成本的情况下获得相同的结果？我们将通过研究如何分区数据以优化数据本地性，寻求更好的 Spark 作业效能。 第4部分：结构化数据：SQL、数据框和数据集 描述：具备了对 Spark 作业中数据传输成本的理解，以及上周优化数据本地性的经验后，本周我们将着重讨论如何更轻松地实现类似的优化。结构化数据能否帮助我们？我们将研究 Spark SQL 及其强大的优化器，应用结构进行显著的优化，同时介绍数据框和数据集，它们将 RDD 与 Spark SQL 背后的强大自动优化结合起来。
Part: 1
Title:Getting Started + Spark Basics
Description:Get up and running with Scala on your computer. Complete an example assignment to familiarize yourself with our unique way of submitting assignments. In this week, we'll bridge the gap between data parallelism in the shared memory scenario (learned in the Parallel Programming course, prerequisite) and the distributed scenario. We'll look at important concerns that arise in distributed systems, like latency and failure. We'll go on to cover the basics of Spark, a functionally-oriented framework for big data processing in Scala. We'll end the first week by exercising what we learned about Spark by immediately getting our hands dirty analyzing a real-world data set.
Part: 2
Title:Reduction Operations & Distributed Key-Value Pairs
Description:This week, we'll look at a special kind of RDD called pair RDDs. With this specialized kind of RDD in hand, we'll cover essential operations on large data sets, such as reductions and joins.
Part: 3
Title:Partitioning and Shuffling
Description:This week we'll look at some of the performance implications of using operations like joins. Is it possible to get the same result without having to pay for the overhead of moving data over the network? We'll answer this question by delving into how we can partition our data to achieve better data locality, in turn optimizing some of our Spark jobs.
Part: 4
Title:Structured data: SQL, Dataframes, and Datasets
Description:With our newfound understanding of the cost of data movement in a Spark job, and some experience optimizing jobs for data locality last week, this week we'll focus on how we can more easily achieve similar optimizations. Can structured data help us? We'll look at Spark SQL and its powerful optimizer which uses structure to apply impressive optimizations. We'll move on to cover DataFrames and Datasets, which give us a way to mix RDDs with the powerful automatic optimizations behind Spark SQL.
Manipulating big data distributed over a cluster using functional concepts is rampant in industry, and is arguably one of the first widespread industrial uses of functional ideas. This is evidenced by the popularity of MapReduce and Hadoop, and most recently Apache Spark, a fast, in-memory distributed collections framework written in Scala. In this course, we'll see how the data parallel paradigm can be extended to the distributed case, using Spark throughout. We'll cover Spark's programming model in detail, being careful to understand how and when it differs from familiar programming models, like shared-memory parallel collections or sequential Scala collections. Through hands-on examples in Spark and Scala, we'll learn when important issues related to distribution like latency and network communication should be considered and how they can be addressed effectively for improved performance. Learning Outcomes. By the end of this course you will be able to: - read data from persistent storage and load it into Apache Spark, - manipulate data with Spark and Scala, - express algorithms for data analysis in a functional style, - recognize how to avoid shuffles and recomputation in Spark, Recommended background: You should have at least one year programming experience. Proficiency with Java or C# is ideal, but experience with other languages such as C/C++, Python, Javascript or Ruby is also sufficient. You should have some familiarity using the command line. This course is intended to be taken after Parallel Programming: https://www.coursera.org/learn/parprog1. Note that this version of the course uses Scala 2.13. You can find a more recent version of the course that uses Scala 3 here: https://www.coursera.org/learn/scala-spark-big-data
