|
所在平台: Coursera |
课程主页: https://www.coursera.org/learn/spark-sql
课程评论：没有评论
课程名称：使用 Spark SQL 的分布式计算 课程概述：本课程专注于大数据，旨在为具备 SQL 基础的学生提供学习 Apache Spark 进行分布式计算的机会。学生将深入了解这一开源标准，以处理大型数据集，并获得使用 Spark 进行数据分析的基础知识，为后续结合数据与高级分析的学习奠定基础。课程包含四个模块，逐步推进，最终学生将理解 Spark 的架构、在 Spark 中的查询、优化 Spark SQL 的常见方法以及如何构建可靠的数据管道。 模块一：引入 Spark 该模块介绍 Spark 及 Databricks 环境，包括 Spark 如何分布计算以及 Spark SQL 的基本概念。学生将能够讨论分布式计算的核心概念，并识别使用场景，了解 Apache Spark™ 的基本数据结构，即 DataFrame。 模块二：Spark 核心概念 在这一模块中，学生将学习 Spark 的核心概念，如存储与计算、缓存、分区，以及通过 Spark UI 排查性能问题。还将探讨 Apache Spark 3.x 的新特性，如自适应查询执行，并学习提高查询性能的常见方法。 模块三：工程数据管道 该模块帮助学生识别数据应用的一般需求，能够以多种格式访问数据，并比较这些格式之间的优缺点。学生将探索半结构化的 JSON 数据（在大数据环境中常见），研究模式及并行数据写入，创建一个端到端的数据管道，实现数据读取、转换及结果保存。 模块四：数据湖、数据仓库和湖屋 本模块将识别数据湖、数据仓库和湖屋的关键特性。湖屋结合了数据湖的可扩展性和低成本存储与数据仓库的速度和 ACID 事务保障。学生将通过将 Spark 与开源项目 Delta Lake 相结合，构建生产级湖屋。 通过本课程，学生将提升 SQL 和分布式计算技能，更加熟练地进行高级分析，为未来转型为数据科学家奠定基础。
Name:Introduction to Spark
Description:In this module, you will be able to discuss the core concepts of distributed computing and be able to recognize when and where to apply them. You'll be able to identify the basic data structure of Apache Spark™, known as a DataFrame. Additionally, you will use the collaborative Databricks workspace and write SQL code that executes against a cluster of machines.
Name:Spark Core Concepts
Description:In this module, you will be able to explain the core concepts of Spark. You will learn common ways to increase query performance by caching data and modifying Spark configurations. You will also use the Spark UI to analyze performance and identify bottlenecks, as well as optimize queries with Adaptive Query Execution.
Name:Engineering Data Pipelines
Description:In this module, you will be able to identify and discuss the general demands of data applications. You'll be able to access data in a variety of formats and compare and contrast the tradeoffs between these formats. You will explore and examine semi-structured JSON data (common in big data environments) as well as schemas and parallel data writes. You will be able to create an end-to-end pipeline that reads data, transforms it, and saves the result.
Name:Data Lakes, Warehouses and Lakehouses
Description:In this module, you will identify the key characteristics of data lakes, data warehouses, and lakehouses. Lakehouses combine the scalability and low-cost storage of data lakes with the speed and ACID transactional guarantees of data warehouses. You will build a production grade lakehouse by combining Spark with the open-source project, Delta Lake. Whoever said time travel isn't possible hasn't been to a lakehouse!
This course is all about big data. It’s for students with SQL experience that want to take the next step on their data journey by learning distributed computing using Apache Spark. Students will gain a thorough understanding of this open-source standard for working with large datasets. Students will gain an understanding of the fundamentals of data analysis using SQL on Spark, setting the foundation for how to combine data with advanced analytics at scale and in production environments. The four modules build on one another and by the end of the course you will understand: the Spark architecture, queries within Spark, common ways to optimize Spark SQL, and how to build reliable data pipelines. The first module introduces Spark and the Databricks environment including how Spark distributes computation and Spark SQL. Module 2 covers the core concepts of Spark such as storage vs. compute, caching, partitions, and troubleshooting performance issues via the Spark UI. It also covers new features in Apache Spark 3.x such as Adaptive Query Execution. The third module focuses on Engineering Data Pipelines including connecting to databases, schemas and data types, file formats, and writing reliable data. The final module covers data lakes, data warehouses, and lakehouses. Students build production grade data pipelines by combining Spark with the open-source project Delta Lake. By the end of this course, students will hone their SQL and distributed computing skills to become more adept at advanced analysis and to set the stage for transitioning to more advanced analytics as Data Scientists.
