Spark, Hadoop, and Snowflake for Data Engineering

所在平台: Coursera

课程主页: https://www.coursera.org/learn/spark-hadoop-snowflake-data-engineering

课程评论：没有评论

第一个写评论        关注课程

课程简介

课程名称：Spark、Hadoop 和 Snowflake 数据工程 课程概述：本课程主要面向对工程和科学感兴趣的本科一、二年级学生、高中生以及对编程有兴趣的专业人士。学习者将掌握构建高效和可扩展的数据管道所需的技能，探索核心数据工程平台（如Hadoop、Spark 和 Snowflake），并学习如何优化和管理这些平台。课程还将深入Databricks，这是一款能够执行数据分析和机器学习任务的强大平台。 课程大纲： 1. **概览与PySpark介绍**： 在此模块中，学习者将了解不同数据工程平台（如Hadoop 和 Spark）的工作原理，并将其应用于实际场景。首先，探索Hadoop的基础知识，用于存储和处理大数据。接下来，深入了解Spark的概念、分布式计算、延迟执行和Spark SQL。到本周结束时，学习者将获得PySpark DataFrames的实践经验，包括DataFrame方法和延迟执行策略。 2. **Snowflake**： 本模块探索Snowflake平台，深入了解其架构和关键概念。通过在Snowflake Web UI中的实践，学习者将学习创建表、管理资源组，并使用Snowflake Python连接器与表进行交互。本周结束时，学习者将全面了解Snowflake的架构和实际应用，有能力有效导航和利用该平台进行数据管理和分析。 3. **Azure Databricks和MLFlow**： 在此模块中，学习者将实践管理机器学习工作流的基本技能，使用Databricks和MLFlow。首先创建Databricks工作区并配置集群，为高效数据分析奠定基础。接下来，利用PySpark在Databricks工作区加载示例数据集，实现数据操作和探索。最后，在Databricks环境中安装MLFlow，让学习者能够协调整个机器学习生命周期。本周结束时，学习者将能够在Databricks中设计、跟踪和管理机器学习实验，确保数据驱动过程中的精准性、可重复性和最佳决策。 4. **DataOps 和运营方法论**： 此模块探讨Kaizen、DevOps 和 DataOps的概念，以及这些方法论如何协同促进高效顺畅的数据工程工作流。通过实际案例，学习者将了解Kaizen的持续改进理念、DevOps的协作实践以及DataOps对数据质量和整合的关注，如何共同提升数据工程平台的开发、部署和管理能力。到本周末，学习者将掌握优化数据工程流程所需的知识和视角，从而交付可扩展、可靠且高质量的解决方案。

课程大纲

Name:Overview and Introduction to PySpark

Description:In this module, you will learn how to work with different data engineering platforms, such as Hadoop and Spark, and apply their concepts to real-world scenarios. First, you will explore the fundamentals of Hadoop to store and process big data. Next, you will delve into Spark concepts, distributed computing, deferred execution, and Spark SQL. By the end of the week, you will gain hands-on experience with PySpark DataFrames, DataFrame methods, and deferred execution strategies.

Name:Snowflake

Description:In this module, you will explore the Snowflake platform, gaining insights into its architecture and key concepts. Through hands-on practice in the Snowflake Web UI, you'll learn to create tables, manage warehouses, and use the Snowflake Python Connector to interact with tables. By the end of this week, you'll solidify your understanding of Snowflake's architecture and practical applications, emerging with the ability to effectively navigate and leverage the platform for data management and analysis.

Name:Azure Databricks and MLFLow

Description:In this module, you will practice the essential skills for seamlessly managing machine learning workflows using Databricks and MLFlow. First, you will create a Databricks workspace and configure a cluster, setting the stage for efficient data analysis. Next, you will load a sample dataset into the Databricks workspace using the power of PySpark, enabling data manipulation and exploration. Finally, you will install MLFlow either locally or within the Databricks environment, gaining the ability to orchestrate the entire machine learning lifecycle. By the end of this week, you will be able to craft, track, and manage machine learning experiments within Databricks, ensuring precision, reproducibility, and optimal decision-making throughout your data-driven journey.

Name:DataOps and Operations Methodologies

Description:In this module, you will explore the concepts of Kaizen, DevOps, and DataOps and how these methodologies synergistically contribute to efficient and seamless data engineering workflows. Through practical examples, you will learn how Kaizen's continuous improvement philosophy, DevOps' collaborative practices, and DataOps' focus on data quality and integration converge to enhance the development, deployment, and management of data engineering platforms. By the end of this week, you will have the knowledge and perspective needed to optimize data engineering processes and deliver scalable, reliable, and high-quality solutions.

课程评论(0条)