|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/learning-pyspark/
课程评论：没有评论
课程名称：学习 PySpark 课程概述：Apache Spark 是一个开源的分布式引擎，用于查询和处理数据。本教程将简要介绍 Spark 及其技术栈，提供有效且节省时间的技巧，教您如何利用 Python 的强大功能在 Spark 生态系统中进行操作。您将首先了解 Apache Spark 的架构，并学习如何为 Spark 设置 Python 环境。接下来，将学习不同的数据收集技术，并区分数据处理的方法。我们还将深入探讨 RDD（弹性分布式数据集）并与 DataFrame 进行比较，提供从文件和 HDFS 读取数据的示例，以及如何使用反射或编程方式指定 DataFrame 的模式。课程将描述惰性执行的概念，并概述与 RDD 和 DataFrame 相关的各种转换和操作。最后，您将学习如何使用 SQL 与 DataFrame 进行交互。完成本教程后，您将掌握使用 Spark DataFrames 处理数据的方法，熟练掌握分布式数据处理的数据收集技术。 关于作者：Tomasz Drabas 是微软的数据科学家，现居于西雅图地区。他在数据分析和数据科学领域拥有超过 12 年的国际经验，涉及多个行业，包括先进技术、航空公司、电信、金融和咨询。Tomasz 于 2003 年在波兰华沙的波兰航空公司开始他的职业生涯，同时完成战略管理的硕士学位。2007 年，他移居悉尼，攻读新南威尔士大学航空学院的运筹学博士学位；他的研究跨越离散选择建模与航空公司运筹研究。在悉尼期间，他在 Beyond Analysis Australia 担任数据分析师，并在 Vodafone Hutchison Australia 担任高级数据分析师/数据科学家等职务。他还发表过科学论文，参加国际会议，并担任科学期刊的审稿人。2015 年，他迁居西雅图，开始在微软工作，并参与多个涉及高维特征空间问题的项目。
Apache Spark is an open-source distributed engine for querying and processing data. In this tutorial, we provide a brief overview of Spark and its stack. This tutorial presents effective, time-saving techniques on how to leverage the power of Python and put it to use in the Spark ecosystem. You will start by getting a firm understanding of the Apache Spark architecture and how to set up a Python environment for Spark. You'll learn about different techniques for collecting data, and distinguish between (and understand) techniques for processing data. Next, we provide an in-depth review of RDDs and contrast them with DataFrames. We provide examples of how to read data from files and from HDFS and how to specify schemas using reflection or programmatically (in the case of DataFrames). The concept of lazy execution is described and we outline various transformations and actions specific to RDDs and DataFrames. Finally, we show you how to use SQL to interact with DataFrames. By the end of this tutorial, you will have learned how to process data using Spark DataFrames and mastered data collection techniques by distributed data processing.About the AuthorTomasz Drabas is a Data Scientist working for Microsoft and currently residing in the Seattle area. He has over 12 years' international experience in data analytics and data science in numerous fields: advanced technology, airlines, telecommunications, finance, and consulting. Tomasz started his career in 2003 with LOT Polish Airlines in Warsaw, Poland while finishing his Master's degree in strategy management. In 2007, he moved to Sydney to pursue a doctoral degree in operations research at the University of New South Wales, School of Aviation; his research crossed boundaries between discrete choice modeling and airline operations research. During his time in Sydney, he worked as a Data Analyst for Beyond Analysis Australia and as a Senior Data Analyst/Data Scientist for Vodafone Hutchison Australia among others. He has also published scientific papers, attended international conferences, and served as a reviewer for scientific journals. In 2015 he relocated to Seattle to begin his work for Microsoft. While there, he has worked on numerous projects involving solving problems in high-dimensional feature space.
