|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/real-world-spark-2-interactive-python-pyspark-core/
课程评论：没有评论
**课程名称：** Real World Spark 2 - Interactive Python pyspark Core **课程概述：** 本课程是基于“Real World Vagrant - Build an Apache Spark Development Env!”课程的进阶内容，旨在教授如何使用PySpark进行交互式数据分析。PySpark的Python shell不仅是学习Spark API的简单方式，也是强大的交互式数据分析工具。 **核心概念：** * **RDD (Resilient Distributed Dataset)：** Spark的核心抽象，是一种分布式的数据集合。RDD可以通过集合、Hadoop InputFormats（如HDFS文件）创建，或通过转换其他RDD来生成。 * **Spark监控与仪表盘：** 在创建RDD、执行转换和操作时，您将广泛使用Web UI的监控视图。每个SparkContext都会启动一个Web UI（默认端口4040），提供有关应用程序的有用信息，包括： * 调度器阶段和任务列表 * RDD大小和内存使用摘要 * 环境信息 * 运行中的Executor信息 * **Apache Spark的优势：** * **速度快：** 在内存中运行速度比Hadoop MapReduce快100倍，在磁盘上快10倍。 * **先进的DAG执行引擎：** 支持循环数据流和内存计算。 * **丰富的API：** 提供超过80个高级运算符，方便构建并行应用程序。 * **交互式使用：** 可通过Scala、Python和R shell进行交互式使用。 * **功能整合：** 可无缝结合SQL、流处理和复杂分析。 * **强大的生态系统：** 支持SQL和DataFrames、MLlib（机器学习）、GraphX（图计算）和Spark Streaming（流处理）等库，并可在同一应用程序中组合使用。 **先修课程：** 建议在学习本课程前，先完成“Real World Vagrant - Build an Apache Spark Development Env!”课程，以确保您已安装Spark开发环境。 **学习目标：** 掌握使用PySpark进行交互式数据分析的基本方法，理解Spark的核心概念，并了解如何利用Spark的强大功能进行数据处理和分析。
Note: This course is built on top of the "Real World Vagrant - Build an Apache Spark Development Env! - Toyin Akin" course. So if you do not have a Spark environment already installed (within a VM or directly installed), you can take the stated course above. Spark's python shell provides a simple way to learn the API, as well as a powerful tool to analyze data interactively. It is available in Python. Start it by running the following anywhere within a bash terminal within the built Virtual Machine pyspark Spark's primary abstraction is a distributed collection of items called a Resilient Distributed Dataset (RDD). RDDs can be created from collections, Hadoop InputFormats (such as HDFS files) or by transforming other RDDs Spark Monitoring and Instrumentation While creating RDDs, performing transformations and executing actions, you will be working heavily within the monitoring view of the Web UI. Every SparkContext launches a web UI, by default on port 4040, that displays useful information about the application. This includes: A list of scheduler stages and tasks A summary of RDD sizes and memory usage Environmental information. Information about the running executors Why Apache Spark. Apache Spark run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk. Apache Spark has an advanced DAG execution engine that supports cyclic data flow and in-memory computing. Apache Spark offers over 80 high-level operators that make it easy to build parallel apps. And you can use it interactively from the Scala, Python and R shells. Apache Spark can combine SQL, streaming, and complex analytics. Apache Spark powers a stack of libraries including SQL and DataFrames, MLlib for machine learning, GraphX, and Spark Streaming. You can combine these libraries seamlessly in the same application.
