|
所在平台: Coursera |
课程主页: https://www.coursera.org/learn/introduction-to-big-data-with-spark-hadoop
课程评论：没有评论
课程名称: 《大数据与Spark和Hadoop入门》 课程概述: 在这个数字时代，Bernard Marr 定义大数据为我们所产生的数字痕迹。本课程将教你大数据的特征及其在大数据分析中的应用。你将了解一些大数据处理工具的特点、优势、局限性和应用，探索Hadoop和Hive如何利用大数据的优势并克服一些挑战。 Hadoop是一个开源框架，允许在集群计算机上使用简单的编程模型进行大数据集的分布式处理。Hive作为一个数据仓库软件，提供了类似SQL的接口，以有效查询和操作与Hadoop集成的各种数据库和文件系统中的大数据集。 Apache Spark是一个开源处理引擎，提供了存储和利用大数据的新方式，围绕速度、易用性和分析构建。在课程中，你将学习如何利用Spark提供可靠的洞察，课程还将概述该平台的不同组成部分。 此外，你还将学习到弹性分布式数据集（RDD），它使得Spark集群节点间的并行处理成为可能。 课程大纲: 1. **什么是大数据?** 探索最新的大数据定义及其对日常生活和商业交易的影响，包括大数据用例。学习大数据如何利用并行处理、扩展和数据并行性，并对常用大数据工具进行探讨。 2. **Hadoop生态系统简介** 学习Apache Hadoop架构、生态系统及应用，包括分布式文件系统（HDFS）、MapReduce、Hive和HBase。通过实践实验，查询使用Hive添加的数据，启动单节点Hadoop集群并运行MapReduce作业。 3. **Apache Spark** 探索Apache Spark平台的属性与优势，学习功能编程和Lambda函数。深入了解RDD、并行编程和Spark的弹性。 4. **DataFrames和Spark SQL** 学习RDD在Apache Spark中的使用及其转换与行动。比较数据集与Spark最新数据抽象DataFrames的使用。优化Spark SQL，并探讨如何使用Catalyst和Tungsten提升内存优化。 5. **开发与运行环境选项** 探索Spark如何处理应用请求，以及如何使用Spark应用程序UI跟踪工作。了解Apache集群管理器及其组件，设置本地Standalone Spark实例的最佳实践。 6. **监控与调优** 学习如何连接Apache Spark用户界面web服务器，管理应用程序过程及调试问题。深入理解Spark如何管理内存和处理器资源。 7. **最终项目与评估** 进行实践实验，处理RDD和构建JSON数据的DataFrame，应用多种转换和行动。完成最终项目，创建DataFrame并使用Spark SQL进行数据处理，最后接受课程评估。 通过本课程，你将能够掌握大数据处理的基础知识，并有效应用Hadoop和Spark等工具进行数据分析。
Name:What Is Big Data?
Description:In this module, you’ll begin your acquisition of Big Data knowledge with the most up-to-date definition of Big Data. You’ll explore the impact of Big Data on everyday personal tasks and business transactions with Big Data Use Cases. You’ll also learn how Big Data uses parallel processing, scaling, and data parallelism. Going further, you’ll explore commonly used Big Data tools and explain the role of open-source in Big Data. Finally, you’ll go beyond the hype and explore additional Big Data viewpoints.
Name:Introduction to the Hadoop Ecosystem
Description:In this module, you'll gain a fundamental understanding of the Apache Hadoop architecture, ecosystem, practices, and commonly used applications, including Distributed File System (HDFS), MapReduce, Hive, and HBase. You’ll also gain practical skills in hands-on labs when you query the data added using Hive, launch a single-node Hadoop cluster using Docker, and run MapReduce jobs.
Name:Apache Spark
Description:In this module, you’ll turn your attention to the popular Apache Spark platform, where you will explore the attributes and benefits of Apache Spark and distributed computing. You'll gain key insights about functional programming and Lambda functions. You’ll also explore Resilient Distributed Datasets (RDDs), parallel programming, resilience in Apache Spark, and relate RDDs and parallel programming with Apache Spark. Then, you’ll dive into additional Apache Spark components and learn how Apache Spark scales with Big Data. Working with Big Data signals the need for working with queries, including structured queries using SQL. You’ll also learn about the functions, parts, and benefits of Spark SQL and DataFrame queries, and discover how DataFrames work with Spark SQL.
Name:DataFrames and Spark SQL
Description:In this module, you’ll learn about Resilient Distributed Datasets (RDDs), their uses in Apache Spark, and RDD transformations and actions. You'll compare the use of datasets with Spark's latest data abstraction, DataFrames. You'll learn to identify and apply basic DataFrame operations. You’ll explore Apache Spark SQL optimization and learn how Spark SQL and memory optimization benefit from using Catalyst and Tungsten. Finally, you’ll fortify your skills with guided hands-on lab to create a table view and apply data aggregation techniques.
Name:Development and Runtime Environment Options
Description:In this module, you’ll explore how Spark processes the requests that your application submits and learn how you can track work using the Spark Application UI. Because Spark application work happens on the cluster, you need to be able to identify Apache Cluster Managers, their components, and benefits. You’ll also know how to connect with each cluster manager and how and when you might want to set up a local, standalone Spark instance. Next, you’ll learn about Apache Spark application submission, including the use of Spark’s unified interface, “spark-submit,” and learn about options and dependencies. You’ll also describe and apply options for submitting applications, identify external application dependency management techniques, and list Spark Shell benefits. You’ll also look at recommended practices for Spark's static and dynamic configuration options and perform hands-on labs to use Apache Spark on IBM Cloud and run Spark on Kubernetes.
Name:Monitoring and Tuning
Description:Platforms and applications require monitoring and tuning to manage issues that inevitably happen. In this module, you'll learn about connecting the Apache Spark user interface web server and using the same UI web server to manage application processes. You’ll also identify common Apache Spark application issues and learn about debugging issues using the application UI and locating related log files. Further, you’ll discover and gain real-world knowledge about how Spark manages memory and processor resources using the hands-on lab.
Name:Final Project and Assessment
Description:In this module, you’ll perform a practice lab where you’ll explore two critical aspects of data processing using Spark: working with Resilient Distributed Datasets (RDDs) and constructing DataFrames from JSON data. You will also apply various transformations and actions on both RDDs and DataFrames to gain insights and manipulate the data effectively. Further, you’ll apply your knowledge in a final project where you will create a DataFrame by loading data from a CSV file and applying transformations and actions using Spark SQL. Finally, you’ll be assessed based on your learning from the course.
Bernard Marr defines Big Data as the digital trace that we are generating in this digital era. In this course, you will learn about the characteristics of Big Data and its application in Big Data Analytics. You will gain an understanding about the features, benefits, limitations, and applications of some of the Big Data processing tools. You’ll explore how Hadoop and Hive help leverage the benefits of Big Data while overcoming some of the challenges it poses. Hadoop is an open-source framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. Hive, a data warehouse software, provides an SQL-like interface to efficiently query and manipulate large data sets residing in various databases and file systems that integrate with Hadoop. Apache Spark is an open-source processing engine that provides users new ways to store and make use of big data. It is an open-source processing engine built around speed, ease of use, and analytics. In this course, you will discover how to leverage Spark to deliver reliable insights. The course provides an overview of the platform, going into the different components that make up Apache Spark. In this course, you will also learn about Resilient Distributed Datasets, or RDDs, that enable parallel processing across the nodes of a Spark cluster.
