|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/real-time-spark-project-for-beginners-hadoop-spark-docker/
课程评论：没有评论
**课程名称：** 初学者实时 Spark 项目：Hadoop, Spark, Docker **课程概述：** 本课程旨在教授学员如何构建一个实时数据管道，以处理数据中心服务器实时生成的海量事件数据。通过学习，学员将能够利用 Apache Kafka、Apache Spark、Hadoop、PostgreSQL、Django 和 Flexmonster 等技术，结合 Docker 容器化技术，构建一个端到端的数据处理和可视化解决方案。 **核心技术与概念：** * **Apache Spark:** 一个开源的统一分析引擎，用于大规模数据处理。课程将重点介绍 Spark 的分布式计算能力、容错机制以及在 Hadoop 集群上的 Scala 和 PySpark 应用。 * **Apache Kafka:** 一个分布式事件存储和流处理平台，用于构建高吞吐量、低延迟的实时数据馈送平台。 * **Apache Hadoop:** 一个开源软件框架，用于分布式存储和处理大数据，课程将介绍其 MapReduce 编程模型。 * **Docker:** 一个开源平台，用于自动化部署、扩展和管理应用程序的容器化。 * **PostgreSQL:** 一个强大的、开源的对象-关系数据库系统。 * **Django:** 一个高性能的 Python Web 框架，用于快速开发 Web 应用程序。 * **Flexmonster:** 一个用于数据可视化和交互式数据分析的 JavaScript 组件。 **项目目标：** 学员将学习如何： 1. **搭建实时数据管道：** 使用 Kafka 接收和缓冲实时服务器事件数据。 2. **进行实时数据处理：** 利用 Spark 对流入的数据进行清洗、转换和分析。 3. **构建分布式存储：** 在 Hadoop 集群上实现数据的分布式存储。 4. **实现数据可视化：** 使用 Django 和 Flexmonster 构建 Web 应用，将处理后的数据以图表等形式可视化展示。 5. **应用容器化技术：** 将整个项目部署在 Docker 容器中，实现环境的隔离和易于管理。 **最终成果：** 通过本课程的学习和实践，学员将能够独立完成一个完整的实时数据处理与可视化项目，从而提升服务器稳定性，并为数据中心监控人员提供洞察，以便及时发现和解决问题。
In many data centers, different type of servers generate large amount of data(events, Event in this case is status of the server in the data center) in real-time. There is always a need to process these data in real-time and generate insights which will be used by the server/data center monitoring people and they have to track these server's status regularly and find the resolution in case of issues occurring, for better server stability. Since the data is huge and coming in real-time, we need to choose the right architecture with scalable storage and computation frameworks/technologies.Hence we want to build the Real Time Data Pipeline Using Apache Kafka, Apache Spark, Hadoop, PostgreSQL, Django and Flexmonster on Docker to generate insights out of this data.The Spark Project/Data Pipeline is built using Apache Spark with Scala and PySpark on Apache Hadoop Cluster which is on top of Docker.Data Visualization is built using Django Web Framework and Flexmonster.Apache Spark is an open-source unified analytics engine for large-scale data processing. Spark provides an interface for programming clusters with implicit data parallelism and fault tolerance.Apache Kafka is a distributed event store and stream-processing platform. It is an open-source system developed by the Apache Software Foundation written in Java and Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.Apache Hadoop is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. It provides a software framework for distributed storage and processing of big data using the MapReduce programming model.A NoSQL (originally referring to "non-SQL" or "non-relational") database provides a mechanism for storage and retrieval of data that is modeled in means other than the tabular relations used in relational databases.
