|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/system-design-for-big-data-pipelines/
课程评论：没有评论
Coursera 课程《大数据管道系统设计》课程总结 本课程旨在弥合软件工程师在大数据管道设计和构建方面的技能差距。随着大数据技术的快速发展，构建可扩展、可靠且经济高效的大数据管道已成为一项核心技能，对于支持不断增长的数据量和应用程序至关重要。 课程内容涵盖： * **大数据管道构建模块：** 介绍大数据管道的基本组成部分、它们的功能以及在构建过程中可能遇到的挑战。 * **结构化设计流程：** 提供一个系统化的方法来设计和构建大数据管道。 * **核心构建模块深度解析：** 深入探讨各个构建模块的设计模式、优缺点、适用场景以及相关技术。 * **最佳实践推荐：** 在课程中贯穿始终地提供用于构建高质量大数据管道的最佳实践。 * **实际用例实现：** 通过分析和实现一个批处理用例和一个实时处理用例，展示如何将课程中学到的知识应用于解决实际业务问题。 通过本课程，学员将学习如何端到端地构建能有效利用大数据技术解决业务问题的大数据管道，并掌握如何以可扩展、可靠的方式整合这些管道，以及如何进行部署、安全和运维。课程强调从宏观层面理解整个大数据管道的运作，而不仅仅是关注单个组件。
Big data technologies have been growing exponentially over the past few years and have penetrated into every domain and industry in software development. It has become a core skill for a software engineer. Robust and effective big data pipelines are needed to support the growing volume of data and applications in the big data world. These pipelines have become business critical and help increase revenues and reduce cost.Do quality big data pipelines happen by magic? High quality designs that are scalable, reliable and cost effective are needed to build and maintain these pipelines.How do you build an end-to-end big data pipeline that leverages big data technologies and practices effectively to solve business problems? How do you integrate them in a scalable and reliable manner? How do you deploy, secure and operate them? How do you look at the overall forest and not just the individual trees? This course focuses on this skill gap.What are the topics covered in this course?We start off by discussing the building blocks of big data pipelines, their functions and challenges.We introduce a structured design process for building big data pipelines.We then discuss individual building blocks, focusing on the design patterns available, their advantages, shortcomings, use cases and available technologies.We recommend several best practices across the course.We finally implement two use cases for illustration on how to apply the learnings in the course to a real world problem. One is a batch use case and another is a real time use case.
