|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/apache-airflow-bootcamp-hands-on-workflow-automation/
课程评论：没有评论
**课程名称：** Apache Airflow Bootcamp: Hands-On Workflow Automation with Practical Examples **课程概述：** 本课程旨在帮助零基础学员快速掌握Apache Airflow，成为一名自信的工作流自动化专家。课程内容涵盖从安装到高级功能的全面讲解，并通过实际项目和真实世界的案例，提供大量的动手实践机会。学完本课程，您将能够自信地构建和管理海量工作流，显著提升数据处理效率。 **课程主要内容：** 1. **Airflow入门：** * Airflow的作用与优势 * 在项目中使用Airflow的利弊分析 2. **Airflow架构：** * Airflow核心组件详解 * Airflow关键术语解读 3. **配置与安装：** * Airflow分步安装指南 * `airflow.cfg`文件的作用与配置 4. **Airflow Web UI：** * Web UI的启动与导航 * Grid View, Graph View, Calendar View, Task Duration View, Code View, Variable View, Gantt View 等常用视图介绍 5. **DAGs (Directed Acyclic Graphs)：** * DAG概念解析 * 创建DAG定义文件 * DAG创建方法 * DAG Run, `default_arguments`, DAG参数 * DAGs中的参数传递 (`TriggerDagRunOperator`) * 调度概念：`depends_on_past`, `wait_for_downstream`, `catchup`, `backfill` 6. **Airflow CLI与速查表：** * Airflow命令行界面的使用 * 常用命令速查 7. **Airflow中的Tasks：** * Task与Task Instance的定义 * Task的生命周期 8. **Airflow Operators：** * 详细介绍BashOperator, PostgresOperator, PythonOperator, SqliteOperator, EmailOperator等常用Operator 9. **Sensors：** * 介绍FileSensor, SQLSensor, TimeDeltaSensor, TimeSensor等多种Sensor的应用 10. **Branching：** * 实现分支逻辑：BranchSQLOperator, BranchPythonOperator, BranchDayOfWeekOperator, BranchDateTimeOperator, ShortCircuitOperator 11. **DAG依赖与TaskGroups：** * 管理DAG依赖关系 * 使用TaskGroups组织Tasks * `TriggerDagRunOperator`与`ExternalTaskSensor`的应用 12. **Hooks：** * 理解并使用PostgresHook, SHook等Hook 13. **资源管理：** * 使用Pools和Task Priorities管理资源 14. **Executors：** * StandardExecutor与LocalExecutor介绍 * 从SequentialExecutor迁移到LocalExecutor 15. **Airflow元数据数据库与Roles：** * Airflow元数据数据库详解 * 用户与角色管理：Admin, Public, User, Operator 等角色；创建自定义角色 16. **SLA (Service Level Agreement)：** * 设置Task和DAG级别的SLA * 处理SLA超时 17. **高级概念：** * XComs用于Tasks间通信 * 获取Context参数与Callback函数 * 处理Zombie Tasks, SIGTERM, SIGKILL错误 **课程目标：** 通过本课程，您将能够熟练运用Apache Airflow自动化复杂的数据管道，为您的数据工程职业生涯开启新的机遇。掌握Airflow意味着能够更高效地处理数据，提升工作效率。无论您是想晋升职业、构建更高效的数据管道，还是仅仅想学习Airflow，本课程都将是您的理想选择。
Hello and welcome to the Apache Airflow Bootcamp: Hands-On Workflow Automation with Practical Examples! Throughout my career, I've built and managed countless workflows using Apache Airflow, and I'm excited to share my knowledge with you.This course is designed to take you from a complete beginner to a confident user of Apache Airflow. We'll cover everything from installation to advanced features, and you'll get hands-on experience through practical examples and real-world projectsWhat's included in the course ?Introduction to AirflowUnderstanding the purpose and benefits of using Apache Airflow.Pros and cons of adopting Airflow in your projects.Airflow ArchitectureA detailed look into the components that make up Airflow.Key terminology used in Airflow.Configuration and InstallationStep-by-step guide to installing Airflow.The role and configuration of the airflow.cfg file.Airflow Web UI ViewsLaunching and navigating the Airflow Web UI.DAG ViewGrid ViewGraph ViewCalendar ViewTask Duration ViewCode ViewVariable ViewGantt ViewDAGs (Directed Acyclic Graphs)What is a DAG?Creating a DAG definition file.Different methods for DAG creation.Understanding DAG Run, default_arguments, and DAG arguments.Using parameters in DAGs and passing parameters through TriggerDagRunOperator.Scheduling concepts including depends_on_past, wait_for_downstream, catchup, and backfill.Airflow CLI and CheatsheetUtilizing the Airflow CLI for various operations.Handy cheatsheet for quick reference.Tasks in AirflowWhat are tasks and task instances?The lifecycle of a task. Operators in AirflowDetailed exploration of operators including BashOperator, PostgresOperator, PythonOperator, SqliteOperator, and EmailOperator.SensorsUsing sensors like FileSensor, SQLSensor, TimeDeltaSensor, and TimeSensor.BranchingImplementing branching logic with BranchSQLOperator, BranchPythonOperator, BranchDayOfWeekOperator, BranchDateTimeOperator, and ShortCircuitOperator.DAG Dependencies and TaskGroupsManaging DAG dependencies and using TaskGroups.Using TriggerDagRunOperator and ExternalTaskSensor.HooksUnderstanding and using hooks such as PostgresHook and SHook.Resource ManagementManaging resources with pools and task priorities.Executors in AirflowDifferent types of executors: SequentialExecutor and LocalExecutor.Transitioning from SequentialExecutor to LocalExecutor.Airflow Metadata Database and RolesUnderstanding the Airflow metadata database.Managing roles: creating users with different roles, including admin, public, user, and operator roles.Creating custom roles and modifying existing ones.SLA (Service Level Agreement)Setting and managing task-level and DAG-level SLAs.Handling SLA misses.Advanced ConceptsUsing XComs for inter-task communication.Retrieving context parameters and using callback functions.Dealing with zombie tasks, SIGTERM, and SIGKILL errors.I believe that mastering workflow automation with Airflow can open up incredible opportunities in the field of data engineering. I've seen firsthand how it can transform the way we handle data, and I can't wait to see what you'll achieve with these skills.So, whether you're looking to advance your career, work on more efficient data pipelines, or just curious about Airflow, you're in the right place. Let's dive in and start creating some amazing workflows together. Are you ready? Let's get started!I wish you a great success!
