|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/writing-production-ready-etl-pipelines-in-python-pandas/
课程评论：没有评论
课程名称：使用Python/Pandas编写生产就绪的ETL管道 课程概述：本课程将详细介绍如何从零开始使用Python编写ETL管道，直到投入生产。我们将使用Python 3.9、Jupyter Notebook、Git和Github、Visual Studio Code、Docker和Docker Hub等必要工具，以及Pandas、boto3、pyyaml、awscli、jupyter、pylint、moto、coverage和memory-profiler等Python包。课程中将介绍并应用两种数据工程编码方法——函数式编程和面向对象编程。此外，我们还将介绍并应用开发Python代码的最佳实践，包括设计原则、代码整洁、虚拟环境、项目/文件夹设置、配置、日志记录、异常处理、代码质量检查、依赖管理、性能调优、单元测试、集成测试和容器化。 课程目标：我们将使用Xetra数据集，该数据集为德国交易所集团的电子交易平台，按分钟实时从德国交易所的交易系统提取，并存储在公开可用的AWS S3桶中。我们将创建的ETL管道将定期从AWS S3源桶提取Xetra数据集，经过转换后生成报告，并将转换后的数据加载到另一个AWS S3目标桶。该管道将以便于几乎任何可以处理容器化应用的生产环境进行部署的方式编写。我们设计的生产环境包含GitHub代码仓库、DockerHub镜像仓库、Kubernetes等执行平台，以及如Argo Workflows或Apache Airflow这样的编排工具。 课程预期：您将参与主要以实践为主的交互式课程，需在课程中编写并实现管道，同时在必要时提供理论课程。每课时都将提供相应的Python代码、整个项目在GitHub上的代码以及可在Docker Hub上使用的现成Docker镜像。每个理论课程还将提供可下载的PPT幻灯片和与每个主题、步骤相关的实用链接，以便您获取更多信息并深入学习。
This course will show each step to write an ETL pipeline in Python from scratch to production using the necessary tools such as Python 3.9, Jupyter Notebook, Git and Github, Visual Studio Code, Docker and Docker Hub and the Python packages Pandas, boto3, pyyaml, awscli, jupyter, pylint, moto, coverage and the memory-profiler.Two different approaches how to code in the Data Engineering field will be introduced and applied - functional and object oriented programming.Best practices in developing Python code will be introduced and applied: design principlesclean codingvirtual environmentsproject/folder setupconfigurationloggingexeption handlinglintingdependency managementperformance tuning with profilingunit testingintegration testingdockerizationWhat is the goal of this course?In the course we are going to use the Xetra dataset. Xetra stands for Exchange Electronic Trading and it is the trading platform of the Deutsche Börse Group. This dataset is derived near-time on a minute-by-minute basis from Deutsche Börse's trading system and saved in an AWS S3 bucket available to the public for free. The ETL Pipeline we are going to create will extract the Xetra dataset from the AWS S3 source bucket on a scheduled basis, create a report using transformations and load the transformed data to another AWS S3 target bucket. The pipeline will be written in a way that it can be deployed easily to almost any production environment that can handle containerized applications. The production environment we are going to write the ETL pipeline for consists of a GitHub Code repository, a DockerHub Image Repository, an execution platform such as Kubernetes and an Orchestration tool such as the container-native Kubernetes workflow engine Argo Workflows or Apache Airflow.So what can you expect in the course?You will receive primarily practical interactive lessons where you have to code and implement the pipeline and theory lessons when needed. Furthermore you will get the python code for each lesson in the course material, the whole project on GitHub and the ready to use docker image with the application code on Docker Hub.There will be power point slides for download for each theoretical lesson and useful links for each topic and step where you find more information and can even dive deeper.
