|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/generative-ai-for-data-engineering/
课程评论：没有评论
课程名称：使用生成式人工智能的数据工程入门 课程概述： 在这门课程中，您将学习如何利用生成式人工智能工具（如ChatGPT、Claude和Bard）来简化和提高数据工程的效率。无论您是经常使用电子表格或商业智能工具，但对Python或SQL不太熟悉，生成式AI都能帮助您分析数据并构建数据管道及ETL/ELT流程。数据工程师或数据分析师的工作不会被这些AI工具替代，但掌握这些工具的人能够更快地构建更强大、更可靠的数据管道。 课程内容包括： 1. 大型语言模型和基础模型的介绍，以及与数据工程相关的其他AI主题。 2. 使用CSV和JSON文件。 3. 数据质量与数据清洗，包括统计和可视化。 4. 提取、转化和加载（ETL）以及提取、加载和转化（ELT）过程。 5. 关系型与NoSQL数据库。 6. 使用维度数据模型模式进行数据建模。 7. 在关系数据库（如PostgreSQL）中处理JSON数据。 课程将从基本的数据工程任务开始，教授如何处理CSV和JSON文件。您将学习生成使用命令行工具（如awk和jq）的代码，了解这些工具的适用场景。数据质量是每个数据工程项目的核心，使用生成式AI和基本的数据质量检查，您可以快速生成脚本以识别和纠正常见数据质量问题，学习统计和可视化工具的应用，从而减少在调试代码时所花费的时间。 此外，课程还将讲解数据库的结构和数据建模，特别是数据分析中常用的维度建模，您将学习如何生成SQL代码来实施维度模型、将数据加载到数据库中，并查询和分析已加载的数据。随着数据工程技能需求的增加，现在正是成为数据工程师的好时机，生成式AI工具的使用能够加速创建可扩展、可靠的数据管道的进程。
Updated description 3/12/2024Generative AI tools such as ChatGPT, Claude, and Bard are making data engineering more accessible and more efficient. If you work with spreadsheets or business intelligence tools but aren't too familiar with Python or SQL, then generative AI can help you analyze data and build your own data pipelines and ETL/ELT processes.Generative AI and LLMs will not replace data engineers or data analysts but those who know how to use these AI tools will be able to build more capable and reliable data pipelines faster. They will also have access to a tool that can help you develop your Python, SQL, and data modeling skills by providing a variety of examples of functional code and help with error messages and troubleshooting processes that do not work as expected.Learn Data Engineering Techniques as Well as Data Engineering ToolsIn this course, you will learn how to break down data engineering problems into a series of tasks that can be automated using Python, SQL, and command line scripts generated by a large language model (LLM). Prompting an AI to "generate a data processing script to do X, Y, and Z" will probably not get you the results you expect. LLMs are powerful tools, but they are not oracles. As with any tool, we need to understand what the tool is capable of and how to use the capabilities to meet our needs. This course shows you how to think through a data transformation and loading problems, incrementally building components of a solution. This course is organized into several topics that cover the fundamental skills needed to begin work in data engineering using GenAI, including:Introduction to large language models, foundation models, and other AI topics related to data engineering. This course uses Claude AI from Anthropic, a large language model that is both well suited to data engineering code generation and free to use.Working with CSV and JSON filesData quality and data cleaning, including statistics and visualizationsExtraction transformation and load (ETL)/ extraction, load, and transform (ELT) processesRelational and NoSQL databasesData modeling using dimensional data model patternsWorking with JSON data in relational databases such as PostgreSQLThe course begins with the most basic of data engineering tasks: working with files. You will learn how to quickly filter, transform, and find problems in data sets made up of comma-separated value (CSV) and JSON files. You'll also see how we can create samples from large data sets to efficiently experiment with different solutions to our data engineer needs. You will learn how to generate code that uses command line utilities like awk, a text processing and data extraction tool, and jq, a tool for parsing, filtering, and transforming JSON data. If you are not familiar with tools like awk and jq, that is no problem. In this course, you will learn how to describe what you want in a solution so the LLM can choose an appropriate tool for the job.Data quality is a primary concern in any data engineering project. Fortunately, with GenAI and a basic understanding of data quality checks, you can quickly generate scripts to check for common data quality problems and apply transformations to the data to correct for those problems. Statistics and visualizations are important tools for ensuring data quality. In this course, you will learn how to use basic statistics and visualizations to help with data quality and data exploration. And because generative AI is used to generate code, you can spend more time learning about statistics, visualizations, and how to apply them to your problem domain and less time trying to find syntax errors or debug a logic error in your code.Databases are the foundation of many applications and data analysis platforms. You will learn about relational databases as well as NoSQL databases and when to use them. Databases are complicated systems that require that we describe how we want to structure our data. This process is known as data modeling. This course will introduce data modeling with a focus on dimensional modeling, a commonly used data model pattern in data analytics. You will also learn how to generate SQL code to implement dimensional models, load data into your database, and query and analyze data once it is loaded.Now is a great time to become a data engineer because the demand for data engineering skills is high and we now have tools in place that allow us to focus on the problems we are solving while accelerating how quickly we can create scalable, reliable data pipelines.
