Course Overview
Updated description 3/12/2024
Generative AI tools such as ChatGPT, Claude, and Bard are making data engineering more accessible and more efficient. If you work with spreadsheets or business intelligence tools but aren't too familiar with Python or SQL, then generative AI can help you analyze data and build your own data pipelines and ETL/ELT processes.
Generative AI and LLMs will not replace data engineers or data analysts but those who know how to use these AI tools will be able to build more capable and reliable data pipelines faster. They will also have access to a tool that can help you develop your Python, SQL, and data modeling skills by providing a variety of examples of functional code and help with error messages and troubleshooting processes that do not work as expected.
Learn Data Engineering Techniques as Well as Data Engineering Tools
In this course, you will learn how to break down data engineering problems into a series of tasks that can be automated using Python, SQL, and command line scripts generated by a large language model (LLM).
Prompting an AI to "generate a data processing script to do X, Y, and Z" will probably not get you the results you expect. LLMs are powerful tools, but they are not oracles. As with any tool, we need to understand what the tool is capable of and how to use the capabilities to meet our needs.
This course shows you how to think through a data transformation and loading problems, incrementally building components of a solution. This course is organized into several topics that cover the fundamental skills needed to begin work in data engineering using GenAI, including:
Introduction to large language models, foundation models, and other AI topics related to data engineering. This course uses Claude AI from Anthropic, a large language model that is both well suited to data engineering code generation and free to use.
Working with CSV and JSON files
Data quality and data cleaning, including statistics and visualizations
Extraction transformation and load (ETL)/ extraction, load, and transform (ELT) processes
Relational and NoSQL databases
Data modeling using dimensional data model patterns
Working with JSON data in relational databases such as PostgreSQL
The course begins with the most basic of data engineering tasks: working with files. You will learn how to quickly filter, transform, and find problems in data sets made up of comma-separated value (CSV) and JSON files. You'll also see how we can create samples from large data sets to efficiently experiment with different solutions to our data engineer needs. You will learn how to generate code that uses command line utilities like awk, a text processing and data extraction tool, and jq, a tool for parsing, filtering, and transforming JSON data. If you are not familiar with tools like awk and jq, that is no problem. In this course, you will learn how to describe what you want in a solution so the LLM can choose an appropriate tool for the job.
Data quality is a primary concern in any data engineering project. Fortunately, with GenAI and a basic understanding of data quality checks, you can quickly generate scripts to check for common data quality problems and apply transformations to the data to correct for those problems. Statistics and visualizations are important tools for ensuring data quality. In this course, you will learn how to use basic statistics and visualizations to help with data quality and data exploration. And because generative AI is used to generate code, you can spend more time learning about statistics, visualizations, and how to apply them to your problem domain and less time trying to find syntax errors or debug a logic error in your code.
Databases are the foundation of many applications and data analysis platforms. You will learn about relational databases as well as NoSQL databases and when to use them. Databases are complicated systems that require that we describe how we want to structure our data. This process is known as data modeling. This course will introduce data modeling with a focus on dimensional modeling, a commonly used data model pattern in data analytics. You will also learn how to generate SQL code to implement dimensional models, load data into your database, and query and analyze data once it is loaded.
Now is a great time to become a data engineer because the demand for data engineering skills is high and we now have tools in place that allow us to focus on the problems we are solving while accelerating how quickly we can create scalable, reliable data pipelines.
FAQ
FAQ area empty
Course curriculum
Chương 1 Introduction 5 bài · 23 phút
- Introduction Học thử 4:10
- Course Overview Học thử 4:03
- Data Engineering, Data Analysis, and Data Science Học thử 6:05
-
Generative AI and Large Language Models (LLMs) 8:14
-
Key Concepts of GenAI Quiz
Chương 2 Working with CSV Files 11 bài · 52 phút
-
CSV and JSON Files 7:17
- Command Line Utilities for Working with Files Học thử 8:03
-
Filtering Rows of a CSV File 9:06
-
Combining Commands in a Shell Script 7:07
- CSV File Quiz Học thử
-
Hands-on Assignment: Filter Data in a CSV File using a bash script
- Sampling with Python Học thử 6:17
- Modifying a Shell Script Học thử 5:14
-
Scheduling Jobs with Cron 2:49
-
Cron Job Scheduling Example 6:30
-
Shell Script Quiz
Chương 3 Working with JSON FIles 5 bài · 17 phút
-
Working with JSON Files 6:56
-
Installing jq 1:30
-
Filtering JSON files with jq 4:03
-
Loading JSON into Python 4:25
-
JSON Quiz
Chương 4 Data Quality in Data Engineering 10 bài · 54 phút
-
Overiew of Data Quality 6:05
-
Sales Data 2:55
-
Missing Values in Data Files 8:31
-
Adding Documentation to Scripts 4:19
-
Range Checks 7:36
-
Working with Dates and Times 4:18
-
Checking Date and Time Format 8:44
-
Visualizations for Data Quality 11:46
-
Data Quality Quiz
-
Hands-On Assignment: Check passenger flight data with invalid airport codes
Chương 5 Working with Pandas in Python 4 bài · 17 phút
-
Working with Pandas 4:49
-
Statistics using Dataframes 8:06
-
Generating Synthetic Data 4:02
-
Hands-on Assignment: Load JSON into a Pandas DataFrame
Chương 6 JSON Schemas 3 bài · 18 phút
-
JSON Schemas for Data Validation 1 10:20
-
JSON Schemas for Data Validation 2 7:52
-
Pandas and JSON Schema Quiz
Chương 7 Working with Databases 8 bài · 43 phút
-
Relational Databases 7:44
-
NoSQL Databases 7:18
-
PostgreSQL 9:46
-
Installing PostgreSQL 5:30
-
Creating PostgreSQL Schemas 4:37
-
Creating Tables in PostgreSQL 7:58
-
Database Quiz
-
Hands-on Assignment: Create a table for bus schedules
Chương 8 Dimensional Modeling for Data Analysis 7 bài · 59 phút
-
Dimensional Modeling 15:38
-
Loading Sales Data into Staging Tables 12:38
-
Loading Dimension Data into Staging Tables 8:56
-
Creating Location Dimension 10:35
-
Creating Products Dimension 4:49
-
Create Date Dimension 6:26
-
Dimensional Model Quiz 1
Chương 9 Populating a Fact Table in a Dimensional Model 6 bài · 33 phút
-
Creating a Sales Fact Table 14:18
-
Preparing Data for Aggregation 1:30
-
Aggregating Staging Data 5:29
-
Loading Sales Fact Table 5:29
-
Generating SQL Queries for a Dimensional Model 5:45
-
Dimensional Modeling Quiz 2
Chương 10 JSON in PostgreSQL 5 bài · 28 phút
-
JSON in PostgreSQL 6:19
-
Creating a Table with a JSON Column in PostgreSQL 7:42
-
Loading JSON Data into PostgreSQL 11:59
-
Querying JSON Data in PostgreSQL 2:12
-
JSON in PostgreSQL Quiz
Chương 11 Next Steps Learning Data Engineering 5 bài · 21 phút
-
What to Learn Next? 1:32
-
Apache Airflow for Orchestration 4:48
-
Apache Spark for ETL/ELT and Analytics 5:27
-
Great Expectations for Data Quality Control 4:53
-
Dbt for Data Transformation with SQL 3:58
Chương 12 Course Wrap Up 1 bài · 3 phút
-
Conclusion 3:04
Requirment
-
Familiarity with working with data, such as with spreadsheets
-
Ability to read and follow Python and SQL code is helpful
Outcomes
-
Use large language models to create Python code to implement data pipelines
-
Use LLMs to solve data loading, data transformation, and data quality assessment challenges
-
Create databases and analytic data models using generative AI
-
Create Python, SQL, and Bash scripts to implement common data engineering tasks
0 Student
1 Course
0 review