Udemy - Databricks Data Engineering - Build Pipelines With Lakefl...

Databricks Data Engineering: Build Pipelines With Lakeflow

https://WebToolTip.com

MP4 | Video: h264, 1920x1080 | Audio: AAC, 44.1 KHz
Language: English | Size: 2.99 GB | Duration: 4h 11m

Build production-grade, dynamic data pipelines with Lakeflow Connect, Lakeflow Jobs, Auto Loader, and Delta Lake.

What you'll learn
Course Overview & Learning Path
What Databricks Lakeflow Is and Why It Matters
Understanding Lakeflow Connect and Lakeflow Jobs
Building Production-Grade Data Ingestion Pipelines
From Raw Data to Delta Tables
Working with Local Files in Databricks
Creating Tables from Local Files
Understanding Data Ingestion in Databricks
Building and Managing Data Connections
Standard Connectors vs Managed Connectors
Understanding Modern Data Pipeline Orchestration
Introduction to Lakeflow Jobs ·
How Jobs, Tasks, and Dependencies Work
Building Your First Lakeflow Job
Preparing Notebook and SQL File Tasks
Creating a Silver Transformation Notebook
Building Data Quality Check SQL Tasks
Creating Gold Aggregation Outputs
Connecting Tasks with Dependencies
Running an End-to-End Lakeflow Job
Validating Silver, Quality Check, and Gold Outputs
Understanding Workflow Automation in Databricks
Conditional Workflows in Lakeflow Jobs
Building If/Else Logic in Pipelines
Creating Alternative Execution Paths
Using Job Metadata in Conditions
Running and Validating Conditional Pipelines
Building Dynamic Pipelines with For Each Loops
Creating Parameterized Notebooks
Passing Loop Inputs into Notebook Tasks
Running Multiple Iterations Automatically
Using Concurrency in Loop-Based Pipelines
Monitoring and Debugging Databricks Jobs
Understanding Successful, Skipped, and Failed Tasks
Debugging Task Outputs and Execution Details
Managing Quota Limit Issues in Databricks Free Edition
Using SQL Rows Output in Dynamic Pipelines
Generating Loop Inputs from SQL Query Results
Connecting SQL Output Rows to For Each Loops
Building Data-Aware Dynamic Pipelines
Testing Pipelines with New Incoming Data
Driving Pipelines with First Row Logic
Selecting the Most Frequent Order Status
Passing First Row Values Between Tasks
Creating Focused Gold Analysis Tables
Data-Driven Pipelines with Dynamic Parameter Passing
Generating Dynamic Workloads from Silver Tables
Using Task Values to Share Data Between Tasks
Building Warehouse and Shipping-Based Pipeline Logic
Creating If/Else Controls for Empty Workloads
Passing Multiple Parameters into Notebook Tasks
Building Delivery Summary Outputs with Dynamic Parameters
Using SQL Tables as Dynamic Loop Inputs
Creating SQL Lookup Tables for Pipeline Control
Reading Lookup Tables as SQL Row Outputs
Creating Target Gold Tables Before Loop Execution
Running SQL Table-Driven For Each Pipelines
Avoiding Duplicate Results in Repeated Pipeline Runs
Building Production-Ready Data-Driven Workflow Patterns
Designing Scalable Lakeflow Pipeline Architectures
End-to-End Pipeline Automation with Databricks Lakeflow

Requirements
A basic understanding of Databricks fundamentals, including notebooks, clusters, and Delta tables, is recommended
A working computer (Windows, Mac, or Linux)
A stable internet connection to access Databricks
Access to Databricks Free Edition or any Databricks workspace
Basic understanding of SQL
Basic understanding of Python is helpful but not mandatory
Basic familiarity with data tables, columns, and simple queries
Interest in data engineering and real-world data pipelines
Curiosity about workflow orchestration and pipeline automation
Motivation to build dynamic, scalable, and production-ready data workflows
No prior Lakeflow experience required
No advanced Spark knowledge required
Just you, your keyboard, and your passion for becoming a modern data engineer!