Catalogue
/
Data Engineering
/
Databricks Lakehouse for Enterprise Data Engineering, Analytics, and AI

Databricks Lakehouse for Enterprise Data Engineering, Analytics, and AI

A practical four day technical course for participants who are new to Databricks and want to become productive quickly with enterprise data engineering, analytics, and AI workloads. The agenda builds from the Databricks Lakehouse architecture and Apache Spark fundamentals through Delta Lake, ingestion, transformation, Lakeflow pipelines, orchestration, Unity Catalog, analytics, and production deployment. The course focuses on the patterns and platform capabilities needed to build reliable, governed, and maintainable data pipelines rather than covering the platform as a collection of isolated features.

What will you learn?

You will learn how to use Databricks to ingest, transform, govern, orchestrate, and serve enterprise data for analytics and AI workloads. You will work with Apache Spark, Delta Lake, Lakeflow, Unity Catalog, SQL, and production pipeline concepts while understanding how the main Databricks components fit together. You will also learn how to move from notebook-based development toward reliable, automated, governed, and monitored production workloads.

  • Build data processing workloads with Apache Spark and Delta Lake
  • Implement batch and incremental ingestion and transformation pipelines
  • Orchestrate and govern data workloads with Lakeflow and Unity Catalog
  • Prepare reliable production data pipelines for analytics and AI use cases

Requirements:

  • Technical background in software development, data engineering, analytics, databases, or a related field
  • Basic SQL knowledge
  • Basic programming experience, preferably with Python
  • Previous Apache Spark or Databricks experience is not required

Course Outline*:

*We customize the course outline and content to your specific needs and relevant use cases.

Module 1: Databricks Lakehouse architecture and platform fundamentals

  • Understanding the Databricks Data Intelligence Platform and Lakehouse architecture
  • Navigating workspaces, notebooks, catalogs, compute, SQL warehouses, and jobs
  • Understanding how data engineering, analytics, machine learning, and AI workloads share the platform
  • Reviewing a typical enterprise data flow from source systems to governed data products

Module 2: Apache Spark fundamentals in Databricks

  • Understanding Spark drivers, executors, distributed processing, and lazy evaluation
  • Working with Spark DataFrames using Python and SQL
  • Reading, filtering, joining, aggregating, and writing datasets
  • Understanding transformations, actions, partitions, and common performance considerations

Module 3: Working productively with notebooks and Spark SQL

  • Combining Python, SQL, and Spark operations within Databricks notebooks
  • Creating reusable transformations and organizing notebook code
  • Working with temporary views and SQL queries over Spark data
  • Understanding when to use Spark DataFrames, Spark SQL, or Databricks SQL

Module 4: Delta Lake foundations

  • Understanding Delta Lake tables, the transaction log, and ACID transactions
  • Creating, reading, updating, and deleting Delta tables
  • Working with schema enforcement, schema evolution, and table history
  • Understanding how Delta Lake supports both batch and streaming workloads

Module 5: Data ingestion patterns

  • Comparing ingestion from cloud storage, databases, applications, and streaming sources
  • Loading files using Spark, SQL, and Databricks ingestion capabilities
  • Introducing Auto Loader for scalable incremental file ingestion
  • Selecting ingestion patterns based on volume, latency, source type, and operational requirements

Module 6: Incremental processing with Delta Lake

  • Applying MERGE operations for inserts, updates, and upserts
  • Using Change Data Feed to process row-level changes
  • Handling changing schemas and incremental source data
  • Designing idempotent processing so pipelines can be safely rerun

Module 7: Transformation and medallion architecture

  • Structuring Bronze, Silver, and Gold layers for progressive refinement
  • Cleaning, validating, joining, and enriching ingested data
  • Separating raw ingestion from reusable business transformations
  • Designing curated datasets for analytics, reporting, and downstream AI workloads

Module 8: Data quality and Delta table optimization

  • Applying data quality checks during transformation
  • Managing duplicates, missing values, invalid records, and late arriving data
  • Understanding file size, data layout, and query performance considerations
  • Introducing optimization techniques such as liquid clustering for appropriate workloads

Module 9: Building Lakeflow pipelines

  • Understanding pipelines, flows, streaming tables, and materialized views
  • Creating declarative transformations with SQL and Python
  • Building dependencies between ingestion and transformation stages
  • Using incremental processing for efficient batch and streaming pipelines

Module 10: Production pipeline patterns with Lakeflow

  • Designing maintainable pipelines around clear source, transformation, and output boundaries
  • Applying expectations and data quality controls within pipeline processing
  • Managing pipeline configuration and environment-specific settings
  • Monitoring pipeline execution and diagnosing common failures

Module 11: Orchestration with Lakeflow Jobs

  • Creating jobs that coordinate notebooks, pipelines, SQL, and other tasks
  • Defining task dependencies and multi-step workflows
  • Configuring schedules, parameters, retries, and failure handling
  • Choosing between pipeline-managed dependencies and explicit job orchestration

Module 12: Governance with Unity Catalog

  • Understanding catalogs, schemas, tables, views, volumes, and the three-level namespace
  • Configuring permissions and governed access to data and AI assets
  • Using lineage, discovery, and auditing to understand how assets are used
  • Organizing catalogs and schemas around environments, domains, and access boundaries

Module 13: Databricks SQL and analytics workloads

  • Using SQL warehouses to query governed Lakehouse data
  • Building reusable views and curated datasets for analytics
  • Understanding performance considerations for interactive SQL workloads
  • Providing governed datasets to BI and reporting tools

Module 14: Preparing data for machine learning and AI

  • Creating reliable feature and training datasets from governed Lakehouse data
  • Connecting data engineering outputs with Databricks machine learning and AI workloads
  • Understanding governance of models, functions, and other AI assets through Unity Catalog
  • Designing pipelines so analytics and AI teams can reuse trusted data products

Module 15: Developing production-ready Databricks workloads

  • Moving from exploratory notebooks to structured production code and pipelines
  • Separating development, testing, and production environments
  • Managing configuration, parameters, dependencies, and reusable components
  • Applying Git-based development and deployment practices to Databricks projects

Module 16: Monitoring, troubleshooting, and production readiness

  • Monitoring jobs, pipelines, compute, and data quality
  • Diagnosing failures, performance bottlenecks, and inefficient Spark processing
  • Designing retry, recovery, and operational support approaches
  • Building a practical production readiness checklist covering reliability, security, governance, performance, and maintainability

Hands-on learning with expert instructors at your location for organizations.

7,722€*
Graph Icon - Education X Webflow Template
Level:
advanced
Clock Icon - Education X Webflow Template
Duration:
28
Hours (days:
4
)
Camera Icon - Education X Webflow Template
Training customized to your needs
Star Icon - Education X Webflow Template
Immersive hands-on experience in a dedicated setting
*Price can range depending on number of participants, change of outline, location etc.

Master new skills guided by experienced instructors from anywhere.

6,087€*
Graph Icon - Education X Webflow Template
Level:
advanced
Clock Icon - Education X Webflow Template
Duration:
28
Hours (days:
4
)
Camera Icon - Education X Webflow Template
Training customized to your needs
Star Icon - Education X Webflow Template
Reduced training costs
*Price can range depending on number of participants, change of outline, location etc.

Upcoming Sessions

8-11 Feb 2027
Munich
22-25 Feb 2027
Madrid
29 Mar - 1 Apr 2027
Madrid
12-15 Apr 2027
Paris
10-13 May 2027
London
24-27 May 2027
Madrid
21-24 Jun 2027
Barcelona
28 Jun - 1 Jul 2027
Stockholm

Can't find a suitable date? Get in touch and we'll arrange one that works for you.

We use cookies to improve site navigation, analyse how the site is used, and support our marketing. You can accept all cookies, reject non-essential ones, or choose individual categories. Read our Cookie Notice