*We customize the course outline and content to your specific needs and relevant use cases.
Module 1: Databricks Lakehouse architecture and platform fundamentals
- Understanding the Databricks Data Intelligence Platform and Lakehouse architecture
- Navigating workspaces, notebooks, catalogs, compute, SQL warehouses, and jobs
- Understanding how data engineering, analytics, machine learning, and AI workloads share the platform
- Reviewing a typical enterprise data flow from source systems to governed data products
Module 2: Apache Spark fundamentals in Databricks
- Understanding Spark drivers, executors, distributed processing, and lazy evaluation
- Working with Spark DataFrames using Python and SQL
- Reading, filtering, joining, aggregating, and writing datasets
- Understanding transformations, actions, partitions, and common performance considerations
Module 3: Working productively with notebooks and Spark SQL
- Combining Python, SQL, and Spark operations within Databricks notebooks
- Creating reusable transformations and organizing notebook code
- Working with temporary views and SQL queries over Spark data
- Understanding when to use Spark DataFrames, Spark SQL, or Databricks SQL
Module 4: Delta Lake foundations
- Understanding Delta Lake tables, the transaction log, and ACID transactions
- Creating, reading, updating, and deleting Delta tables
- Working with schema enforcement, schema evolution, and table history
- Understanding how Delta Lake supports both batch and streaming workloads
Module 5: Data ingestion patterns
- Comparing ingestion from cloud storage, databases, applications, and streaming sources
- Loading files using Spark, SQL, and Databricks ingestion capabilities
- Introducing Auto Loader for scalable incremental file ingestion
- Selecting ingestion patterns based on volume, latency, source type, and operational requirements
Module 6: Incremental processing with Delta Lake
- Applying MERGE operations for inserts, updates, and upserts
- Using Change Data Feed to process row-level changes
- Handling changing schemas and incremental source data
- Designing idempotent processing so pipelines can be safely rerun
Module 7: Transformation and medallion architecture
- Structuring Bronze, Silver, and Gold layers for progressive refinement
- Cleaning, validating, joining, and enriching ingested data
- Separating raw ingestion from reusable business transformations
- Designing curated datasets for analytics, reporting, and downstream AI workloads
Module 8: Data quality and Delta table optimization
- Applying data quality checks during transformation
- Managing duplicates, missing values, invalid records, and late arriving data
- Understanding file size, data layout, and query performance considerations
- Introducing optimization techniques such as liquid clustering for appropriate workloads
Module 9: Building Lakeflow pipelines
- Understanding pipelines, flows, streaming tables, and materialized views
- Creating declarative transformations with SQL and Python
- Building dependencies between ingestion and transformation stages
- Using incremental processing for efficient batch and streaming pipelines
Module 10: Production pipeline patterns with Lakeflow
- Designing maintainable pipelines around clear source, transformation, and output boundaries
- Applying expectations and data quality controls within pipeline processing
- Managing pipeline configuration and environment-specific settings
- Monitoring pipeline execution and diagnosing common failures
Module 11: Orchestration with Lakeflow Jobs
- Creating jobs that coordinate notebooks, pipelines, SQL, and other tasks
- Defining task dependencies and multi-step workflows
- Configuring schedules, parameters, retries, and failure handling
- Choosing between pipeline-managed dependencies and explicit job orchestration
Module 12: Governance with Unity Catalog
- Understanding catalogs, schemas, tables, views, volumes, and the three-level namespace
- Configuring permissions and governed access to data and AI assets
- Using lineage, discovery, and auditing to understand how assets are used
- Organizing catalogs and schemas around environments, domains, and access boundaries
Module 13: Databricks SQL and analytics workloads
- Using SQL warehouses to query governed Lakehouse data
- Building reusable views and curated datasets for analytics
- Understanding performance considerations for interactive SQL workloads
- Providing governed datasets to BI and reporting tools
Module 14: Preparing data for machine learning and AI
- Creating reliable feature and training datasets from governed Lakehouse data
- Connecting data engineering outputs with Databricks machine learning and AI workloads
- Understanding governance of models, functions, and other AI assets through Unity Catalog
- Designing pipelines so analytics and AI teams can reuse trusted data products
Module 15: Developing production-ready Databricks workloads
- Moving from exploratory notebooks to structured production code and pipelines
- Separating development, testing, and production environments
- Managing configuration, parameters, dependencies, and reusable components
- Applying Git-based development and deployment practices to Databricks projects
Module 16: Monitoring, troubleshooting, and production readiness
- Monitoring jobs, pipelines, compute, and data quality
- Diagnosing failures, performance bottlenecks, and inefficient Spark processing
- Designing retry, recovery, and operational support approaches
- Building a practical production readiness checklist covering reliability, security, governance, performance, and maintainability