*We customize the course outline and content to your specific needs and relevant use cases.
Module 1: Apache Iceberg and the open lakehouse model
- Understanding the role of open table formats between compute engines and object storage
- Comparing traditional data lake file management with Iceberg table management
- Understanding data files, manifests, manifest lists, metadata files, and snapshots
- Positioning Iceberg within a complete lakehouse architecture
Module 2: Catalogs and lakehouse infrastructure
- Understanding the role of Iceberg catalogs in table discovery and metadata management
- Working with REST, Hive, Hadoop, or platform-specific catalog approaches
- Connecting Spark or Flink to an Iceberg catalog and warehouse location
- Designing clear boundaries between object storage, catalog, compute, and query services
Module 3: Creating and querying Iceberg tables
- Creating namespaces and tables using SQL or the selected compute API
- Defining schemas, properties, partition specifications, and storage locations
- Reading Iceberg tables through Spark or Flink
- Inspecting table metadata, snapshots, files, and partitions
Module 4: Partitioning and table layout
- Understanding hidden partitioning and partition transforms
- Designing partition strategies for time-based and high-volume analytical data
- Avoiding excessive partitions and small-file patterns
- Understanding how table layout influences pruning and query performance
Module 5: Batch ingestion and write operations
- Appending data from files, transformations, and upstream datasets
- Working with inserts, overwrites, and supported row-level operations
- Understanding atomic commits and snapshot creation
- Designing reliable ingestion workflows that can be safely retried
Module 6: Streaming ingestion with Spark or Flink
- Writing continuous data streams into Iceberg tables
- Understanding checkpoints, commits, and delivery guarantees
- Designing appropriate commit frequency for streaming workloads
- Managing the small-file and metadata effects of frequent streaming writes
Module 7: Schema and partition evolution
- Adding, removing, renaming, and changing compatible table fields
- Evolving partition specifications without rewriting historical data
- Understanding how field IDs protect schema correctness
- Planning compatible changes for long-lived production tables
Module 8: Updates, deletes, and changing data
- Implementing upsert and change-processing patterns with the selected compute engine
- Understanding copy-on-write and merge-on-read concepts
- Applying MERGE, UPDATE, and DELETE where supported
- Choosing write patterns based on freshness, query performance, and maintenance requirements
Module 9: Snapshots, time travel, and table history
- Understanding how Iceberg snapshots represent atomic table versions
- Querying previous table states using time travel
- Working with rollback and recovery scenarios
- Understanding branches and tags for controlled data workflows
Module 10: Table maintenance and optimisation
- Compacting small data files to improve scan efficiency
- Rewriting manifests to reduce metadata overhead
- Expiring old snapshots according to retention requirements
- Removing orphaned files safely and understanding maintenance dependencies
Module 11: Building the complete lakehouse data flow
- Combining ingestion, object storage, catalogs, Iceberg tables, and compute engines
- Designing batch and streaming paths into shared analytical tables
- Supporting downstream SQL, analytics, and data science workloads
- Defining clear responsibilities for ingestion, transformation, maintenance, and consumption
Module 12: Production operations and lakehouse readiness
- Monitoring file counts, snapshot growth, commit behaviour, and table performance
- Managing concurrency, failed writes, catalog availability, and operational recovery
- Establishing retention, maintenance, schema evolution, and deployment procedures
- Building a practical production readiness checklist covering storage, catalogs, ingestion, evolution, maintenance, security, and recovery