Catalogue
/
Data Engineering
/
Apache Iceberg for Open Lakehouse Architecture and Large-Scale Data Platforms

Apache Iceberg for Open Lakehouse Architecture and Large-Scale Data Platforms

A practical three day technical course for participants who are new to Apache Iceberg and want to build reliable lakehouse architectures on open object storage. The agenda progresses from Iceberg table architecture, catalogs, schemas, and partitioning through batch and streaming ingestion, table evolution, snapshots, maintenance, and production design. The course uses Spark or Flink as the primary compute engine and focuses on building a complete, maintainable lakehouse workflow rather than covering every supported engine in depth.

What will you learn?

You will learn how Apache Iceberg adds reliable table management to large-scale data stored in formats such as Parquet on cloud or distributed object storage. You will create and manage Iceberg tables, configure catalogs, implement batch and streaming ingestion, evolve schemas and partitioning, work with snapshots, and maintain table performance over time. You will also learn how to combine storage, catalogs, compute, ingestion, maintenance, and query workloads into a practical lakehouse architecture.

  • Design an Apache Iceberg lakehouse using object storage, catalogs, and Spark or Flink
  • Create, read, update, and evolve large analytical tables safely
  • Implement batch and streaming ingestion while managing snapshots and table state
  • Operate Iceberg tables through compaction, metadata maintenance, monitoring, and lifecycle management

Requirements:

  • Technical background in data engineering, data platforms, analytics engineering, or a related field
  • Solid understanding of data engineering concepts and distributed data processing
  • Familiarity with cloud or distributed object storage and columnar file formats such as Parquet
  • Working knowledge of Spark or Flink and general SQL concepts

Course Outline*:

*We customize the course outline and content to your specific needs and relevant use cases.

Module 1: Apache Iceberg and the open lakehouse model

  • Understanding the role of open table formats between compute engines and object storage
  • Comparing traditional data lake file management with Iceberg table management
  • Understanding data files, manifests, manifest lists, metadata files, and snapshots
  • Positioning Iceberg within a complete lakehouse architecture

Module 2: Catalogs and lakehouse infrastructure

  • Understanding the role of Iceberg catalogs in table discovery and metadata management
  • Working with REST, Hive, Hadoop, or platform-specific catalog approaches
  • Connecting Spark or Flink to an Iceberg catalog and warehouse location
  • Designing clear boundaries between object storage, catalog, compute, and query services

Module 3: Creating and querying Iceberg tables

  • Creating namespaces and tables using SQL or the selected compute API
  • Defining schemas, properties, partition specifications, and storage locations
  • Reading Iceberg tables through Spark or Flink
  • Inspecting table metadata, snapshots, files, and partitions

Module 4: Partitioning and table layout

  • Understanding hidden partitioning and partition transforms
  • Designing partition strategies for time-based and high-volume analytical data
  • Avoiding excessive partitions and small-file patterns
  • Understanding how table layout influences pruning and query performance

Module 5: Batch ingestion and write operations

  • Appending data from files, transformations, and upstream datasets
  • Working with inserts, overwrites, and supported row-level operations
  • Understanding atomic commits and snapshot creation
  • Designing reliable ingestion workflows that can be safely retried

Module 6: Streaming ingestion with Spark or Flink

  • Writing continuous data streams into Iceberg tables
  • Understanding checkpoints, commits, and delivery guarantees
  • Designing appropriate commit frequency for streaming workloads
  • Managing the small-file and metadata effects of frequent streaming writes

Module 7: Schema and partition evolution

  • Adding, removing, renaming, and changing compatible table fields
  • Evolving partition specifications without rewriting historical data
  • Understanding how field IDs protect schema correctness
  • Planning compatible changes for long-lived production tables

Module 8: Updates, deletes, and changing data

  • Implementing upsert and change-processing patterns with the selected compute engine
  • Understanding copy-on-write and merge-on-read concepts
  • Applying MERGE, UPDATE, and DELETE where supported
  • Choosing write patterns based on freshness, query performance, and maintenance requirements

Module 9: Snapshots, time travel, and table history

  • Understanding how Iceberg snapshots represent atomic table versions
  • Querying previous table states using time travel
  • Working with rollback and recovery scenarios
  • Understanding branches and tags for controlled data workflows

Module 10: Table maintenance and optimisation

  • Compacting small data files to improve scan efficiency
  • Rewriting manifests to reduce metadata overhead
  • Expiring old snapshots according to retention requirements
  • Removing orphaned files safely and understanding maintenance dependencies

Module 11: Building the complete lakehouse data flow

  • Combining ingestion, object storage, catalogs, Iceberg tables, and compute engines
  • Designing batch and streaming paths into shared analytical tables
  • Supporting downstream SQL, analytics, and data science workloads
  • Defining clear responsibilities for ingestion, transformation, maintenance, and consumption

Module 12: Production operations and lakehouse readiness

  • Monitoring file counts, snapshot growth, commit behaviour, and table performance
  • Managing concurrency, failed writes, catalog availability, and operational recovery
  • Establishing retention, maintenance, schema evolution, and deployment procedures
  • Building a practical production readiness checklist covering storage, catalogs, ingestion, evolution, maintenance, security, and recovery

Hands-on learning with expert instructors at your location for organizations.

4,347€*
Graph Icon - Education X Webflow Template
Level:
intermediate
Clock Icon - Education X Webflow Template
Duration:
21
Hours (days:
3
)
Camera Icon - Education X Webflow Template
Training customized to your needs
Star Icon - Education X Webflow Template
Immersive hands-on experience in a dedicated setting
*Price can range depending on number of participants, change of outline, location etc.

Master new skills guided by experienced instructors from anywhere.

3,012€*
Graph Icon - Education X Webflow Template
Level:
intermediate
Clock Icon - Education X Webflow Template
Duration:
21
Hours (days:
3
)
Camera Icon - Education X Webflow Template
Training customized to your needs
Star Icon - Education X Webflow Template
Reduced training costs
*Price can range depending on number of participants, change of outline, location etc.

Upcoming Sessions

7-9 Jan 2027
London
4-6 Feb 2027
Paris
11-13 Feb 2027
Barcelona
18-20 Mar 2027
Barcelona
7-9 Apr 2027
Milan
13-15 May 2027
Dublin
16-18 Jun 2027
Vienna

Can't find a suitable date? Get in touch and we'll arrange one that works for you.

We use cookies to improve site navigation, analyse how the site is used, and support our marketing. You can accept all cookies, reject non-essential ones, or choose individual categories. Read our Cookie Notice