Catalogue
/
Data Engineering
/
Apache Flink for Stateful Stream Processing and Real-Time Data Pipelines

Apache Flink for Stateful Stream Processing and Real-Time Data Pipelines

A practical four day technical course for participants who are new to Apache Flink and want to build stateful real-time data processing applications and pipelines. The agenda progresses from Flink architecture and stream processing fundamentals through event time, windows, state, connectors, checkpointing, fault tolerance, deployment, and production operations. The course focuses on practical patterns for building reliable and scalable streaming applications rather than covering every available API or connector.

What will you learn?

You will learn how Apache Flink processes bounded and unbounded data streams and how its runtime coordinates parallel, stateful computations. You will work with event time, watermarks, windows, keyed state, checkpoints, connectors, and deployment concepts using the Flink interface most relevant to your environment. You will also learn how to design fault-tolerant pipelines, manage application state, monitor running jobs, and prepare Flink workloads for production environments.

  • Build real-time data processing pipelines using Flink SQL, Table API, DataStream API, or PyFlink
  • Apply event time, watermarks, windows, and stateful processing to streaming workloads
  • Configure checkpointing and recovery for fault-tolerant applications
  • Deploy, monitor, troubleshoot, and operate Flink applications in production environments

Requirements:

  • Technical background in software development, data engineering, analytics, or a related field
  • Comfortable working with at least one primary Flink interface: SQL / Table API, Java or Scala/Kotlin, or Python with PyFlink
  • General understanding of distributed systems and data processing concepts
  • Familiarity with streaming platforms such as Kafka is helpful but not required

Course Outline*:

*We customize the course outline and content to your specific needs and relevant use cases.

Module 1: Apache Flink architecture and stream processing fundamentals

  • Understanding Flink as a distributed engine for bounded and unbounded data streams
  • Reviewing JobManager, TaskManagers, tasks, subtasks, slots, and parallel execution
  • Understanding sources, transformations, sinks, and operator chains
  • Following a Flink job from submission through execution and data movement

Module 2: Working with Flink APIs

  • Comparing Flink SQL, Table API, DataStream API, and PyFlink approaches
  • Creating basic streaming transformations with the selected interface
  • Working with filtering, mapping, projection, aggregation, and joins
  • Choosing the appropriate abstraction level for different application requirements

Module 3: Sources, sinks, and connectors

  • Connecting Flink to Kafka, files, databases, and other supported systems
  • Understanding source partitions and parallel consumption
  • Configuring sinks for streaming and analytical outputs
  • Considering delivery guarantees and external system behaviour when selecting connectors

Module 4: Parallelism and data partitioning

  • Understanding operator parallelism and task distribution
  • Using key-based partitioning for stateful processing
  • Recognising how upstream partitions affect downstream workload distribution
  • Identifying skew and other common causes of uneven processing

Module 5: Event time and watermarks

  • Distinguishing event time from processing time
  • Assigning timestamps from incoming events
  • Understanding watermarks and progress through event-time streams
  • Handling out-of-order and delayed events in real-time pipelines

Module 6: Windowed stream processing

  • Understanding tumbling, sliding, session, and global windows
  • Applying event-time windows to keyed streams
  • Using window triggers, allowed lateness, and late event handling
  • Selecting window strategies based on analytical and operational requirements

Module 7: Keyed state and stateful processing

  • Understanding keyed state and operator state
  • Working with value, list, map, and reducing state patterns
  • Using state to implement counters, session behaviour, enrichment, and event correlation
  • Managing state lifetime with TTL and appropriate cleanup strategies

Module 8: Stateful processing patterns

  • Combining keyed streams with timers and event-time logic
  • Building stateful joins and correlation workflows
  • Managing deduplication and sequence-dependent processing
  • Designing state carefully to control size, latency, and recovery requirements

Module 9: Checkpointing fundamentals

  • Understanding consistent snapshots and distributed checkpoints
  • Configuring checkpoint intervals, timeouts, and storage
  • Understanding how source positions and operator state are captured together
  • Distinguishing checkpoints from savepoints

Module 10: Fault tolerance and recovery

  • Understanding restart strategies and failure recovery
  • Recovering application state after task or infrastructure failure
  • Working with exactly-once and at-least-once processing guarantees
  • Understanding how source and sink capabilities influence end-to-end guarantees

Module 11: Savepoints and application evolution

  • Creating and restoring applications from savepoints
  • Using savepoints during planned upgrades and deployment changes
  • Understanding operator identity and state mapping
  • Planning state-compatible application changes

Module 12: Reliable streaming pipeline design

  • Combining sources, transformations, state, and sinks into maintainable pipelines
  • Designing idempotent and recoverable processing where external systems are involved
  • Managing backpressure and avoiding uncontrolled buffering
  • Selecting checkpoint, state, and recovery settings based on workload characteristics

Module 13: Flink deployment architectures

  • Understanding session, application, and deployment models
  • Running Flink in standalone, containerised, and Kubernetes environments
  • Understanding native Kubernetes deployment concepts
  • Introducing the Flink Kubernetes Operator for declarative application management

Module 14: Monitoring and troubleshooting Flink jobs

  • Using the Flink Web UI to inspect jobs, operators, checkpoints, and task metrics
  • Identifying failed checkpoints, backpressure, skew, and slow operators
  • Reviewing logs and metrics during application troubleshooting
  • Following data flow through the execution graph to locate bottlenecks

Module 15: Performance and resource optimisation

  • Configuring parallelism and task resources for workload requirements
  • Understanding network buffers, memory, state backends, and checkpoint overhead
  • Reducing unnecessary serialization, shuffles, and expensive state operations
  • Tuning applications based on measured throughput, latency, and backpressure

Module 16: Production operations and readiness

  • Managing configuration, application versions, secrets, and environment-specific settings
  • Planning upgrades, savepoints, restart behaviour, and recovery procedures
  • Establishing monitoring and alerting around jobs, checkpoints, lag, and resource usage
  • Building a practical production readiness checklist covering state, fault tolerance, deployment, monitoring, scaling, and recovery

Hands-on learning with expert instructors at your location for organizations.

5,622€*
Graph Icon - Education X Webflow Template
Level:
intermediate
Clock Icon - Education X Webflow Template
Duration:
28
Hours (days:
4
)
Camera Icon - Education X Webflow Template
Training customized to your needs
Star Icon - Education X Webflow Template
Immersive hands-on experience in a dedicated setting
*Price can range depending on number of participants, change of outline, location etc.

Master new skills guided by experienced instructors from anywhere.

3,987€*
Graph Icon - Education X Webflow Template
Level:
intermediate
Clock Icon - Education X Webflow Template
Duration:
28
Hours (days:
4
)
Camera Icon - Education X Webflow Template
Training customized to your needs
Star Icon - Education X Webflow Template
Reduced training costs
*Price can range depending on number of participants, change of outline, location etc.

Upcoming Sessions

4-7 Jan 2027
Paris
11-14 Jan 2027
Vienna
15-18 Feb 2027
Zurich
22-25 Feb 2027
Brussels
22-25 Mar 2027
Frankfurt
19-22 Apr 2027
Vienna
5-8 Jul 2027
Stockholm

Can't find a suitable date? Get in touch and we'll arrange one that works for you.

We use cookies to improve site navigation, analyse how the site is used, and support our marketing. You can accept all cookies, reject non-essential ones, or choose individual categories. Read our Cookie Notice