*We customize the course outline and content to your specific needs and relevant use cases.
Module 1: Apache Flink architecture and stream processing fundamentals
- Understanding Flink as a distributed engine for bounded and unbounded data streams
- Reviewing JobManager, TaskManagers, tasks, subtasks, slots, and parallel execution
- Understanding sources, transformations, sinks, and operator chains
- Following a Flink job from submission through execution and data movement
Module 2: Working with Flink APIs
- Comparing Flink SQL, Table API, DataStream API, and PyFlink approaches
- Creating basic streaming transformations with the selected interface
- Working with filtering, mapping, projection, aggregation, and joins
- Choosing the appropriate abstraction level for different application requirements
Module 3: Sources, sinks, and connectors
- Connecting Flink to Kafka, files, databases, and other supported systems
- Understanding source partitions and parallel consumption
- Configuring sinks for streaming and analytical outputs
- Considering delivery guarantees and external system behaviour when selecting connectors
Module 4: Parallelism and data partitioning
- Understanding operator parallelism and task distribution
- Using key-based partitioning for stateful processing
- Recognising how upstream partitions affect downstream workload distribution
- Identifying skew and other common causes of uneven processing
Module 5: Event time and watermarks
- Distinguishing event time from processing time
- Assigning timestamps from incoming events
- Understanding watermarks and progress through event-time streams
- Handling out-of-order and delayed events in real-time pipelines
Module 6: Windowed stream processing
- Understanding tumbling, sliding, session, and global windows
- Applying event-time windows to keyed streams
- Using window triggers, allowed lateness, and late event handling
- Selecting window strategies based on analytical and operational requirements
Module 7: Keyed state and stateful processing
- Understanding keyed state and operator state
- Working with value, list, map, and reducing state patterns
- Using state to implement counters, session behaviour, enrichment, and event correlation
- Managing state lifetime with TTL and appropriate cleanup strategies
Module 8: Stateful processing patterns
- Combining keyed streams with timers and event-time logic
- Building stateful joins and correlation workflows
- Managing deduplication and sequence-dependent processing
- Designing state carefully to control size, latency, and recovery requirements
Module 9: Checkpointing fundamentals
- Understanding consistent snapshots and distributed checkpoints
- Configuring checkpoint intervals, timeouts, and storage
- Understanding how source positions and operator state are captured together
- Distinguishing checkpoints from savepoints
Module 10: Fault tolerance and recovery
- Understanding restart strategies and failure recovery
- Recovering application state after task or infrastructure failure
- Working with exactly-once and at-least-once processing guarantees
- Understanding how source and sink capabilities influence end-to-end guarantees
Module 11: Savepoints and application evolution
- Creating and restoring applications from savepoints
- Using savepoints during planned upgrades and deployment changes
- Understanding operator identity and state mapping
- Planning state-compatible application changes
Module 12: Reliable streaming pipeline design
- Combining sources, transformations, state, and sinks into maintainable pipelines
- Designing idempotent and recoverable processing where external systems are involved
- Managing backpressure and avoiding uncontrolled buffering
- Selecting checkpoint, state, and recovery settings based on workload characteristics
Module 13: Flink deployment architectures
- Understanding session, application, and deployment models
- Running Flink in standalone, containerised, and Kubernetes environments
- Understanding native Kubernetes deployment concepts
- Introducing the Flink Kubernetes Operator for declarative application management
Module 14: Monitoring and troubleshooting Flink jobs
- Using the Flink Web UI to inspect jobs, operators, checkpoints, and task metrics
- Identifying failed checkpoints, backpressure, skew, and slow operators
- Reviewing logs and metrics during application troubleshooting
- Following data flow through the execution graph to locate bottlenecks
Module 15: Performance and resource optimisation
- Configuring parallelism and task resources for workload requirements
- Understanding network buffers, memory, state backends, and checkpoint overhead
- Reducing unnecessary serialization, shuffles, and expensive state operations
- Tuning applications based on measured throughput, latency, and backpressure
Module 16: Production operations and readiness
- Managing configuration, application versions, secrets, and environment-specific settings
- Planning upgrades, savepoints, restart behaviour, and recovery procedures
- Establishing monitoring and alerting around jobs, checkpoints, lag, and resource usage
- Building a practical production readiness checklist covering state, fault tolerance, deployment, monitoring, scaling, and recovery