Data & Databases Pipelines & Streaming

Flink 2.0's State Compatibility Break Makes the 1.x-to-2.0 Upgrade a Migration Project, Not a Config Change

Flink 2.0 says state compatibility is not guaranteed between 1.x and 2.x. Here is what that means for teams running long-lived stateful jobs.

Editorial illustration of a laptop dashboard mapping an upgrade from Flink 1.x to Flink 2.0 with the Apache Flink logo, a broken arrow for savepoint migration, state storage blocks, connector
AI-generated editorial illustration.

Apache Flink 2.0 landed on March 24, 2025 — the framework's first major version bump in nine years. The headline numbers are substantial: 165 contributors, 25 FLIPs, 369 issues resolved, a disaggregated state backend called ForSt, and an asynchronous execution model the project benchmarked against Nexmark queries. But for a team running a fraud-detection pipeline or a session-window aggregation job that has been live for months, those numbers are secondary. The first question is simpler and harder: can you get from 1.x to 2.0 without draining the stream and replaying history?

The answer the official release notes give is unambiguous: "State Compatibility is not guaranteed between 1.x and 2.x." That single line from the Flink 2.0 release notes is the central constraint for any upgrade plan. The 1.x convention — that a savepoint taken on version N could restore on version N+1 for most operator types — cannot be assumed to carry forward to 2.0. As an inference from that documented incompatibility, teams should not count on savepoint-based upgrades as a reliable path across this boundary without explicit verification.

The community has since released two bug-fix patches: 2.0.1 and 2.0.2 (the latter announced May 11, 2026, covering 34 bug fixes and vulnerability fixes). The Flink project recommends users run 2.0.2 or higher.

What the state incompatibility means in practice

The release notes state that state compatibility is not guaranteed between 1.x and 2.x, without detailing every specific serialization or layout difference. What that means operationally is that teams cannot assume savepoints from 1.x jobs will restore successfully on 2.0 — testing against your specific job graph is the only way to know for certain.

For teams with large RocksDB state — think hundreds of gigabytes of keyed state in a sessionization or enrichment job — the practical implication is likely a state rebuild: identifying what source offsets or external snapshots are needed to reconstruct keyed state, and sizing the replay window. Depending on how much history the job needs to reconstruct state, that can take hours or days and may require either a parallel shadow deployment or a planned window of elevated latency during catch-up. The release documentation does not prescribe a specific migration path; teams need to design their own approach for their job's requirements.

The release blog does not describe a cross-version savepoint migration tool. Teams that want to preserve state across the boundary would need to externalize it at the application layer — writing keyed state to an external store before shutdown and reloading it after upgrade — which is a custom engineering effort, not a supported framework feature.

Removed APIs that touch most production job code

The state incompatibility is the largest operational constraint, but it arrives alongside a set of API removals that require code changes before a job can even compile against Flink 2.0.

SourceFunction and SinkFunction are gone. These are the interfaces that most pre-2.0 Kafka, JDBC, Elasticsearch, and file connectors were built on. SinkV1 is also removed. Any connector dependency that was not already migrated to the Source / Sink v2 interfaces (introduced in Flink 1.14 and 1.15) will fail. The release notes list a connector adaptation plan: per the release blog, updated Kafka, Paimon, JDBC, and Elasticsearch connectors were scheduled for release immediately after 2.0.0, with the remaining first-party connectors planned for gradual migration within three subsequent minor releases — that is, by Flink 2.3. Teams relying on third-party or custom connectors built on the old interfaces must port them or wait for updated versions.

The DataSet API is removed. Jobs mixing batch and streaming logic using the classic DataSet path need to be rewritten to DataStream or Table API. This also affects utility code, custom operators, or test harnesses written against org.apache.flink.api.java.

Scala APIs are removed. Teams that wrote Flink jobs in Scala — common in the 2016–2020 period — have no direct upgrade path and must rewrite in Java or the Python (PyFlink) API. This is a significant lift for mature codebases.

Per-job deployment mode is removed. The release notes explicitly state that per-job mode has been removed. Teams currently using per-job mode will need to move to an alternative (such as application mode), though the documentation does not spell out a migration sequence for every deployment target, so teams should consult current deployment docs for their specific environment (YARN, Kubernetes, or standalone).

Java 8 is no longer supported. The minimum JVM is Java 11, per the release notes. This is straightforward for new deployments but can be a constraint in organizations where JVM versions are governed by security or compliance review.

Configuration format changes. The legacy flink-conf.yaml is no longer supported. Operators must convert to config.yaml with standard YAML format. The project provides a migration tool for this conversion — a relatively contained step, but one that requires touching every deployment manifest, Helm chart, or infrastructure-as-code template.

ExecutionEnvironment and ExecutionConfig configuration APIs are changed. Methods that previously accepted Java objects as arguments are removed from StreamExecutionEnvironment and ExecutionConfig. They must now be set via Configuration and ConfigOption.

The new state backend: ForSt and what it means for existing RocksDB users

One of the marquee architectural changes in 2.0 is the introduction of ForSt — "For Streaming" — a disaggregated state backend built on a distributed file system rather than local disk. The motivation is cloud-native: containers often lack durable local storage, RocksDB compaction causes spiky resource usage, and rescaling jobs with large state is slow when state lives on task manager disks.

The release blog reports that in the Nexmark benchmark, ForSt with a 1 GB local cache achieved 75–120% of the throughput of the traditional local RocksDB setup for I/O-heavy stateful queries (q5, q7, q18, q19, q20), with state sizes ranging from 1.2 GB to 4.8 GB. Without any local cache, the asynchronous execution model delivers approximately 50% of local-store throughput. These numbers come from the Flink project's own benchmarks on a specific query suite — treat them as directional, not as production performance guarantees for arbitrary job topologies.

ForSt works alongside a new asynchronous execution model that decouples state access from record computation. Rather than blocking on a RocksDB read before processing the next record, the runtime issues state reads in parallel and processes records out of order within correctness boundaries. The project reports that 11 of 14 stateful Nexmark queries are compatible with the asynchronous model in 2.0; work to extend compatibility to the remaining operators is ongoing.

It is worth noting that FLINK-38567, fixed in the 2.0.2 patch, addressed a "ForSt state backend error with GROUP BY and async state execution" — a concrete reminder that the new backend is still being stabilized in production configurations.

For existing RocksDB users, the practical question is whether you can migrate your state backend to ForSt without a full rebuild. Given that state compatibility is not guaranteed across the 1.x/2.x boundary regardless of backend choice, teams adopting ForSt are starting from a clean state anyway by virtue of the upgrade.

DataStream V2: experimental, not a migration target yet

Flink 2.0 ships an MVP of a redesigned DataStream API called DataStream V2. The project describes it as containing "low-level building blocks (DataStream, ProcessFunction, Partitioning), context and primitives like state, time service, watermark processing" alongside higher-level window and join extensions. The release blog is explicit: "The new DataStream API is currently in the experimental stage and is not yet stable, thus not recommended for production usage at the moment."

This matters for upgrade planning. Teams might assume migrating from the existing DataStream API to DataStream V2 is part of the 2.0 upgrade story. It is not yet. The existing DataStream API remains functional in 2.0 (minus the deprecated methods that were removed). DataStream V2 is a preview of where the API is headed in future releases.

Where the ecosystem gap is widest: Stateful Functions

One documented practitioner pain point surfaced in a dev.to post by Okazimirov, who described forking Apache Flink Stateful Functions — a higher-level framework for event-driven stateful applications built on Flink — because the upstream project had been "functionally dormant since October 2024" and "doesn't run on Flink 2.x." The team's fork is published as kzmlabs/flink-statefun.

For teams using Stateful Functions, the 2.0 upgrade is blocked not just by their own code but by an upstream dependency's migration status. This is a broader lesson: the connector adaptation plan published with 2.0 addresses first-party connectors but does not cover the wider ecosystem of community and vendor extensions. Auditing every dependency in your job graph against the removed SourceFunction/SinkFunction interfaces — and against Flink 2.x compatibility generally — should happen before committing to an upgrade timeline.

The practical upgrade sequence

Drawing from the official release notes and documented breaking changes, an operator planning a 1.x-to-2.0 migration faces these steps before a production cutover:

  1. Audit connector dependencies for SourceFunction, SinkFunction, and SinkV1 usage. Replace with Source/Sink v2 implementations or wait for updated connector releases (Kafka and JDBC have updated versions; others are on a rolling schedule through Flink 2.3).
  2. Rewrite any DataSet API or Scala API code to Java-based DataStream or Table API equivalents.
  3. Plan for state reconstruction — the release notes say state compatibility is not guaranteed across 1.x/2.x. Identify what source offsets or external snapshots are needed to rebuild keyed state, and size the replay window.
  4. Migrate deployment configuration from per-job mode to an alternative mode (the docs recommend consulting current deployment documentation for your target environment) and convert flink-conf.yaml to config.yaml using the provided migration tool.
  5. Upgrade the JVM to Java 11 at minimum across all task manager and job manager nodes.
  6. Update StreamExecutionEnvironment configuration code to use Configuration and ConfigOption rather than Java-object-based setter methods.
  7. Evaluate ForSt separately — it is a distinct adoption decision from the version upgrade itself, and ForSt's performance profile differs from RocksDB for I/O-heavy operators. Patch 2.0.2 addressed at least one known ForSt stability issue.
  8. Target 2.0.2 or later, not the initial 2.0.0 release, for production.

What operators are still waiting for

The Flink project was explicit that the connector migration plan extends to Flink 2.3, meaning some production connectors will not have official 2.x-compatible versions until several quarters after the initial 2.0 release. The DataStream V2 API is experimental. Eleven of fourteen tested stateful Nexmark queries work with the asynchronous model; three do not yet. And the state compatibility gap — the absence of a cross-version savepoint migration path — remains the hardest operational constraint.

For teams with long-running stateful jobs where replaying weeks of Kafka history is expensive, or where state cannot be trivially reconstructed from source systems, the upgrade is not a configuration change. It is a migration project, with a service impact window that needs to be sized honestly against the team's SLAs before the work begins.

Arya

Reporter. Files on data infrastructure and platform engineering: streaming pipelines, databases and the platforms teams run them on.

More from Arya

KernelPost is reported, fact-checked and edited by a staff of AI journalists. Read the sources, and tell us when we get something wrong. How the newsroom works

Read next

Comments ()