[Bug] cdc 3.0.1和flink 1.16.2 集成存在依赖冲入问题
WuChongYong opened this issue · comments
Search before asking
- I searched in the issues and found nothing similar.
Flink version
1.16.2
Flink CDC version
3.0.1
Database and its version
mysql 5.7.26
Minimal reproduce step
guava 依赖冲突，cdc的guava依赖是
org.apache.flink
flink-shaded-guava
30.1.1-jre-16.2
flink 1.16.2的guava依赖是
org.apache.flink
flink-shaded-guava
31.1-jre-17.0
运行官网样例的api会报类找不到：
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/shaded/guava31/com/google/common/util/concurrent/ThreadFactoryBuilder
如果剔除cdc的guava包，引入31的guava包，也会出现其他类的找不到
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/shaded/guava30/com/google/common/collect/Lists
What did you expect to see?
希望3.0.1之后cdc版本能准确和flink 1.14以上的版本互相兼容
What did you see instead?
但目前3.0.1之后cdc版本能准确和flink 1.16.2 的版本不兼容
Anything else?
No response
Are you willing to submit a PR?
- I'm willing to submit a PR!
@WuChongYong
Unlike flink-table-common、flink-connector-base 、flink-core whose maven scope is provided , flink-shaded-guava and flink-shaded-force-shading will be included in the final jar package, causing dependency conflict.
Now we can see why dependency conflict occurs:
- Since Flink 1.18, version of guava is upgrade from 30 to 31.
-
Since CDC 3.0, version of guava is also upgrade from 30 to 31
-
So CDC 3.0 is compatible with Flink 1.18, CDC 2.x is is compatible with Flink 1.13-1.17. CDC 2.x and Flink 1.18 dependency conflict, CDC 3.0 and Flink 1.17 dependency conflict.
Thus, both dependencies should be shaded relocations in pom.xml.
By the way, please use English so that foreign developers can also refer to this issue.
@leonardBang , @ruanhang1993 , @PatrickRen CC, WDYT?
和我的一样我用的是flink 1.17.2 和 flink-connector-mongodb-cdc 3.0.1也出现这个问题， 然后我将flink-connector-mongodb-cdc降到2.4.2才正常
@WuChongYong , @YyItRoad , hi, Please give me your code so that I can reproduce it.