apache / flink-cdc

Flink CDC is a streaming data integration tool

Home Page:https://nightlies.apache.org/flink/flink-cdc-docs-stable

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

The db history topic or its content is fully or partially missing.

NoobsZero opened this issue · comments

Flink version
1.18.0

Flink CDC version
flink-connector-oracle-cdc:3.0.1

Database and its version
Oracle 19c PDB + CDB

Oracle 19c PDB + CDB mode SplitFetcher thread 0 received unexpected exception while polling the records
Properties debeziumProperties = new Properties();
debeziumProperties.setProperty("log.mining.strategy", "online_catalog");
debeziumProperties.setProperty("database.pdb.name", "PDB");

JdbcIncrementalSource<String> oracleChangeEventSource =
        new OracleSourceBuilder<String>()
                .hostname("10.196.**.**")
                .port(1521)
                .databaseList("CDB") // monitor XE database
                .schemaList("C##FLINKUSER") // monitor inventory schema
                .tableList("APPLICATION_WEB_LANG") // monitor products table
                .username("c##flinkuser")
                .password("flinkpw")
                .deserializer(new JsonDebeziumDeserializationSchema())

// .includeSchemaChanges(true) // output the schema changes as well
.startupOptions(StartupOptions.initial())
.debeziumProperties(debeziumProperties)
.splitSize(2)
.build();

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// enable checkpoint
env.enableCheckpointing(3000L);
// set the source parallelism to 4
env.fromSource(
                oracleChangeEventSource,
                WatermarkStrategy.noWatermarks(),
                "OracleParallelSource")
        .setParallelism(1)
        .print()
        .setParallelism(1);
env.execute("Print Oracle Snapshot + RedoLog");

Caused by: io.debezium.DebeziumException: The db history topic or its content is fully or partially missing. Please check database history topic configuration and re-execute the snapshot.
at io.debezium.relational.HistorizedRelationalDatabaseSchema.recover(HistorizedRelationalDatabaseSchema.java:59) ~[debezium-core-1.9.8.Final.jar:1.9.8.Final]
at org.apache.flink.cdc.connectors.oracle.source.reader.fetch.OracleSourceFetchTaskContext.validateAndLoadDatabaseHistory(OracleSourceFetchTaskContext.java:275) ~[classes/:?]
at org.apache.flink.cdc.connectors.oracle.source.reader.fetch.OracleSourceFetchTaskContext.configure(OracleSourceFetchTaskContext.java:118) ~[classes/:?]
at org.apache.flink.cdc.connectors.base.source.reader.external.IncrementalSourceStreamFetcher.submitTask(IncrementalSourceStreamFetcher.java:84) ~[classes/:?]
at org.apache.flink.cdc.connectors.base.source.reader.IncrementalSourceSplitReader.submitStreamSplit(IncrementalSourceSplitReader.java:261) ~[classes/:?]
at org.apache.flink.cdc.connectors.base.source.reader.IncrementalSourceSplitReader.pollSplitRecords(IncrementalSourceSplitReader.java:153) ~[classes/:?]
at org.apache.flink.cdc.connectors.base.source.reader.IncrementalSourceSplitReader.fetch(IncrementalSourceSplitReader.java:98) ~[classes/:?]
at org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58) ~[flink-connector-base-1.18.0.jar:1.18.0]
at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:165) ~[flink-connector-base-1.18.0.jar:1.18.0]
at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:117) ~[flink-connector-base-1.18.0.jar:1.18.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_221]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_221]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_221]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_221]
... 1 more
40428 [Source: OracleParallelSource -> Sink: Print to Std. Out (1/1)https://github.com/apache/flink-cdc/issues/18] INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for Source: OracleParallelSource -> Sink: Print to Std. Out (1/1)#18 (8719e776463c5834ef8a61c2bc1be1a1_cbc357ccb763df2852fee8c4fc7d55f2_0_18).
40429 [flink-pekko.actor.default-dispatcher-5] INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and sending final execution state FAILED to JobManager for task Source: OracleParallelSource -> Sink: Print to Std. Out (1/1)#18 8719e776463c5834ef8a61c2bc1be1a1_cbc357ccb763df2852fee8c4fc7d55f2_0_18.
40430 [flink-pekko.actor.default-dispatcher-8] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: OracleParallelSource -> Sink: Print to Std. Out (1/1) (8719e776463c5834ef8a61c2bc1be1a1_cbc357ccb763df2852fee8c4fc7d55f2_0_18) switched from RUNNING to FAILED on 76866f22-09ba-4739-bcab-23fd0386fbbc @ 127.0.0.1 (dataPort=-1).
java.lang.RuntimeException: One or more fetchers have encountered exception
at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:263) ~[flink-connector-base-1.18.0.jar:1.18.0]
at org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:185) ~[flink-connector-base-1.18.0.jar:1.18.0]
at org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:147) ~[flink-connector-base-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:419) ~[flink-streaming-java-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68) ~[flink-streaming-java-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[flink-streaming-java-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:562) ~[flink-streaming-java-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231) ~[flink-streaming-java-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:858) ~[flink-streaming-java-1.18.0.jar:1.18.0]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:807) ~[flink-streaming-java-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953) ~[flink-runtime-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:932) ~[flink-runtime-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746) ~[flink-runtime-1.18.0.jar:1.18.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) ~[flink-runtime-1.18.0.jar:1.18.0]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_221]
Caused by: java.lang.RuntimeException: SplitFetcher thread 0 received unexpected exception while polling the records
at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:168) ~[flink-connector-base-1.18.0.jar:1.18.0]
at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:117) ~[flink-connector-base-1.18.0.jar:1.18.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_221]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_221]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_221]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_221]
... 1 more

Sorry we no longer accept new issues on GitHub. Please report it on Apache Jira under the project Flink, using the component tag Flink CDC. Thank you!