[Bug] [Doris Sink] Sink Doris Error

Question

[Bug] [Doris Sink] Sink Doris Error

taohaozhi1129 opened this issue 3 months ago · comments

Search before asking

I had searched in the issues and found no similar issues.

What happened

When using Sink Doris to write data into Doris, in the current state evaluation, if either of the conditions is not met, the write failure is not logged. However, the write operation ultimately fails. Therefore, the judgment should be modified to consider any status other than "success" as a failure.

SeaTunnel Version

SeaTunnel Version 2.3.4

SeaTunnel Config

sink {
    Doris {
      fenodes = "localhost:8030"
      username = "root"
      password = "123#7*"
      database = "ods_itd_sit"
      table = "ods_oa_ecology_hrmdepartment_df"
      sink.enable-2pc = "true"
      sink.label-prefix = "st"
      data_save_mode="DROP_DATA"
      doris.config = {
      format = "json"
      read_json_by_line="true"
      }
  }
}

Running Command

seatunnel run job

Error Exception

2024-04-23 01:53:08,513 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Reason:SeaTunnel job executed failed 
	
	2024-04-23 01:53:08,514 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
		at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191)
		at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
		at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
	Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.engine.server.checkpoint.CheckpointException: Checkpoint notify complete failed
		at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:255)
		at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:251)
		at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.notifyCompleted(CheckpointCoordinator.java:328)
		at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:736)
		at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$startTriggerPendingCheckpoint$7(CheckpointCoordinator.java:480)
		at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
		at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
		at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
		at java.lang.Thread.run(Thread.java:750)
	Caused by: java.util.concurrent.CompletionException: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.connectors.doris.exception.DorisConnectorException: ErrorCode:[Doris-02], ErrorDescription:[commit error] - {
	    "status": "ANALYSIS_ERROR",
	    "msg": "errCode = 2, detailMessage = transaction [1501371] is already visible, not pre-committed."
	}
		at org.apache.seatunnel.connectors.doris.sink.committer.DorisCommitter.commitTransaction(DorisCommitter.java:132)
		at org.apache.seatunnel.connectors.doris.sink.committer.DorisCommitter.commit(DorisCommitter.java:71)
		at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.notifyCheckpointComplete(SinkFlowLifeCycle.java:239)
		at org.apache.seatunnel.engine.server.task.SeaTunnelTask.lambda$notifyCheckpointComplete$7(SeaTunnelTask.java:379)
		at org.apache.seatunnel.engine.common.utils.ExceptionUtil.sneaky(ExceptionUtil.java:133)
		at org.apache.seatunnel.engine.server.task.SeaTunnelTask.lambda$notifyAllAction$12(SeaTunnelTask.java:399)
		at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
		at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
		at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
		at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
		at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
		at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
		at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
		at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
		at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
		at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
		at org.apache.seatunnel.engine.server.task.SeaTunnelTask.notifyAllAction(SeaTunnelTask.java:399)
		at org.apache.seatunnel.engine.server.task.SeaTunnelTask.notifyCheckpointComplete(SeaTunnelTask.java:379)
		at org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointFinishedOperation.lambda$run$0(CheckpointFinishedOperation.java:91)
		at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48)
		at org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointFinishedOperation.run(CheckpointFinishedOperation.java:81)
		at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
		at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
		at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
		at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213)
		at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:175)
		at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:139)
		at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
		at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
	
		at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
		at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
		at java.util.concurrent.CompletableFuture.biRelay(CompletableFuture.java:1300)
		at java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1284)
		at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
		at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
		at com.hazelcast.spi.impl.AbstractInvocationFuture.onComplete(AbstractInvocationFuture.java:1243)
		at com.hazelcast.spi.impl.AbstractInvocationFuture.complete0(AbstractInvocationFuture.java:1234)
		at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionallyInternal(AbstractInvocationFuture.java:1223)
		at com.hazelcast.spi.impl.operationservice.impl.Invocation.completeExceptionally(Invocation.java:680)
		at com.hazelcast.spi.impl.operationservice.impl.Invocation.notifyThrowable(Invocation.java:386)
		at com.hazelcast.spi.impl.operationservice.impl.Invocation.notifyError(Invocation.java:330)
		at com.hazelcast.spi.impl.operationservice.impl.Invocation.sendResponse(Invocation.java:230)
		at com.hazelcast.spi.impl.operationservice.Operation.sendResponse(Operation.java:483)
		at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.sendResponseAfterOperationError(OperationRunnerImpl.java:426)
		at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.handleOperationError(OperationRunnerImpl.java:420)
		at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:253)
		at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213)
		at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:175)
		at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:139)
		at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
		at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
	Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.connectors.doris.exception.DorisConnectorException: ErrorCode:[Doris-02], ErrorDescription:[commit error] - {
	    "status": "ANALYSIS_ERROR",
	    "msg": "errCode = 2, detailMessage = transaction [1501371] is already visible, not pre-committed."
	}
		at org.apache.seatunnel.connectors.doris.sink.committer.DorisCommitter.commitTransaction(DorisCommitter.java:132)

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct