confluentinc / kafka-connect-hdfs

Kafka Connect HDFS connector

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

HDFS Sink Connector 5.4.3 NullPointerException when tasks are started

smj19 opened this issue · comments

commented

I'm using HDFS Sink Connector version 5.4.3 and sometimes (not always) when I start tasks, I get the below exception:

org.apache.kafka.connect.runtime.ConnectorConfig$EnrichedConnectorConfig logAll - EnrichedConnectorConfig values:
    config.action.reload = restart
    connector.class = io.confluent.connect.hdfs.HdfsSinkConnector
    errors.log.enable = false
    errors.log.include.messages = false
    errors.retry.delay.max.ms = 60000
    errors.retry.timeout = 0
    errors.tolerance = none
    header.converter = null
    key.converter = null
    name = test-las4
    tasks.max = 81
    transforms = []
    value.converter = null        
org.apache.kafka.connect.runtime.Worker startTask - Failed to start task test-52
    java.lang.NullPointerException
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:103)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:142)
        at org.apache.kafka.connect.runtime.TaskConfig.<init>(TaskConfig.java:51)
        at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:433)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:1186)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1700(DistributedHerder.java:126)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder$13.call(DistributedHerder.java:1201)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder$13.call(DistributedHerder.java:1197)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

The failed tasks remain in FAILED state and don't recover. The below is the trace for one of the tasks when I run status:

10:30:38   - taskId: 9
10:30:38     taskState: FAILED
10:30:38     trace:  java.lang.NullPointerException
10:30:38 	at org.apache.kafka.connect.runtime.Worker$ConnectorStatusMetricsGroup.recordTaskRemoved(Worker.java:917)
10:30:38 	at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:476)
10:30:38 	at org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:1186)
10:30:38 	at org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1700(DistributedHerder.java:126)
10:30:38 	at org.apache.kafka.connect.runtime.distributed.DistributedHerder$13.call(DistributedHerder.java:1201)
10:30:38 	at org.apache.kafka.connect.runtime.distributed.DistributedHerder$13.call(DistributedHerder.java:1197)
10:30:38 	at java.util.concurrent.FutureTask.run(Unknown Source)
10:30:38 	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
10:30:38 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
10:30:38 	at java.lang.Thread.run(Unknown Source)

The only other error before this is:

 [ERROR] org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader connectorLoader - Plugin class loader for connector: 'io.confluent.connect.hdfs.HdfsSinkConnector' was not found. Returning: org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@5e37fb82

But I verified that this plugin is present when I list it:

HdfsConnector: { "class": "io.confluent.connect.hdfs.HdfsSinkConnector", "type": "sink", "version": "5.4.3" }

I see the above even otherwise and tasks start fine. Can someone please point out where I'm going wrong?

NOTE - I'm using 2 workers and starting the task from one of the workers. Also, I encounter this issue sometimes only.

commented

Can someone please help with the above?