Error Writing Rows to BigQuery

[Luke Jasko]

While attempting to use kafka-connect-bigquery connector, we see the following error in tasks running in Kafka connect after a period of time:

Caused by: com.wepay.kafka.connect.bigquery.exception.BigQueryConnectException: Failed to write rows after BQ table creation or schema update within 30 attempts for: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=bot_production_thing_data, tableId=farm_pit_0_annotated_thing_data}}
  at com.wepay.kafka.connect.bigquery.write.row.AdaptiveBigQueryWriter.performWriteRequest(AdaptiveBigQueryWriter.java:138)
  at com.wepay.kafka.connect.bigquery.write.row.BigQueryWriter.writeRows(BigQueryWriter.java:112)
  at com.wepay.kafka.connect.bigquery.write.batch.TableWriter.run(TableWriter.java:93)
  ... 3 more

The error is confusing and doesn't offer any additional insight into the cause of the problem. I don't think the culprit could relate to table creation or schema updates because we manually set up the table along with its schema ahead of time. How do we determine what is causing our connector tasks to fail in this fashion? Thanks in advance

[Luke Jasko]

I was able to find the culprit by changing Kafka Connect's log level to TRACE. It ended up being unexpected invalid schema errors for some records in Kafka. When the write response from BigQuery contains invalid schema errors, the following is logged in onlyContainsInvalidSchemaErrors():

    logger.trace("write response contained errors: \n{}", errors);