[Bug]: sqlserver writemode 配置 update(id)报错
sxiongzhang opened this issue · comments
What happened?
sqlserver version：
Microsoft SQL Server 2019 (RTM-CU24) (KB5031908) - 15.0.4345.5 (X64)
Dec 4 2023 14:44:16
Copyright (C) 2019 Microsoft Corporation
Developer Edition (64-bit) on Linux (Ubuntu 20.04.6 LTS)
error info:
2024-02-06 16:48:52.498 [0-0-0-writer] ERROR StdoutPluginCollector -
com.microsoft.sqlserver.jdbc.SQLServerException: Incorrect syntax near ')'.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:262) ~[mssql-jdbc-9.2.1.jre8.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1632) ~[mssql-jdbc-9.2.1.jre8.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:602) ~[mssql-jdbc-9.2.1.jre8.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:524) ~[mssql-jdbc-9.2.1.jre8.jar:na]
Version
4.1.3 (Default)
OS Type
No response
Java JDK Version
Oracle JDK 1.8.0
Relevant log output
No response
请贴出完整的输出日志
是这样的，我这边用的是您这边的sqlserverwriter 的writerMode 逻辑改的原版datax，不知是否是我这边修改的有问题，麻烦大佬给看看，非常感谢，完整日志如下：
{
"content":[
{
"reader":{
"name":"sqlserverreader",
"parameter":{
"connection":[
{
"jdbcUrl":[
"jdbc:sqlserver://xxx:1433;DatabaseName=TestDB"
],
"querySql":[
"select id,name,age from test_writemode;"
]
}
],
"password":"**********",
"username":"SA",
"where":""
}
},
"writer":{
"name":"sqlserverwriter",
"parameter":{
"column":[
"*"
],
"connection":[
{
"jdbcUrl":"jdbc:sqlserver://xxx:1433;DatabaseName=TestDB",
"table":[
"test_writemode_copy"
]
}
],
"password":"**********",
"postSql":[],
"preSql":[],
"session":[],
"username":"SA",
"writeMode":"update(id)"
}
}
}
],
"setting":{
"speed":{
"channel":1
}
}
}
2024-02-06 16:07:11.783 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null
2024-02-06 16:07:11.785 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2024-02-06 16:07:11.785 [main] INFO JobContainer - DataX jobContainer starts job.
2024-02-06 16:07:11.787 [main] INFO JobContainer - Set jobId = 0
2024-02-06 16:07:29.157 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:sqlserver://xxx:1433;DatabaseName=TestDB.
2024-02-06 16:07:48.594 [job-0] INFO OriginalConfPretreatmentUtil - table:[test_writemode_copy] all columns:[
id,name,age
].
2024-02-06 16:07:48.594 [job-0] WARN OriginalConfPretreatmentUtil - 您的配置文件中的列配置信息存在风险. 因为您配置的写入数据库表的列为*，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改.
2024-02-06 16:34:53.410 [job-0] INFO OriginalConfPretreatmentUtil - Write data [
MERGE INTO %s A USING ( SELECT ) TMP ON ( ) WHEN MATCHED THEN UPDATE SET id = ?,name = ?,age = ? WHEN NOT MATCHED THEN INSERT (id,name,age) VALUES(?,?,?)
], which jdbcUrl like:[jdbc:sqlserver://xxx:1433;DatabaseName=TestDB]
2024-02-06 16:36:11.882 [job-0] INFO JobContainer - jobContainer starts to do prepare ...
2024-02-06 16:36:12.492 [job-0] INFO JobContainer - DataX Reader.Job [sqlserverreader] do prepare work .
2024-02-06 16:36:12.495 [job-0] INFO JobContainer - DataX Writer.Job [sqlserverwriter] do prepare work .
2024-02-06 16:36:13.245 [job-0] INFO JobContainer - jobContainer starts to do split ...
2024-02-06 16:36:14.027 [job-0] INFO JobContainer - Job set Channel-Number to 1 channels.
2024-02-06 16:36:14.062 [job-0] INFO JobContainer - DataX Reader.Job [sqlserverreader] splits to [1] tasks.
2024-02-06 16:36:14.074 [job-0] INFO JobContainer - DataX Writer.Job [sqlserverwriter] splits to [1] tasks.
2024-02-06 16:36:15.094 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2024-02-06 16:36:17.908 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2024-02-06 16:36:17.952 [job-0] INFO JobContainer - Running by standalone Mode.
2024-02-06 16:36:18.019 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2024-02-06 16:36:18.062 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated.
2024-02-06 16:36:18.063 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
2024-02-06 16:36:18.173 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2024-02-06 16:36:20.148 [0-0-0-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select id,name,age from test_writemode;
] jdbcUrl:[jdbc:sqlserver://xxx:1433;DatabaseName=TestDB].
2024-02-06 16:36:22.701 [0-0-0-writer] INFO CommonRdbmsWriter$Task - write SQLServer using update(id) mode
2024-02-06 16:36:28.550 [job-0] INFO StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00%
2024-02-06 16:36:30.259 [0-0-0-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select id,name,age from test_writemode;
] jdbcUrl:[jdbc:sqlserver://xxx:1433;DatabaseName=TestDB].
2024-02-06 16:36:40.933 [job-0] INFO StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00%
2024-02-06 16:36:58.924 [job-0] INFO StandAloneJobContainerCommunicator - Total 4 records, 29 bytes | Speed 2B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00%
2024-02-06 16:40:31.797 [0-0-0-writer] WARN CommonRdbmsWriter$Task - 回滚此次写入, 采用每次写入一行方式提交. 因为:Incorrect syntax near ')'.
2024-02-06 16:40:31.798 [job-0] INFO StandAloneJobContainerCommunicator - Total 4 records, 29 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00%
2024-02-06 16:40:47.982 [job-0] INFO StandAloneJobContainerCommunicator - Total 4 records, 29 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00%
2024-02-06 16:40:49.456 [0-0-0-writer] ERROR StdoutPluginCollector -
com.microsoft.sqlserver.jdbc.SQLServerException: Incorrect syntax near ')'.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:262) ~[mssql-jdbc-9.2.1.jre8.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1632) ~[mssql-jdbc-9.2.1.jre8.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:602) ~[mssql-jdbc-9.2.1.jre8.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:524) ~[mssql-jdbc-9.2.1.jre8.jar:na]
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7418) ~[mssql-jdbc-9.2.1.jre8.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:3272) ~[mssql-jdbc-9.2.1.jre8.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:247) ~[mssql-jdbc-9.2.1.jre8.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:222) ~[mssql-jdbc-9.2.1.jre8.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.execute(SQLServerPreparedStatement.java:505) ~[mssql-jdbc-9.2.1.jre8.jar:na]
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.doOneInsert(CommonRdbmsWriter.java:437) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.doBatchInsert(CommonRdbmsWriter.java:417) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.startWriteWithConnection(CommonRdbmsWriter.java:326) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.startWrite(CommonRdbmsWriter.java:348) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.sqlserverwriter.SqlServerWriter$Task.startWrite(SqlServerWriter.java:82) [sqlserverwriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56) [datax-core-0.0.1-SNAPSHOT.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
2024-02-06 16:40:49.482 [0-0-0-writer] ERROR StdoutPluginCollector - 脏数据:
{"exception":"Incorrect syntax near ')'.","record":[{"byteSize":1,"index":0,"rawData":1,"type":"LONG"},{"byteSize":3,"index":1,"rawData":"Tom","type":"STRING"},{"byteSize":2,"index":2,"rawData":18,"type":"LONG"},{"byteSize":1,"index":3,"rawData":1,"type":"LONG"},{"byteSize":3,"index":4,"rawData":"Tom","type":"STRING"},{"byteSize":2,"index":5,"rawData":18,"type":"LONG"}],"type":"writer"}
2024-02-06 16:41:03.153 [job-0] INFO StandAloneJobContainerCommunicator - Total 4 records, 29 bytes | Speed 0B/s, 0 records/s | Error 1 records, 12 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00%
2024-02-06 16:43:04.773 [0-0-0-writer] ERROR StdoutPluginCollector - 脏数据:
{"exception":"Incorrect syntax near ')'.","record":[{"byteSize":1,"index":0,"rawData":2,"type":"LONG"},{"byteSize":5,"index":1,"rawData":"Jerry","type":"STRING"},{"byteSize":2,"index":2,"rawData":19,"type":"LONG"},{"byteSize":1,"index":3,"rawData":2,"type":"LONG"},{"byteSize":5,"index":4,"rawData":"Jerry","type":"STRING"},{"byteSize":2,"index":5,"rawData":19,"type":"LONG"}],"type":"writer"}
2024-02-06 16:43:04.777 [job-0] INFO StandAloneJobContainerCommunicator - Total 4 records, 29 bytes | Speed 0B/s, 0 records/s | Error 1 records, 12 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00%
Listening for transport dt_socket at address: 9999
2024-02-06 16:43:04.784 [job-0] INFO VMInfo -
[delta cpu info] =>
curDeltaCpu | averageCpu | maxDeltaCpu | minDeltaCpu
-1.00% | -1.00% | -1.00% | -1.00%
[delta memory info] =>
NAME | used_size | used_percent | max_used_size | max_percent
PS Eden Space | 143.50MB | 28.00% | 143.50MB | 28.00%
Code Cache | 5.49MB | 97.59% | 5.49MB | 97.59%
Compressed Class Space | 2.09MB | 92.86% | 2.09MB | 92.86%
PS Survivor Space | 0.00MB | 0.00% | 0.00MB | 0.00%
PS Old Gen | 0.00MB | 0.00% | 0.00MB | 0.00%
Metaspace | 19.84MB | 96.76% | 19.84MB | 96.76%
[delta gc info] =>
NAME | curDeltaGCCount | totalGCCount | maxDeltaGCCount | minDeltaGCCount | curDeltaGCTime | totalGCTime | maxDeltaGCTime | minDeltaGCTime
PS MarkSweep | 0 | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s | 0.000s
PS Scavenge | 0 | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s | 0.000s
2024-02-06 16:43:04.785 [0-0-0-writer] ERROR StdoutPluginCollector - 脏数据:
{"exception":"Incorrect syntax near ')'.","record":[{"byteSize":1,"index":0,"rawData":3,"type":"LONG"},{"byteSize":5,"index":1,"rawData":"Alice","type":"STRING"},{"byteSize":2,"index":2,"rawData":20,"type":"LONG"},{"byteSize":1,"index":3,"rawData":3,"type":"LONG"},{"byteSize":5,"index":4,"rawData":"Alice","type":"STRING"},{"byteSize":2,"index":5,"rawData":20,"type":"LONG"}],"type":"writer"}
2024-02-06 16:43:04.787 [0-0-0-writer] ERROR StdoutPluginCollector - 脏数据:
{"exception":"Incorrect syntax near ')'.","record":[{"byteSize":1,"index":0,"rawData":4,"type":"LONG"},{"byteSize":4,"index":1,"rawData":"Jack","type":"STRING"},{"byteSize":2,"index":2,"rawData":21,"type":"LONG"},{"byteSize":1,"index":3,"rawData":4,"type":"LONG"},{"byteSize":4,"index":4,"rawData":"Jack","type":"STRING"},{"byteSize":2,"index":5,"rawData":21,"type":"LONG"}],"type":"writer"}
2024-02-06 16:43:04.875 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[406712]ms
2024-02-06 16:43:04.876 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed it's tasks.
2024-02-06 16:43:14.786 [job-0] INFO StandAloneJobContainerCommunicator - Total 4 records, 29 bytes | Speed 0B/s, 0 records/s | Error 4 records, 58 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2024-02-06 16:43:14.787 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks.
2024-02-06 16:43:14.788 [job-0] INFO JobContainer - DataX Writer.Job [sqlserverwriter] do post work.
2024-02-06 16:43:14.789 [job-0] INFO JobContainer - DataX Reader.Job [sqlserverreader] do post work.
2024-02-06 16:43:14.790 [job-0] INFO JobContainer - DataX jobId [0] completed successfully.
2024-02-06 16:43:14.792 [job-0] INFO HookInvoker - No hook invoked, because base dir not exists or is a file: /opt/module/datax/hook
2024-02-06 16:43:14.793 [job-0] INFO JobContainer -
[total cpu info] =>
averageCpu | maxDeltaCpu | minDeltaCpu
-1.00% | -1.00% | -1.00%
[total gc info] =>
NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime
PS MarkSweep | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s
PS Scavenge | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s
2024-02-06 16:43:14.793 [job-0] INFO JobContainer - PerfTrace not enable!
2024-02-06 16:43:14.794 [job-0] INFO StandAloneJobContainerCommunicator - Total 4 records, 29 bytes | Speed 0B/s, 0 records/s | Error 4 records, 58 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2024-02-06 16:43:14.795 [job-0] INFO JobContainer -
任务启动时刻 : 2024-02-06 16:07:11
任务结束时刻 : 2024-02-06 16:43:14
任务总计耗时 : 2163s
任务平均流量 : 0B/s
记录写入速度 : 0rec/s
读出记录总数 : 4
读写失败总数 : 4
本地测试无问题，过程如下：
- 创建表
create table test_writemode
(
id int,
name varchar(100),
age int
);
任务执行的完整日志：
2024-02-08 11:32:55.263 [ main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSyst
emImpl
2024-02-08 11:32:55.272 [ main] INFO Engine -
{
"content":{
"reader":{
"name":"streamreader",
"parameter":{
"column":[
{
"value":"100",
"type":"long"
},
{
"value":"updated",
"type":"string"
},
{
"random":"100,1000",
"type":"long"
}
],
"sliceRecordCount":1000
}
},
"writer":{
"name":"sqlserverwriter",
"parameter":{
"column":[
"*"
],
"connection":[
{
"jdbcUrl":"jdbc:sqlserver://wgzhao-pc:1433;DatabaseName=master",
"table":[
"test_writemode"
]
}
],
"username":"sa",
"password":"*****",
"writeMode":"update (id)"
}
}
},
"setting":{
"speed":{
"bytes":-1,
"channel":1
}
}
}
2024-02-08 11:32:55.284 [ main] INFO JobContainer - The jobContainer begins to process the job.
2024-02-08 11:32:55.613 [ job-0] INFO OriginalConfPretreatmentUtil - The table [test_writemode] has columns [id,name,age].
2024-02-08 11:32:55.613 [ job-0] WARN OriginalConfPretreatmentUtil - There are some risks in the column configuration. Bec
ause you did not configure the columns to read the database table, changes in the number and types of fields in your table may af
fect the correctness of the task or even cause errors.
2024-02-08 11:32:55.614 [ job-0] INFO OriginalConfPretreatmentUtil - Writing data using [MERGE INTO %s A USING ( SELECT ?
AS id ) TMP ON (TMP.id = A.id ) WHEN MATCHED THEN UPDATE SET name = ?,age = ? WHEN NOT MATCHED THEN INSERT (id,name,age) VALUES (
?,?,? );].
2024-02-08 11:32:55.615 [ job-0] INFO JobContainer - The Reader.Job [streamreader] perform prepare work .
2024-02-08 11:32:55.615 [ job-0] INFO JobContainer - The Writer.Job [sqlserverwriter] perform prepare work .
2024-02-08 11:32:55.615 [ job-0] INFO JobContainer - Job set Channel-Number to 1 channel(s).
2024-02-08 11:32:55.616 [ job-0] INFO JobContainer - The Reader.Job [streamreader] is divided into [1] task(s).
2024-02-08 11:32:55.616 [ job-0] INFO JobContainer - The Writer.Job [sqlserverwriter] is divided into [1] task(s).
2024-02-08 11:32:55.630 [ job-0] INFO JobContainer - The Scheduler launches [1] taskGroup(s).
2024-02-08 11:32:55.634 [ taskGroup-0] INFO TaskGroupContainer - The taskGroupId=[0] started [1] channels for [1] tasks.
2024-02-08 11:32:55.636 [ taskGroup-0] INFO Channel - The Channel set byte_speed_limit to -1, No bps activated.
2024-02-08 11:32:55.636 [ taskGroup-0] INFO Channel - The Channel set record_speed_limit to -1, No tps activated.
2024-02-08 11:32:55.698 [ writer-0-0] INFO CommonRdbmsWriter$Task - write SQLServer using update (id) mode
2024-02-08 11:32:58.640 [ job-0] INFO AbstractScheduler - The scheduler has completed all tasks.
2024-02-08 11:32:58.640 [ job-0] INFO JobContainer - The Writer.Job [sqlserverwriter] perform post work.
2024-02-08 11:32:58.641 [ job-0] INFO JobContainer - The Reader.Job [streamreader] perform post work.
2024-02-08 11:32:58.643 [ job-0] INFO StandAloneJobContainerCommunicator - Total 1000 records, 18000 bytes | Speed 5.86KB/
s, 333 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.001s | All Task WaitReaderTime 0.057s | Percentage 100.
00%
2024-02-08 11:32:58.644 [ job-0] INFO JobContainer -
Job start at : 2024-02-08 11:32:55
Job end at : 2024-02-08 11:32:58
Job took secs : 3s
Average bps : 5.86KB/s
Average rps : 333rec/s
Number of rec : 1000
Failed record : 0
我看你的报错是说语法错误，你尝试着把 "writeMode":"update(id)" 改成 "writeMode":"update (id)" 看看，是不是 update 之后少了一个空格导致的。
不过我这里特意测试了有空格和无空格的情况，都是可以正常运行的。
从你日志的输出
Write data [
MERGE INTO %s A USING ( SELECT ) TMP ON ( ) WHEN MATCHED THEN UPDATE SET id = ?,name = ?,age = ? WHEN NOT MATCHED THEN INSERT (id,name,age) VALUES(?,?,?)
], which jdbcUrl like:[jdbc:sqlserver://xxx:1433;DatabaseName=TestDB]
来看，是 update(id) 的 id 没有解析出来，导致这个语句报语法错误。
正确的拼接语句应该是
[MERGE INTO %s A USING ( SELECT ?
AS id ) TMP ON (TMP.id = A.id ) WHEN MATCHED THEN UPDATE SET name = ?,age = ? WHEN NOT MATCHED THEN INSERT (id,name,age) VALUES (
?,?,? );].
所以检查你的分析 writeMode 参数的代码
TO %s A USING ( SELECT ?
AS id ) TMP ON (TMP.id = A.id ) WHEN MATCHED THEN UPDATE SET name = ?,age = ? WHEN NOT MATCHED THEN INSERT (id,name,age) VALUES (
?,?,? );].
嗯嗯，应该是这个问题，我看一下，多谢大佬，非常感谢