[Bug]: 任务oom后exit code=0
yinlice opened this issue · comments
What happened?
2024-01-30 13:40:10.440 INFO [Thread-966] TaskLog-517-58215-0:183 - 2024-01-30 13:40:10.435 INFO [ job-0] c.w.addax.plugin.writer.hivewriter.HiveWriter$Job:315 - end splitting.
2024-01-30 13:40:10.440 INFO [Thread-966] TaskLog-517-58215-0:183 - 2024-01-30 13:40:10.435 INFO [ job-0] com.wgzhao.addax.core.job.JobContainer:689 - Addax Writer.Job [hivewriter] splits to [6144] tasks.
2024-01-30 13:40:13.333 INFO [Thread-966] TaskLog-517-58215-0:183 - java.lang.OutOfMemoryError: Java heap space
2024-01-30 13:40:13.333 INFO [Thread-966] TaskLog-517-58215-0:183 - Dumping heap to /opt/addax/addax-4.0.9-SNAPSHOT/java_pid2424.hprof ...
2024-01-30 13:40:14.678 INFO [Thread-966] TaskLog-517-58215-0:183 - Heap dump file created [352954372 bytes in 1.346 secs]
2024-01-30 13:40:14.682 INFO [Thread-966] TaskLog-517-58215-0:183 - 2024-01-30 13:40:14.681 INFO [ job-0] c.w.addax.plugin.writer.hivewriter.HiveWriter$Job:276 - FileSystem, schema defaultFS: oss://**** closed
2024-01-30 13:40:14.682 INFO [Thread-966] TaskLog-517-58215-0:183 - 2024-01-30 13:40:14.682 INFO [ job-0] c.wgzhao.addax.plugin.writer.hivewriter.HdfsHelper:880 - 未检测到分区和表名，跳过小文件合并
2024-01-30 13:40:14.682 INFO [Thread-966] TaskLog-517-58215-0:183 - 2024-01-30 13:40:14.682 INFO [ job-0] c.w.addax.plugin.writer.hivewriter.HiveWriter$Job:279 - Writer Job destroy done
2024-01-30 13:40:14.682 INFO [Thread-966] TaskLog-517-58215-0:183 - 2024-01-30 13:40:14.682 INFO [ job-0] c.w.a.plugin.reader.mysqlreader.MysqlReader$Job:95 - Reader Job destroy done
2024-01-30 13:40:14.683 INFO [Thread-966] TaskLog-517-58215-0:183 - 2024-01-30 13:40:14.683 INFO [ job-0] com.wgzhao.addax.core.job.JobContainer:194 - PerfTrace not enable!
2024-01-30 13:40:15.073 INFO [WaitForProcess-java.lang.UNIXProcess@60170af0] TaskLog-517-58215-0:16 - process stopped. exit(0) @wgzhao
Version
4.0.10
OS Type
Linux (Default), MacOS
Java JDK Version
Oracle JDK 1.8.0
Relevant log output
No response
我本地测试出现OOM后，退出码为 -2 ，而不是 0.
你可以收工执行，跑完后，立刻执行 echo $? 看看退出码是多少
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to /Users/hong/Downloads/software/addax-4.1.3/java_pid32040.hprof ...
Heap dump file created [13368395 bytes in 0.079 secs]
Exception in thread "reader-0-2" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "reader-0-1" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "reader-0-0" java.lang.OutOfMemoryError: GC overhead limit exceeded
2024-02-20 17:48:51.445 [ writer-0-0] ERROR WriterRunner - Writer Runner Received Exceptions:
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "taskGroup-0" com.wgzhao.addax.common.exception.AddaxException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at com.wgzhao.addax.common.exception.AddaxException.asAddaxException(AddaxException.java:66)
at com.wgzhao.addax.core.taskgroup.TaskGroupContainer.start(TaskGroupContainer.java:188)
at com.wgzhao.addax.core.taskgroup.runner.TaskGroupContainerRunner.run(TaskGroupContainerRunner.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
2024-02-20 17:48:53.049 [ job-0] INFO StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00%
2024-02-20 17:48:53.050 [ job-0] INFO JobContainer -
Job start at : 2024-02-20 17:48:47
Job end at : 2024-02-20 17:48:51
Job took secs : 4s
Average bps : 0B/s
Average rps : 0rec/s
Number of rec : 0
Failed record : 0
@hongdeMacBook-Pro bin %
hong@hongdeMacBook-Pro bin % echo $?
0
hong@hongdeMacBook-Pro bin %
看着还是0 ,看了下启动脚本 也没有exit -2的逻辑呢
测试方法==> 修改启动脚本jvm 模拟OOM
DEFAULT_JVM="-Xms20m -Xmx40m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${ADDAX_HOME}"
程序 OOM 时的退出码不受程序自身控制，应该是 Java 虚拟机来管理的，我尝试在 bin/addax.sh 里增加了 -XX:+ExitOnOutOfMemoryError 参数后，再出现 OOM 情况，退出码变成 3 了。
运行输出如下：
$ bin/addax.sh job/mysql2stream.json
___ _ _
/ _ \ | | | |
/ /_\ \ __| | __| | __ ___ __
| _ |/ _` |/ _` |/ _` \ \/ /
| | | | (_| | (_| | (_| |> <
\_| |_/\__,_|\__,_|\__,_/_/\_\
:: Addax version :: (v4.1.4-SNAPSHOT)
2024-02-23 13:34:13.304 [ main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2024-02-23 13:34:13.318 [ main] INFO Engine -
{
"setting":{
"speed":{
"channel":3,
"bytes":-1
}
},
"content":{
"reader":{
"name":"mysqlreader",
"parameter":{
"username":"root",
"password":"*****",
"column":[
"*"
],
"autoPk":true,
"connection":[
{
"table":[
"tbl[0-128]"
],
"jdbcUrl":[
"jdbc:mysql://127.0.0.1:3306/test"
]
}
]
}
},
"writer":{
"name":"txtfilewriter",
"parameter":{
"path":"/tmp/out",
"fileName":"test",
"writeMode":"truncate",
"encoding":"UTF-8"
}
}
}
}
2024-02-23 13:34:13.354 [ main] INFO JobContainer - The jobContainer begins to process the job.
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /opt/app/addax/java_pid159876.hprof ...
Heap dump file created [10481570 bytes in 0.028 secs]
Terminating due to java.lang.OutOfMemoryError: Java heap space
$ echo $?
3