CDC内存不够，Final启动后因内存不够退出，修改polardbx-cdc/config/config.properties里的mem_size=3000无效
jmbkeyes opened this issue · comments
There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 13421772800 bytes for committing reserved memory.
Possible reasons:
The system is out of physical RAM or swap space
The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
Possible solutions:
Reduce memory load on the system
Increase physical memory or swap space
Check if swap backing store is full
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes (-Xss)
Set larger code cache with -XX:ReservedCodeCacheSize=
This output file may be truncated or incomplete.
Out of Memory Error (os_linux.cpp:2756), pid=126113, tid=0x00007f2e6906d700
JRE version: (8.0_321-b07) (build )
Java VM: Java HotSpot(TM) 64-Bit Server VM (25.321-b07 mixed mode linux-amd64 compressed oops)
Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
--------------- T H R E A D ---------------
Current thread (0x00007f2e60011000): JavaThread "Unknown thread" [_thread_in_vm, id=126116, stack(0x00007f2e68f6e000,0x00007f2e6906e000)]
Stack: [0x00007f2e68f6e000,0x00007f2e6906e000], sp=0x00007f2e6906c4d0, free space=1017k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xae0635] VMError::report_and_die()+0x2e5
V [libjvm.so+0x4e2aa7] report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x67
V [libjvm.so+0x919ba0] os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x100
V [libjvm.so+0x9111bf] os::commit_memory(char*, unsigned long, unsigned long, bool)+0x1f
V [libjvm.so+0xadc88c] VirtualSpace::initialize(ReservedSpace, unsigned long)+0x20c
V [libjvm.so+0x5efe17] CardGeneration::CardGeneration(ReservedSpace, unsigned long, int, GenRemSet*)+0xc7
V [libjvm.so+0x4bf9fe] ConcurrentMarkSweepGeneration::ConcurrentMarkSweepGeneration(ReservedSpace, unsigned long, int, CardTableRS*, bool, FreeBlockDictionary::DictionaryChoice)+0x2e
V [libjvm.so+0x5f11e2] GenerationSpec::init(ReservedSpace, int, GenRemSet*)+0x182
V [libjvm.so+0x5dc2cf] GenCollectedHeap::initialize()+0x20f
V [libjvm.so+0xaa3faa] Universe::initialize_heap()+0x16a
V [libjvm.so+0xaa4293] universe_init()+0x33
V [libjvm.so+0x6350e0] init_globals()+0x50
V [libjvm.so+0xa869f9] Threads::create_vm(JavaVMInitArgs*, bool*)+0x409
V [libjvm.so+0x6dca8f] JNI_CreateJavaVM+0x4f
C [libjli.so+0x7ee4] JavaMain+0x84
C [libpthread.so.0+0x7ea5] start_thread+0xc5
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
Other Threads:
=>0x00007f2e60011000 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=126116, stack(0x00007f2e68f6e000,0x00007f2e6906e000)]
VM state:not at safepoint (not fully initialized)
VM Mutex/Monitor currently owned by a thread: None
heap address: 0x0000000310000000, size: 19200 MB, Compressed Oops mode: Non-zero based:0x000000030ffff000, Oop shift amount: 3
Narrow klass base: 0x0000000000000000, Narrow klass shift: 0
GC Heap History (0 events):
No events
Deoptimization events (0 events):
No events
Classes redefined (0 events):
No events
Internal exceptions (0 events):
No events
Events (0 events):
No events
Dynamic libraries:
310000000-4a0000000 rw-p 00000000 00:00 0
562462ed8000-562462ed9000 r-xp 00000000 fd:00 67190150 /usr/local/java/bin/java
5624630d8000-5624630d9000 r--p 00000000 fd:00 67190150 /usr/local/java/bin/java
5624630d9000-5624630da000 rw-p 00001000 fd:00 67190150 /usr/local/java/bin/java
562463ee9000-562463f0a000 rw-p 00000000 00:00 0 [heap]
7f2e44000000-7f2e44021000 rw-p 00000000 00:00 0
7f2e44021000-7f2e48000000 ---p 00000000 00:00 0
7f2e49a7f000-7f2e4a6ff000 rw-p 00000000 00:00 0
7f2e4a6ff000-7f2e4bfff000 ---p 00000000 00:00 0
7f2e4bfff000-7f2e4c000000 rw-p 00000000 00:00 0
7f2e4c000000-7f2e4c021000 rw-p 00000000 00:00 0
7f2e4c021000-7f2e50000000 ---p 00000000 00:00 0
7f2e51000000-7f2e51270000 rwxp 00000000 00:00 0
7f2e51270000-7f2e60000000 ---p 00000000 00:00 0
7f2e60000000-7f2e60040000 rw-p 00000000 00:00 0
7f2e60040000-7f2e64000000 ---p 00000000 00:00 0
7f2e65c89000-7f2e66121000 rw-p 00000000 00:00 0
7f2e66121000-7f2e66122000 ---p 00000000 00:00 0
7f2e66122000-7f2e66222000 rw-p 00000000 00:00 0
7f2e66222000-7f2e66223000 ---p 00000000 00:00 0
7f2e66223000-7f2e6632d000 rw-p 00000000 00:00 0
7f2e6632d000-7f2e666e3000 ---p 00000000 00:00 0
7f2e666e3000-7f2e666fe000 r-xp 00000000 fd:00 3616483 /usr/local/java/jre/lib/amd64/libzip.so
7f2e666fe000-7f2e668fd000 ---p 0001b000 fd:00 3616483 /usr/local/java/jre/lib/amd64/libzip.so
7f2e668fd000-7f2e668fe000 r--p 0001a000 fd:00 3616483 /usr/local/java/jre/lib/amd64/libzip.so
7f2e668fe000-7f2e668ff000 rw-p 0001b000 fd:00 3616483 /usr/local/java/jre/lib/amd64/libzip.so
7f2e668ff000-7f2e6690b000 r-xp 00000000 fd:00 949467 /usr/lib64/libnss_files-2.17.so
7f2e6690b000-7f2e66b0a000 ---p 0000c000 fd:00 949467 /usr/lib64/libnss_files-2.17.so
7f2e66b0a000-7f2e66b0b000 r--p 0000b000 fd:00 949467 /usr/lib64/libnss_files-2.17.so
7f2e66b0b000-7f2e66b0c000 rw-p 0000c000 fd:00 949467 /usr/lib64/libnss_files-2.17.so
7f2e66b0c000-7f2e66b12000 rw-p 00000000 00:00 0
7f2e66b12000-7f2e66b3e000 r-xp 00000000 fd:00 3616481 /usr/local/java/jre/lib/amd64/libjava.so
7f2e66b3e000-7f2e66d3e000 ---p 0002c000 fd:00 3616481 /usr/local/java/jre/lib/amd64/libjava.so
7f2e66d3e000-7f2e66d3f000 r--p 0002c000 fd:00 3616481 /usr/local/java/jre/lib/amd64/libjava.so
7f2e66d3f000-7f2e66d41000 rw-p 0002d000 fd:00 3616481 /usr/local/java/jre/lib/amd64/libjava.so
7f2e66d41000-7f2e66d4e000 r-xp 00000000 fd:00 3609083 /usr/local/java/jre/lib/amd64/libverify.so
7f2e66d4e000-7f2e66f4d000 ---p 0000d000 fd:00 3609083 /usr/local/java/jre/lib/amd64/libverify.so
7f2e66f4d000-7f2e66f4f000 r--p 0000c000 fd:00 3609083 /usr/local/java/jre/lib/amd64/libverify.so
7f2e66f4f000-7f2e66f50000 rw-p 0000e000 fd:00 3609083 /usr/local/java/jre/lib/amd64/libverify.so
7f2e66f50000-7f2e66f57000 r-xp 00000000 fd:00 1323448 /usr/lib64/librt-2.17.so
7f2e66f57000-7f2e67156000 ---p 00007000 fd:00 1323448 /usr/lib64/librt-2.17.so
7f2e67156000-7f2e67157000 r--p 00006000 fd:00 1323448 /usr/lib64/librt-2.17.so
7f2e67157000-7f2e67158000 rw-p 00007000 fd:00 1323448 /usr/lib64/librt-2.17.so
7f2e67158000-7f2e67259000 r-xp 00000000 fd:00 556070 /usr/lib64/libm-2.17.so
7f2e67259000-7f2e67458000 ---p 00101000 fd:00 556070 /usr/lib64/libm-2.17.so
7f2e67458000-7f2e67459000 r--p 00100000 fd:00 556070 /usr/lib64/libm-2.17.so
7f2e67459000-7f2e6745a000 rw-p 00101000 fd:00 556070 /usr/lib64/libm-2.17.so
7f2e6745a000-7f2e68152000 r-xp 00000000 fd:00 37055050 /usr/local/java/jre/lib/amd64/server/libjvm.so
7f2e68152000-7f2e68351000 ---p 00cf8000 fd:00 37055050 /usr/local/java/jre/lib/amd64/server/libjvm.so
7f2e68351000-7f2e683e7000 r--p 00cf7000 fd:00 37055050 /usr/local/java/jre/lib/amd64/server/libjvm.so
7f2e683e7000-7f2e68418000 rw-p 00d8d000 fd:00 37055050 /usr/local/java/jre/lib/amd64/server/libjvm.so
7f2e68418000-7f2e68453000 rw-p 00000000 00:00 0
7f2e68453000-7f2e68617000 r-xp 00000000 fd:00 15671 /usr/lib64/libc-2.17.so
7f2e68617000-7f2e68816000 ---p 001c4000 fd:00 15671 /usr/lib64/libc-2.17.so
7f2e68816000-7f2e6881a000 r--p 001c3000 fd:00 15671 /usr/lib64/libc-2.17.so
7f2e6881a000-7f2e6881c000 rw-p 001c7000 fd:00 15671 /usr/lib64/libc-2.17.so
7f2e6881c000-7f2e68821000 rw-p 00000000 00:00 0
7f2e68821000-7f2e68823000 r-xp 00000000 fd:00 524643 /usr/lib64/libdl-2.17.so
7f2e68823000-7f2e68a23000 ---p 00002000 fd:00 524643 /usr/lib64/libdl-2.17.so
7f2e68a23000-7f2e68a24000 r--p 00002000 fd:00 524643 /usr/lib64/libdl-2.17.so
7f2e68a24000-7f2e68a25000 rw-p 00003000 fd:00 524643 /usr/lib64/libdl-2.17.so
7f2e68a25000-7f2e68a3c000 r-xp 00000000 fd:00 73181815 /usr/local/java/lib/amd64/jli/libjli.so
7f2e68a3c000-7f2e68c3b000 ---p 00017000 fd:00 73181815 /usr/local/java/lib/amd64/jli/libjli.so
7f2e68c3b000-7f2e68c3c000 r--p 00016000 fd:00 73181815 /usr/local/java/lib/amd64/jli/libjli.so
7f2e68c3c000-7f2e68c3d000 rw-p 00017000 fd:00 73181815 /usr/local/java/lib/amd64/jli/libjli.so
7f2e68c3d000-7f2e68c54000 r-xp 00000000 fd:00 1323444 /usr/lib64/libpthread-2.17.so
7f2e68c54000-7f2e68e53000 ---p 00017000 fd:00 1323444 /usr/lib64/libpthread-2.17.so
7f2e68e53000-7f2e68e54000 r--p 00016000 fd:00 1323444 /usr/lib64/libpthread-2.17.so
7f2e68e54000-7f2e68e55000 rw-p 00017000 fd:00 1323444 /usr/lib64/libpthread-2.17.so
7f2e68e55000-7f2e68e59000 rw-p 00000000 00:00 0
7f2e68e59000-7f2e68e7b000 r-xp 00000000 fd:00 1323429 /usr/lib64/ld-2.17.so
7f2e68f65000-7f2e68f6d000 rw-s 00000000 fd:00 73176382 /tmp/hsperfdata_polarx/126113
7f2e68f6d000-7f2e68f71000 ---p 00000000 00:00 0
7f2e68f71000-7f2e69072000 rw-p 00000000 00:00 0
7f2e69074000-7f2e69078000 rw-p 00000000 00:00 0
7f2e69078000-7f2e69079000 r--p 00000000 00:00 0
7f2e69079000-7f2e6907a000 rw-p 00000000 00:00 0
7f2e6907a000-7f2e6907b000 r--p 00021000 fd:00 1323429 /usr/lib64/ld-2.17.so
7f2e6907b000-7f2e6907c000 rw-p 00022000 fd:00 1323429 /usr/lib64/ld-2.17.so
7f2e6907c000-7f2e6907d000 rw-p 00000000 00:00 0
7ffc68aab000-7ffc68ad1000 rw-p 00000000 00:00 0 [stack]
7ffc68b28000-7ffc68b2a000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
VM Arguments:
jvm_args: -Xms19200m -Xmx19200m -Xss1m -DtaskName=Final -Dlogback.configurationFile=/home/polarx/polardbx-cdc/bin/..//conf/logback.xml -XX:PermSize=m -XX:MaxPermSize=m -XX:+UseConcMarkSweepGC -XX:-UseAdaptiveSizePolicy -XX:SurvivorRatio=2 -XX:NewRatio=2 -XX:-OmitStackTraceInFastThrow -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true -Dmemory=19200 -Djava.util.prefs.systemRoot=/home/polarx/.java -Djava.util.prefs.userRoot=/home/polarx/.java/.userPrefs -Dfile.encoding=UTF-8 -Dcdc.home.dir=/home/polarx/polardbx-cdc/bin/../ -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/polarx/logs -XX:+CrashOnOutOfMemoryError -XX:ErrorFile=/home/polarx/hs_err_pid%p.log
java_command: com.aliyun.polardbx.binlog.TaskBootStrap taskName=Final
java_class_path (initial): /home/polarx/polardbx-cdc/bin/..//conf::/home/polarx/polardbx-cdc/bin/..//lib/activation-1.1.jar:/home/polarx/polardbx-cdc/bin/..//lib/aliyun-java-sdk-core-4.5.10.jar:/home/polarx/polardbx-cdc/bin/..//lib/aliyun-java-sdk-kms-2.11.0.jar:/home/polarx/polardbx-cdc/bin/..//lib/aliyun-java-sdk-ram-3.1.0.jar:/home/polarx/polardbx-cdc/bin/..//lib/aliyun-sdk-oss-3.16.0.jar:/home/polarx/polardbx-cdc/bin/..//lib/annotations-13.0.jar:/home/polarx/polardbx-cdc/bin/..//lib/aviator-5.3.0.jar:/home/polarx/polardbx-cdc/bin/..//lib/aws-java-sdk-core-1.11.655.jar:/home/polarx/polardbx-cdc/bin/..//lib/aws-java-sdk-kms-1.11.655.jar:/home/polarx/polardbx-cdc/bin/..//lib/aws-java-sdk-s3-1.11.655.jar:/home/polarx/polardbx-cdc/bin/..//lib/checker-qual-3.5.0.jar:/home/polarx/polardbx-cdc/bin/..//lib/commons-beanutils-1.9.3.jar:/home/polarx/polardbx-cdc/bin/..//lib/commons-codec-1.14.jar:/home/polarx/polardbx-cdc/bin/..//lib/commons-collections-3.2.2.jar:/home/polarx/polardbx-cdc/bin/..//lib/commons-dbutils-1.7.jar:/home/polarx/polardbx-cdc/bin/..//lib/commons-io-2.7.jar:/home/polarx/polardbx-cdc/bin/..//lib/commons-lang-2.6.jar:/home/polarx/polardbx-cdc/bin/..//lib/commons-lang3-3.8.1.jar:/home/polarx/polardbx-cdc/bin/..//lib/commons-logging-1.2.jar:/home/polarx/polardbx-cdc/bin/..//lib/commons-math-2.2.jar:/home/polarx/polardbx-cdc/bin/..//lib/commons-text-1.10.0.jar:/home/polarx/polardbx-cdc/bin/..//lib/disruptor-3.4.2.jar:/home/polarx/polardbx-cdc/bin/..//lib/dom4j-1.6.1.jar:/home/polarx/polardbx-cdc/bin/..//lib/druid-1.0.25.2018030201.jar:/home/polarx/polardbx-cdc/bin/..//lib/error_prone_annotations-2.3.4.jar:/home/polarx/polardbx-cdc/bin/..//lib/failureaccess-1.0.1.jar:/home/polarx/polardbx-cdc/bin/..//lib/fastjson-1.2.50_noneautotype.jar:/home/polarx/polardbx-cdc/bin/..//lib/flyway-core-7.3.0.jar:/home/polarx/polardbx-cdc/bin/..//lib/grpc-api-1.30.0.jar:/home/polarx/polardbx-cdc/bin/..//lib/grpc-context-1.30.0.jar:/home/polarx/polardbx-cdc/bin/..//lib/grpc
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=/usr/local/java
PATH=/usr/local/java/bin:/usr/local/apache-maven-3.6.3/bin:/usr/local/java/bin:/usr/local/java/jre/bin:/usr/local/java/bin:/usr/local/apache-maven-3.6.3/bin:/usr/local/java/bin:/usr/local/java/jre/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
LD_LIBRARY_PATH=/home/polarx/polardbx-cdc/bin/..//lib/native:/home/polarx/polardbx-cdc/bin/..//lib/native:
SHELL=/bin/bash
修改polardbx_meta_db_polardbx数据库里binlog_task_config表的mem列：