"restore not finish since 244 items to be applied" tdengine api return error while restoring

Question

"restore not finish since 244 items to be applied" tdengine api return error while restoring

pangmailei opened this issue 2 months ago · comments

Sharla Spencer commented 2 months ago

Bug Description
"restore not finish since 244 items to be applied" tdengine api return error while restoring

To Reproduce
can't reproduce

Expected Behavior
block request while restoring rather than return error

Screenshots

Environment (please complete the following information):

OS: Debian 10
arm64 docker
TDengine Version: 3.0.4.0

Additional Context
{"log":"05/24 15:27:43.804997 00000103 MND trans:1801, perform insert action, row:0x7f3002830c stage:prepare, callfunc:1, startFunc:0\n","stream":"stdout","time":"2024-05-24T07:27:43.805410393Z"}
{"log":"05/24 15:27:43.805300 00000103 MND trans:1801, is proposed and post sem, seq:1\n","stream":"stdout","time":"2024-05-24T07:27:43.805993726Z"}
{"log":"05/24 15:27:43.805579 00000102 MND trans:1801, sync finished, createTime:1716535663790\n","stream":"stdout","time":"2024-05-24T07:27:43.80638281Z"}
{"log":"05/24 15:27:43.805880 00000102 MND trans:1801, prepare finished\n","stream":"stdout","time":"2024-05-24T07:27:43.806990351Z"}
{"log":"05/24 15:27:43.806191 00000102 MND trans:1801, continue to execute, stage:prepare createTime:1716535663790 leader:1\n","stream":"stdout","time":"2024-05-24T07:27:43.807292518Z"}
{"log":"05/24 15:27:43.806466 00000102 MND trans:1801, stage from prepare to redoAction\n","stream":"stdout","time":"2024-05-24T07:27:43.80763406Z"}
{"log":"05/24 15:27:43.806528 00000102 MND trans:1801, continue to execute, stage:redoAction createTime:1716535663790 leader:1\n","stream":"stdout","time":"2024-05-24T07:27:43.807862435Z"}
{"log":"05/24 15:27:43.806692 00000102 MND trans:1801, redoAction:0 write to sdb, type:stb status:creating\n","stream":"stdout","time":"2024-05-24T07:27:43.808163435Z"}
{"log":"05/24 15:27:43.806853 00000102 MND trans:1801, redoAction:1 is sent, msgType:vnode-create-stb numOfEps:1 inUse:0 ep:0-tdengine:6030\n","stream":"stdout","time":"2024-05-24T07:27:43.80830781Z"}
{"log":"05/24 15:27:43.807203 00000102 MND trans:1801, redoAction:2 is sent, msgType:vnode-create-stb numOfEps:1 inUse:0 ep:0-tdengine:6030\n","stream":"stdout","time":"2024-05-24T07:27:43.808510226Z"}
{"log":"05/24 15:27:43.807797 00000102 MND trans:1801, 1 of 3 actions executed\n","stream":"stdout","time":"2024-05-24T07:27:43.808774476Z"}
{"log":"05/24 15:27:43.808145 00000102 MND trans:1801, stage keep on redoAction since Action in progress\n","stream":"stdout","time":"2024-05-24T07:27:43.809013935Z"}
{"log":"05/24 15:27:43.810288 00000126 VND ERROR vgId:2, msg:0x7f90002d88 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00036\n","stream":"stdout","time":"2024-05-24T07:27:43.811169643Z"}
{"log":"05/24 15:27:43.810621 00000126 SYN vgId:2, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:43.811335601Z"}
{"log":"05/24 15:27:43.812677 00000112 VND ERROR vgId:3, msg:0x7f9c00df48 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00037\n","stream":"stdout","time":"2024-05-24T07:27:43.813213351Z"}
{"log":"05/24 15:27:43.813745 00000112 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:43.814295435Z"}
{"log":"05/24 15:27:43.815134 00000126 VND ERROR vgId:2, msg:0x7f90002e58 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00036\n","stream":"stdout","time":"2024-05-24T07:27:43.815621351Z"}
{"log":"05/24 15:27:43.816487 00000126 SYN vgId:2, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:43.817194893Z"}
{"log":"05/24 15:27:43.818609 00000112 VND ERROR vgId:3, msg:0x7f9c004c98 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00037\n","stream":"stdout","time":"2024-05-24T07:27:43.81921381Z"}
{"log":"05/24 15:27:43.819837 00000112 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:43.820377851Z"}
{"log":"05/24 15:27:43.829257 00000126 VND ERROR vgId:2, msg:0x7f90002f28 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00036\n","stream":"stdout","time":"2024-05-24T07:27:43.829801893Z"}
{"log":"05/24 15:27:43.830370 00000126 SYN vgId:2, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:43.830880476Z"}
{"log":"05/24 15:27:43.835475 00000112 VND ERROR vgId:3, msg:0x7f9c004d68 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00037\n","stream":"stdout","time":"2024-05-24T07:27:43.83603306Z"}
{"log":"05/24 15:27:43.836521 00000112 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:43.837251935Z"}
{"log":"05/24 15:27:43.861340 00000112 VND ERROR vgId:3, msg:0x7f9c004e38 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00037\n","stream":"stdout","time":"2024-05-24T07:27:43.861837101Z"}
{"log":"05/24 15:27:43.862289 00000112 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:43.862754393Z"}
{"log":"05/24 15:27:43.868148 00000126 VND ERROR vgId:2, msg:0x7f90002ff8 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00036\n","stream":"stdout","time":"2024-05-24T07:27:43.868715476Z"}
{"log":"05/24 15:27:43.869103 00000126 SYN vgId:2, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:43.869648226Z"}
{"log":"05/24 15:27:43.907369 00000112 VND ERROR vgId:3, msg:0x7f9c004f08 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00037\n","stream":"stdout","time":"2024-05-24T07:27:43.908119935Z"}
{"log":"05/24 15:27:43.908665 00000112 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:43.909309351Z"}
{"log":"05/24 15:27:43.914219 00000126 VND ERROR vgId:2, msg:0x7f900030c8 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00036\n","stream":"stdout","time":"2024-05-24T07:27:43.914795893Z"}
{"log":"05/24 15:27:43.914836 00000126 SYN vgId:2, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:43.915498226Z"}
{"log":"05/24 15:27:43.996997 00000112 VND ERROR vgId:3, msg:0x7f9c004fd8 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00037\n","stream":"stdout","time":"2024-05-24T07:27:43.997548143Z"}
{"log":"05/24 15:27:43.998193 00000126 VND ERROR vgId:2, msg:0x7f90003198 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00036\n","stream":"stdout","time":"2024-05-24T07:27:43.998745143Z"}
{"log":"05/24 15:27:43.998748 00000126 SYN vgId:2, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:43.999247393Z"}
{"log":"05/24 15:27:43.998821 00000112 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:43.999482768Z"}
{"log":"05/24 15:27:44.160217 00000112 VND ERROR vgId:3, msg:0x7f9c0050a8 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00037\n","stream":"stdout","time":"2024-05-24T07:27:44.16066506Z"}
{"log":"05/24 15:27:44.160573 00000112 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:44.161279602Z"}
{"log":"05/24 15:27:44.162270 00000126 VND ERROR vgId:2, msg:0x7f90003268 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00036\n","stream":"stdout","time":"2024-05-24T07:27:44.162795685Z"}
{"log":"05/24 15:27:44.162386 00000126 SYN vgId:2, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:44.163033685Z"}
{"log":"05/24 15:27:44.482976 00000112 VND ERROR vgId:3, msg:0x7f9c005178 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00037\n","stream":"stdout","time":"2024-05-24T07:27:44.490346685Z"}
{"log":"05/24 15:27:44.483117 00000112 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:44.49051556Z"}
{"log":"05/24 15:27:44.484768 00000126 VND ERROR vgId:2, msg:0x7f90003338 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00036\n","stream":"stdout","time":"2024-05-24T07:27:44.490589643Z"}
{"log":"05/24 15:27:44.484863 00000126 SYN vgId:2, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:44.490652643Z"}
{"log":"05/24 15:27:44.564557 00000125 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:44.566380102Z"}
{"log":"05/24 15:27:44.646340 00000102 MND trans:1801, continue to execute, stage:redoAction createTime:1716535663790 leader:1\n","stream":"stdout","time":"2024-05-24T07:27:44.646931727Z"}
{"log":"05/24 15:27:44.647158 00000102 MND trans:1801, 1 of 3 actions executed\n","stream":"stdout","time":"2024-05-24T07:27:44.647566685Z"}
{"log":"05/24 15:27:44.647508 00000102 MND trans:1801, stage keep on redoAction since Action in progress\n","stream":"stdout","time":"2024-05-24T07:27:44.648077977Z"}
{"log":"05/24 15:27:45.126607 00000126 VND ERROR vgId:2, msg:0x7f90003338 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00036\n","stream":"stdout","time":"2024-05-24T07:27:45.127463352Z"}
{"log":"05/24 15:27:45.128044 00000126 SYN vgId:2, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:45.12868806Z"}
{"log":"05/24 15:27:45.133731 00000112 VND ERROR vgId:3, msg:0x7f9c005328 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00037\n","stream":"stdout","time":"2024-05-24T07:27:45.134492519Z"}
{"log":"05/24 15:27:45.135053 00000112 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:45.135700894Z"}
{"log":"05/24 15:27:45.566641 00000125 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:45.567305144Z"}
{"log":"05/24 15:27:46.130637 00000126 VND ERROR vgId:2, msg:0x7f90003338 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00036\n","stream":"stdout","time":"2024-05-24T07:27:46.131560561Z"}
{"log":"05/24 15:27:46.132144 00000126 SYN vgId:2, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:46.132986228Z"}
{"log":"05/24 15:27:46.141176 00000112 VND ERROR vgId:3, msg:0x7f9c005328 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00037\n","stream":"stdout","time":"2024-05-24T07:27:46.142301478Z"}
{"log":"05/24 15:27:46.141513 00000112 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:46.142504186Z"}
{"log":"05/24 15:27:46.569841 00000125 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:46.570878811Z"}
{"log":"05/24 15:27:46.650103 00000102 MND trans:1801, continue to execute, stage:redoAction createTime:1716535663790 leader:1\n","stream":"stdout","time":"2024-05-24T07:27:46.650823186Z"}
{"log":"05/24 15:27:46.650257 00000102 MND trans:1801, 1 of 3 actions executed\n","stream":"stdout","time":"2024-05-24T07:27:46.651039019Z"}
{"log":"05/24 15:27:46.650316 00000102 MND trans:1801, stage keep on redoAction since Action in progress\n","stream":"stdout","time":"2024-05-24T07:27:46.651167061Z"}
{"log":"05/24 15:27:47.135615 00000126 VND ERROR vgId:2, msg:0x7f90003928 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00036\n","stream":"stdout","time":"2024-05-24T07:27:47.136363311Z"}
{"log":"05/24 15:27:47.136599 00000126 SYN vgId:2, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:47.137272728Z"}
{"log":"05/24 15:27:47.143641 00000112 VND ERROR vgId:3, msg:0x7f9c005328 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00037\n","stream":"stdout","time":"2024-05-24T07:27:47.14419252Z"}
{"log":"05/24 15:27:47.143755 00000112 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:47.144366645Z"}
{"log":"05/24 15:27:47.572719 00000125 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:47.573516228Z"}
{"log":"05/24 15:27:48.139148 00000126 VND ERROR vgId:2, msg:0x7f90003928 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00036\n","stream":"stdout","time":"2024-05-24T07:27:48.139714145Z"}
{"log":"05/24 15:27:48.139248 00000126 SYN vgId:2, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:48.13987427Z"}
{"log":"05/24 15:27:48.145562 00000112 VND ERROR vgId:3, msg:0x7f9c005548 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00037\n","stream":"stdout","time":"2024-05-24T07:27:48.14631077Z"}
{"log":"05/24 15:27:48.146790 00000112 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:48.147741103Z"}
{"log":"05/24 15:27:48.574713 00000125 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:48.575462687Z"}
{"log":"05/24 15:27:48.653723 00000102 MND trans:1801, continue to execute, stage:redoAction createTime:1716535663790 leader:1\n","stream":"stdout","time":"2024-05-24T07:27:48.654664187Z"}
{"log":"05/24 15:27:48.653850 00000102 MND trans:1801, 1 of 3 actions executed\n","stream":"stdout","time":"2024-05-24T07:27:48.654826062Z"}
{"log":"05/24 15:27:48.653955 00000102 MND trans:1801, stage keep on redoAction since Action in progress\n","stream":"stdout","time":"2024-05-24T07:27:48.654966354Z"}
{"log":"05/24 15:27:49.141865 00000126 VND ERROR vgId:2, msg:0x7f90003ae8 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00036\n","stream":"stdout","time":"2024-05-24T07:27:49.142608229Z"}
{"log":"05/24 15:27:49.143190 00000126 SYN vgId:2, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:49.143887479Z"}
{"log":"05/24 15:27:49.150665 00000112 VND ERROR vgId:3, msg:0x7f9c005548 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00037\n","stream":"stdout","time":"2024-05-24T07:27:49.151507562Z"}
{"log":"05/24 15:27:49.152287 00000112 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:49.153053687Z"}
{"log":"05/24 15:27:49.578334 00000125 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:49.578854937Z"}
{"log":"05/24 15:27:50.146587 00000126 VND ERROR vgId:2, msg:0x7f9000aec8 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00036\n","stream":"stdout","time":"2024-05-24T07:27:50.147457646Z"}
{"log":"05/24 15:27:50.146689 00000126 SYN vgId:2, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:50.147624771Z"}
{"log":"05/24 15:27:50.155614 00000112 VND ERROR vgId:3, msg:0x7f9c005548 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00037\n","stream":"stdout","time":"2024-05-24T07:27:50.156427563Z"}
{"log":"05/24 15:27:50.156948 00000112 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:50.157751729Z"}
{"log":"05/24 15:27:50.660227 00000102 MND trans:1801, continue to execute, stage:redoAction createTime:1716535663790 leader:1\n","stream":"stdout","time":"2024-05-24T07:27:50.660800021Z"}
{"log":"05/24 15:27:50.661297 00000102 MND trans:1801, 1 of 3 actions executed\n","stream":"stdout","time":"2024-05-24T07:27:50.661793146Z"}
{"log":"05/24 15:27:50.662267 00000102 MND trans:1801, stage keep on redoAction since Action in progress\n","stream":"stdout","time":"2024-05-24T07:27:50.662778396Z"}
{"log":"05/24 15:27:51.149552 00000126 VND ERROR vgId:2, msg:0x7f9000aec8 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00036\n","stream":"stdout","time":"2024-05-24T07:27:51.150173813Z"}
{"log":"05/24 15:27:51.149954 00000126 SYN vgId:2, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:51.150968897Z"}
{"log":"05/24 15:27:51.160881 00000112 VND ERROR vgId:3, msg:0x7f9c0056f8 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00037\n","stream":"stdout","time":"2024-05-24T07:27:51.161595188Z"}
{"log":"05/24 15:27:51.161871 00000112 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:51.162542813Z"}
{"log":"05/24 15:27:52.154049 00000126 VND ERROR vgId:2, msg:0x7f9000aec8 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00036\n","stream":"stdout","time":"2024-05-24T07:27:52.15475898Z"}
{"log":"05/24 15:27:52.154128 00000126 SYN vgId:2, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:52.155060272Z"}
{"log":"05/24 15:27:52.164647 00000112 VND ERROR vgId:3, msg:0x7f9c0056f8 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00037\n","stream":"stdout","time":"2024-05-24T07:27:52.165274439Z"}
{"log":"05/24 15:27:52.166057 00000112 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:52.166579064Z"}
{"log":"05/24 15:27:52.671638 00000102 MND trans:1801, continue to execute, stage:redoAction createTime:1716535663790 leader:1\n","stream":"stdout","time":"2024-05-24T07:27:52.672441356Z"}
{"log":"05/24 15:27:52.671752 00000102 MND trans:1801, 1 of 3 actions executed\n","stream":"stdout","time":"2024-05-24T07:27:52.672602647Z"}
{"log":"05/24 15:27:52.671799 00000102 MND trans:1801, stage keep on redoAction since Action in progress\n","stream":"stdout","time":"2024-05-24T07:27:52.672681689Z"}
{"log":"05/24 15:27:53.157308 00000126 VND ERROR vgId:2, msg:0x7f9000b078 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00036\n","stream":"stdout","time":"2024-05-24T07:27:53.158110981Z"}
{"log":"05/24 15:27:53.159221 00000126 SYN vgId:2, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:53.160168689Z"}
{"log":"05/24 15:27:53.169783 00000112 VND ERROR vgId:3, msg:0x7f9c0056f8 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00037\n","stream":"stdout","time":"2024-05-24T07:27:53.170438564Z"}
{"log":"05/24 15:27:53.170839 00000112 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:53.171432273Z"}
{"log":"05/24 15:27:53.808255 00000138 SYN vgId:3, commit sync barrier. index:378261, term:2, type:sync-noop\n","stream":"stdout","time":"2024-05-24T07:27:53.809153273Z"}
{"log":"05/24 15:27:53.808445 00000138 SYN vgId:3, commit sync barrier. index:378269, term:3, type:sync-noop\n","stream":"stdout","time":"2024-05-24T07:27:53.809310773Z"}
{"log":"05/24 15:27:53.824605 00000138 VND vgId:3, restore not finish since 3898 items to be applied. commit-index:378269, applied-index:374371\n","stream":"stdout","time":"2024-05-24T07:27:53.825607065Z"}
{"log":"05/24 15:27:53.834871 00000138 VND vgId:3, restore not finish since 3780 items to be applied. commit-index:378269, applied-index:374489\n","stream":"stdout","time":"2024-05-24T07:27:53.835846606Z"}
{"log":"05/24 15:27:53.845391 00000138 VND vgId:3, restore not finish since 3670 items to be applied. commit-index:378269, applied-index:374599\n","stream":"stdout","time":"2024-05-24T07:27:53.846011773Z"}
{"log":"05/24 15:27:53.855642 00000138 VND vgId:3, restore not finish since 3545 items to be applied. commit-index:378269, applied-index:374724\n","stream":"stdout","time":"2024-05-24T07:27:53.856400356Z"}
{"log":"05/24 15:27:53.865935 00000138 VND vgId:3, restore not finish since 3418 items to be applied. commit-index:378269, applied-index:374851\n","stream":"stdout","time":"2024-05-24T07:27:53.866728273Z"}
{"log":"05/24 15:27:53.876179 00000138 VND vgId:3, restore not finish since 3296 items to be applied. commit-index:378269, applied-index:374973\n","stream":"stdout","time":"2024-05-24T07:27:53.876887606Z"}
{"log":"05/24 15:27:53.886407 00000138 VND vgId:3, restore not finish since 3170 items to be applied. commit-index:378269, applied-index:375099\n","stream":"stdout","time":"2024-05-24T07:27:53.887286398Z"}
{"log":"05/24 15:27:53.896628 00000138 VND vgId:3, restore not finish since 3043 items to be applied. commit-index:378269, applied-index:375226\n","stream":"stdout","time":"2024-05-24T07:27:53.897532648Z"}
{"log":"05/24 15:27:53.906884 00000138 VND vgId:3, restore not finish since 2925 items to be applied. commit-index:378269, applied-index:375344\n","stream":"stdout","time":"2024-05-24T07:27:53.907638315Z"}
{"log":"05/24 15:27:53.917178 00000138 VND vgId:3, restore not finish since 2804 items to be applied. commit-index:378269, applied-index:375465\n","stream":"stdout","time":"2024-05-24T07:27:53.918060731Z"}
{"log":"05/24 15:27:53.927507 00000138 VND vgId:3, restore not finish since 2684 items to be applied. commit-index:378269, applied-index:375585\n","stream":"stdout","time":"2024-05-24T07:27:53.928313981Z"}
{"log":"05/24 15:27:53.937802 00000138 VND vgId:3, restore not finish since 2564 items to be applied. commit-index:378269, applied-index:375705\n","stream":"stdout","time":"2024-05-24T07:27:53.938571023Z"}
{"log":"05/24 15:27:53.948510 00000138 VND vgId:3, restore not finish since 2441 items to be applied. commit-index:378269, applied-index:375828\n","stream":"stdout","time":"2024-05-24T07:27:53.949106315Z"}
{"log":"05/24 15:27:53.958738 00000138 VND vgId:3, restore not finish since 2318 items to be applied. commit-index:378269, applied-index:375951\n","stream":"stdout","time":"2024-05-24T07:27:53.959462523Z"}
{"log":"05/24 15:27:53.968974 00000138 VND vgId:3, restore not finish since 2197 items to be applied. commit-index:378269, applied-index:376072\n","stream":"stdout","time":"2024-05-24T07:27:53.96949994Z"}
{"log":"05/24 15:27:53.979286 00000138 VND vgId:3, restore not finish since 2074 items to be applied. commit-index:378269, applied-index:376195\n","stream":"stdout","time":"2024-05-24T07:27:53.979975731Z"}
{"log":"05/24 15:27:53.989513 00000138 VND vgId:3, restore not finish since 1967 items to be applied. commit-index:378269, applied-index:376302\n","stream":"stdout","time":"2024-05-24T07:27:53.990182315Z"}
{"log":"05/24 15:27:53.999849 00000138 VND vgId:3, restore not finish since 1838 items to be applied. commit-index:378269, applied-index:376431\n","stream":"stdout","time":"2024-05-24T07:27:54.000702731Z"}
{"log":"05/24 15:27:54.011139 00000138 VND vgId:3, restore not finish since 1714 items to be applied. commit-index:378269, applied-index:376555\n","stream":"stdout","time":"2024-05-24T07:27:54.011640231Z"}
{"log":"05/24 15:27:54.022026 00000138 VND vgId:3, restore not finish since 1607 items to be applied. commit-index:378269, applied-index:376662\n","stream":"stdout","time":"2024-05-24T07:27:54.024468023Z"}
{"log":"05/24 15:27:54.032227 00000138 VND vgId:3, restore not finish since 1489 items to be applied. commit-index:378269, applied-index:376780\n","stream":"stdout","time":"2024-05-24T07:27:54.033162898Z"}
{"log":"05/24 15:27:54.042471 00000138 VND vgId:3, restore not finish since 1357 items to be applied. commit-index:378269, applied-index:376912\n","stream":"stdout","time":"2024-05-24T07:27:54.043533981Z"}
{"log":"05/24 15:27:54.052832 00000138 VND vgId:3, restore not finish since 1281 items to be applied. commit-index:378269, applied-index:376988\n","stream":"stdout","time":"2024-05-24T07:27:54.053456481Z"}
{"log":"05/24 15:27:54.065046 00000138 VND vgId:3, restore not finish since 1145 items to be applied. commit-index:378269, applied-index:377124\n","stream":"stdout","time":"2024-05-24T07:27:54.06779744Z"}
{"log":"05/24 15:27:54.076027 00000138 VND vgId:3, restore not finish since 1008 items to be applied. commit-index:378269, applied-index:377261\n","stream":"stdout","time":"2024-05-24T07:27:54.079387981Z"}
{"log":"05/24 15:27:54.087097 00000138 VND vgId:3, restore not finish since 873 items to be applied. commit-index:378269, applied-index:377396\n","stream":"stdout","time":"2024-05-24T07:27:54.088349148Z"}
{"log":"05/24 15:27:54.098034 00000138 VND vgId:3, restore not finish since 737 items to be applied. commit-index:378269, applied-index:377532\n","stream":"stdout","time":"2024-05-24T07:27:54.101275815Z"}
{"log":"05/24 15:27:54.108564 00000138 VND vgId:3, restore not finish since 626 items to be applied. commit-index:378269, applied-index:377643\n","stream":"stdout","time":"2024-05-24T07:27:54.111168273Z"}
{"log":"05/24 15:27:54.122154 00000138 VND vgId:3, restore not finish since 478 items to be applied. commit-index:378269, applied-index:377791\n","stream":"stdout","time":"2024-05-24T07:27:54.124211898Z"}
{"log":"05/24 15:27:54.133002 00000138 VND vgId:3, restore not finish since 358 items to be applied. commit-index:378269, applied-index:377911\n","stream":"stdout","time":"2024-05-24T07:27:54.134097648Z"}
{"log":"05/24 15:27:54.143241 00000138 VND vgId:3, restore not finish since 244 items to be applied. commit-index:378269, applied-index:378025\n","stream":"stdout","time":"2024-05-24T07:27:54.145313106Z"}
{"log":"05/24 15:27:54.153612 00000138 VND vgId:3, restore not finish since 130 items to be applied. commit-index:378269, applied-index:378139\n","stream":"stdout","time":"2024-05-24T07:27:54.154938398Z"}
{"log":"05/24 15:27:54.162517 00000126 VND ERROR vgId:2, msg:0x7f9000b078 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00036\n","stream":"stdout","time":"2024-05-24T07:27:54.163705606Z"}
{"log":"05/24 15:27:54.162619 00000126 SYN vgId:2, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:54.163955565Z"}
{"log":"05/24 15:27:54.163972 00000138 VND vgId:3, restore not finish since 21 items to be applied. commit-index:378269, applied-index:378248\n","stream":"stdout","time":"2024-05-24T07:27:54.164482315Z"}
{"log":"05/24 15:27:54.165009 00000102 MND trans:1801, redoAction:1 response is received, code:0x80000914, accept:0x80000619 retry:0x8000061a\n","stream":"stdout","time":"2024-05-24T07:27:54.16650619Z"}
{"log":"05/24 15:27:54.165125 00000102 MND trans:1801, continue to execute, stage:redoAction createTime:1716535663790 leader:1\n","stream":"stdout","time":"2024-05-24T07:27:54.166670981Z"}
{"log":"05/24 15:27:54.165249 00000102 MND trans:1801, 2 of 3 actions executed\n","stream":"stdout","time":"2024-05-24T07:27:54.166752065Z"}
{"log":"05/24 15:27:54.165280 00000102 MND trans:1801, stage keep on redoAction since Action in progress\n","stream":"stdout","time":"2024-05-24T07:27:54.166802815Z"}
{"log":"05/24 15:27:54.174655 00000112 VND ERROR vgId:3, msg:0x7f9c0058a8 failed to process since restore not finished, type:vnode-create-stb, gtid:0x6c40017f9ca40029:0x569f3a97f9c00037\n","stream":"stdout","time":"2024-05-24T07:27:54.176066731Z"}
{"log":"05/24 15:27:54.174772 00000112 SYN vgId:3, sync get retry epset numOfEps:1 inUse:0\n","stream":"stdout","time":"2024-05-24T07:27:54.176768481Z"}
{"log":"05/24 15:27:54.174991 00000138 VND vgId:3, no items to be applied, restore finish\n","stream":"stdout","time":"2024-05-24T07:27:54.176872023Z"}
{"log":"05/24 15:27:54.175087 00000138 VND vgId:3, sync restore finished\n","stream":"stdout","time":"2024-05-24T07:27:54.177021648Z"}
{"log":"05/24 15:27:54.175114 00000138 SYN vgId:3, restore finished. term:3, log buffer: [378013 378269 378269, 378270)\n","stream":"stdout","time":"2024-05-24T07:27:54.177087856Z"}
{"log":"05/24 15:27:54.175983 00000102 MND trans:1801, redoAction:2 response is received, code:0x80000914, accept:0x80000619 retry:0x8000061a\n","stream":"stdout","time":"2024-05-24T07:27:54.177447773Z"}
{"log":"05/24 15:27:54.176098 00000102 MND trans:1801, continue to execute, stage:redoAction createTime:1716535663790 leader:1\n","stream":"stdout","time":"2024-05-24T07:27:54.177583398Z"}
{"log":"05/24 15:27:54.176139 00000102 MND ERROR trans:1801, all 3 actions executed, code:0x914\n","stream":"stdout","time":"2024-05-24T07:27:54.177660398Z"}
{"log":"05/24 15:27:54.176161 00000102 MND trans:1801, redoAction:1 execute status is reset\n","stream":"stdout","time":"2024-05-24T07:27:54.177710565Z"}
{"log":"05/24 15:27:54.176178 00000102 MND trans:1801, redoAction:2 execute status is reset\n","stream":"stdout","time":"2024-05-24T07:27:54.177752565Z"}

Yu Chen · Answer 1 · Tue May 28 2024 13:35:47 GMT+0800 (China Standard Time)

等这个244变成 0，结束数据库就恢复了

liuchenn · Answer 2 · Wed Jul 03 2024 15:22:01 GMT+0800 (China Standard Time)

请问下，什么情况下会触发这个restore