add index blocked with error "not found instance” after injection pd leader io delay

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、enable tidb_enable_dist_task and global sort
2、run sysbench
3、add index for sbtest1 and sbtest2 concurrently
4、inject pd leader io delay 5s

operator logs：
the status of ddl job is not synced after 1h0m0s (now: 2024-05-14 21:05:56, jobId: 546, job type: add index /* ingest cloud /, state: running)
operatorLogs:
[2024-05-14 20:05:50] ###### start adding index
ALTER TABLE sbtest2 ADD INDEX index_test_1715688350713(c)
[2024-05-14 20:05:50] ###### wait for ddl job finish
[2024-05-14 21:05:56] ###### wait for ddl job finish timeout(1h0m0s)
select job_id, job_type, state from information_schema.ddl_jobs where query = 'ALTER TABLE sbtest2 ADD INDEX index_test_1715688350713(c)'
jobId: 546, job type: add index / ingest cloud */, state: running

2. What did you expect to see? (Required)

add index can success after fault recover

3. What did you see instead (Required)

add index blocked with error "not found instance”
[2024/05/14 23:31:22.997 +08:00] [INFO] [scheduler.go:264] ["schedule task meet err, reschedule it"] [task-id=78] [task-type=backfill] [allocated-slots=true] [error="not found instance"]

4. What is your TiDB version? (Required)

./tidb-server -V
Release Version: v8.1.0-alpha
Edition: Community
Git Commit Hash: 48a3626
Git Branch: heads/refs/tags/v8.1.0-alpha
UTC Build Time: 2024-05-14 07:09:37
GoVersion: go1.21.6
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2024-05-14T15:24:02.701+0800

[Lily2025]

/type bug
/assign ywqzzy
/severity major