Resource Controller Limiter may not work well

[ShuNing]

What did you do?

Run the test added in #8434 . my env 72 cores:

Concurrency	reserveN	limit	Need Wait	Actually QPS	Actually RU_COST	Pass
10000	1000	4000000	0s	1487846.00	1487846000.00	❌
5000	1000	4000000	1.249s	3999.00	3999000.00	✅
3840	1000	4000000	0s	1419048.00	1419048000.00	❌
1000	1000	4000000	0s	1420002.00	1420002000.00	❌
1000	200	4000000	0s	1474321.00	294864200.00	❌
1000	5000	4000000	1.252s	801.00	4005000.00	✅
10000	50	400000	0s	1375128.00	68756400.00	❌
5000	50	400000	0s	1353481.00	67674050.00	❌
3840	50	400000	0s	1428777.00	71438850.00	❌
1000	50	400000	0s	1406483.00	70324150.00	❌
500	50	400000	0s	1548921.00	77446050.00	❌
200	50	400000	24.015ms	8000.00	400000.00	✅
100	50	400000	12.179ms	8001.00	400050.00	✅
1000	10	400000	0s	1452066.00	14520660.00	❌
1000	250	400000	624.087ms	1599.00	399750.00	✅
10000	500	400000	12.514s	800.00	400000.00	✅
5000	500	400000	6.257s	799.00	399500.00	✅
3840	500	400000	4.805s	799.00	399500.00	✅
1000	500	400000	1.251s	800.00	400000.00	✅
1000	100	200000	500.348ms	2003.00	200300.00	✅
1000	100	400000	249.302ms	3998.00	399800.00	✅

[ShuNing]

This is caused by:

• In high concurrency scenarios, time may appear reversed because the now value passed from outside. high mutext completion leading to more non-sequential execution orders.
• Time reversal allows for advancing more tokens, which can cause the issue. even result in no limit for controller.

[Ting]

/report customer