强化学习：资产配置
ChenglongChen opened this issue · comments
1. Framekwork
- 开发RL框架，包括RLTradeStrategy（集成state/action interpreter，reward，policy，backtest），simulator等
- 开发topkdropout strategy，学习n drop参数
- 开发weight strategy，直接学习仓位管理
- 开发topk strategy，直接学习topk持仓
- 开发TopkDropoutSelectionStrategy，直接学习n drop的sell和buy
- Only tradable集成到weightstrategy：https://github.com/microsoft/qlib/blob/main/qlib/contrib/strategy/signal_strategy.py#L150
- Only tradable集成到topkstrategy：https://github.com/microsoft/qlib/blob/main/qlib/contrib/strategy/signal_strategy.py#L150
- fix start time issue, when start time is not tradable
2. Feature
- add historical features
- add position features
- add n_step historical features
- support Alpha158
3. Model
- 使用PPO学习n drop参数，action空间太小，效果不显著
- 使用DDPG/TD3学习rerank & weighting
- 使用MultiBinary & PPO直接输出topk，权重使用均仓
- 使用imitation learning，学习lgbm的输出，同时学习actor：https://github.com/thu-ml/tianshou/blob/master/tianshou/policy/imitation/td3_bc.py
- add transformer in MetaNet
- add different rewards
4. Trade
- 优化模型效果，超越目前的baseline
- 重构tradestrategy，以支持实盘
1. Overall Performance
| Reward | Strategy | Baseline | MetaPPO | MetaDDPG | MetaTD3 | MetaSAC |
|---|---|---|---|---|---|---|
| InformationRatioReward | TopkDropoutSignalStrategy | 1.656972 | 1.971290 | 1.929209 | 1.179975 | |
| InformationRatioReward | WeightStrategy | 0.785682 | 1.338792 | 0.867902 | ||
| ExessReturnReward | TopkDropoutSignalStrategy | 0.263561 | 0.228271 | 0.191836 | ||
| ExessReturnReward | WeightStrategy | 0.110891 | 0.186833 | 0.167790 |
2. Learning Curve
2.1 InformationRatioReward
a) TopkDropoutSignalStrategy+MetaPPO
b) TopkDropoutSignalStrategy+MetaTD3
c) TopkDropoutSignalStrategy+MetaSAC
d) WeightStrategy+MetaPPO
e) WeightStrategy+MetaTD3
f) WeightStrategy+MetaSAC
2.2 ExessReturnReward
a) TopkDropoutSignalStrategy+MetaPPO
b) TopkDropoutSignalStrategy+MetaTD3
c) TopkDropoutSignalStrategy+MetaSAC
d) WeightStrategy+MetaPPO
e) WeightStrategy+MetaTD3
f) WeightStrategy+MetaSAC
Overall Performance
| Reward | Strategy | Baseline | MetaPPO | MetaSAC |
|---|---|---|---|---|
| EpisodeInformationRatioReward | TopkDropoutSelectionStrategy | 1.644155 | 1.884990 | 1.415276 |
| EpisodeInformationRatioReward | TopkDropoutDynamicStrategy | 1.644155 | 2.143160 |
Learning Curve
