If I add attention from it. Will it work better?

[baipengfeifei]

If I add attention from it. Will it work better?

[Zhihao Lin]

We didn't try it, but I think it's a promising direction to combine the attention in the router, to make it more effective.