Double Q for DIAYN
josiahls opened this issue · comments
Hi,
Forgive me if this is already explained/implemented (part time grad student, pretty new to this):
On reading through the DIAYN code/initial reading of the paper, it seems to not use the double q that is present in SAC. What is the reason for this?
I was also surprised that it seems that DIAYN completely overrides the actor/critic training functions of SAC as opposed to extending them.
The reason for not using the double Q is that the initially published version of SAC did not use it and DIAYN was implemented on top of the first version of SAC. I believe the complete override was done just because it was easier to isolate the two parts of the code, as the implementation was not meant to be used as a library but rather just an example implementation for research purposes. @haarnoja, please correct me if I'm wrong.
This is a great answer, thanks! Wasn't expecting someone to reply so fast. Seems like there is some opportunity for experimenting to see if double q can improve performance.
The overlapping/overriding code part of the question also makes sense. I'll go ahead and close this since my question is answered.
Glad that it helped! Definitely worth trying the double-Q function out. From my recent experience, the minimum over the ensemble really improves the performance in cases where the reward is dense and the task doesn't require much exploration (such as in the standard gym locomotion environments). It does seem to really degrade the performance in some cases, especially in sparse tasks where exploration is necessary, however. I don't have a clear intuition of how this would work for DIAYN though. It would be great to hear if you find anything interesting!
@m-smith, this reminds me that we were just discussing the double Q in the DIAYN context. Did you happen to ever try it out?
I've not compared the two yet! I've only ever used the double Q version. Can confirm that DIAYN still learns with the double Q--as to how it affects the skill structure, I'll look into this soon and be sure to update here when I do!