cross-attention maps are not robust
jinxixiang opened this issue · comments
Thank you for sharing such an interesting idea.
But I think one drawback of the demo, if I could be wrong, is the cross-attention maps are not robust. The results could be corrupted.
Hi @jinxixiang, thank you for your interest and for trying out our demo!
You are right that the token maps are sometimes not quite stable and accurate. We have been working on improving this and had some progress. Here is the example that changes the color of the hair.
@songweige Woo, this improved result is great! What did you modify with the cross-attention map? It seems to be refined.
I think one walkaround of this problem is utilizing the ability of Gounding SAM by splitting the denoising process into two stages. The first stage is to get the region mask, and the second stage is to conduct region-based diffusion. We are working on it.
We made a few changes. One major change was to use self-attention maps to compute the segmentation. Here are more examples. Hopefully we will update the demo by the end of this week.
Using SAM is a natural and nice idea. Please do let me know once you have some results or need any help from me.