[Feature Req] Limit LORA influence when using MASK or AREA
Lancer1408 opened this issue · comments
Hello and thanks for making these nodes!
They are really interesting and I've been playing around with them.
So I'm testing along the MASK and AREA modes, and I've come to wonder if its already possible to limit the influence of a lora only to the MASK or AREA regions without the lora baking the other parts of the generation.
If not, could it be possible to implement something similar to this?
As on Auto1111 the extensions Lora Masks and Composable Lora achieve the desired effect of only allowing a lora to work in a specific region without frying the rest of the generation.
Thanks a lot for your time!
Unfortunately, that goes beyond my capabilities currently; pretty much everything this extension does relies on something ComfyUI can do; I just provide a nice text-based interface to it. So if ComfyUI implements LoRA masking, I'll add support for it, but until then, it's not likely to happen.
As far as I understand ComfyUI's logic, it applies LoRAs as "patches" on top of the model weights, essentially modifying the model before sampling. With that logic, there's no way to apply a mask, so I would have to implement a completely different mechanism for applying LoRAs on top of figuring out how to mask their effects.
Honestly, pretty much all the code dealing with model layers, the different model weight keys and setting up the actual inference calculation goes completely over my head; I don't even know what half the operations in the code mean.
rip, tho I kinda expected it to be really hard to implement, as the only node that does something similar is one of the Impact pack nodes, but that one generates over an already existing image.
I guess the messy code of Auto1111 allowed the extensions I linked possible lol
Thanks a lot!
@Lancer1408 There is probably some way to do this, if I reimplement LoRAs as a transformer patch or something, but it's going to take more than one evening of study to figure things out, which might mean I'll have it done next week or next decade, given that my motivation to work on new features is completely unpredictable. :-)
@asagi4 lol no need to hurry, just wanted to know if it was possible, tho masking loras is a really cool concept that I would be cool to see on Comfy.
Totally, I'm sure it is indeed possible, but seems to be quite hard to implement without messing too much with the ksamplers and lora forwards.