Alternating prompt produces similar results
DeInfernal opened this issue · comments
When you try to prompt [A|B] and [B|A] in Automatic1111, you will inevitably get vastly different results, as first step as A and first step as B drastically changes flow of whole picture. However, doing same thing in this node results in completely identical pictures, winch shouldn't be happening. Something is wrong, somewhere, somehow.
Replicateable used workflow: https://hastebin.pw/WF91epqcW3
Also, maybe as part of a different issue - using Alternating prompt results in really slowed down generation progress. Like, really slowed down...
Alternating can slow things down because what it actually does is just generate a whole bunch of conditionings that are restricted to specific timestep percentages.
Did you try reducing the step percentage or are you using the default 0.1?
Alternating works differently from A1111 due to how time ranges in ComfyUI work; it's nontrivial to implement something that alternates with every step without basically reimplementing sampling entirely.
It is possible there are bugs though. The parser implementation isn't the best and I find myself accidentally breaking it sometimes.
I tested on bunch of percentages from 0.01 to 0.1. Thought that if percentage is low enough (0.02083 corresponds to 1/48th), I could repeat same result as A1111's alternation, but got massive slowdown and slightly better result than 0.1... but even with that, still got same results - [A|B] = [B|A], and even [A|B|B|A] = [B|A|A|B].
I never tested hyperloop, tho. 48 samplers connected to each other with physically alternating requests, limited to a single step each. But that would be pretty strange to say the least :). But if it will yield me the same result of [A|B] = [B|A] then it would be very interesting. I'll be testing that today.
Did an experiment called BFG (big freaking railGun - judging on amount of connections I had made for a 24-steps generation) - https://hastebin.pw/2FSK2wuvas workflow - so to say, True Alternating. Results are indeed different. Is it possible to make it look like this or so?
In my testing, the output of [A|B] and [B|A] do change. It's definitely different from sampling one step at a time though.
Implementing true alternation like in your crazy graph (that's a lot of links) would require a significant refactor. I might make something like that possible if/when the lazy evaluation pull request is merged (comfyanonymous/ComfyUI#931), but don't hold your breath.
My implementation currently completely relies on ComfyUI's support for specifying timesteps for conds. There is probably something with how Comfy calculates what conditioning to apply that makes it work like this.
All the conds are also just passed as an array to the sampler (as if they went trough the ConditioningCombine node). That might also be different from how A1111 does it.
I'm not a 100% sure why it's that slow though. I don't actually understand the math and magic behind Stable Diffusion sampling all that well... Writing these nodes has forced me to learn something, but the exact effect of every operation on each tensor is still unclear to me, so figuring this out would require me to dig deeper into ComfyUI's sampling logic.
As far as I understand it, ComfyUI doesn't do any kind of switching between conds per step; it just does some math to calculate their effect taking the timestep ranges into account.
Okay, so I added some debug prints to ComfyUI's sampling and figured out what's going on.
When determining what conds to include, ComfyUI doesn't actually look at the number of steps sampled, but at the scheduled timestep value, which depends on which scheduler the sampler is using, meaning that the timesteps aren't necessarily linear. This leads to the schedules not being linear either.
I can't really change this behaviour without modifying ComfyUI or reimplementing the sampler. I'll see if I can find a way to hook in a cond filter somehow.
@DeInfernal I added a commit that tries to figure out "absolute" timesteps. You can trigger it with the magic word "#ABS#" anywhere in the prompt (It'll get automatically removed, so it won't otherwise affect generations)
Can you test and see if that behaviour makes more sense?
The way it's triggered will probably change. Upstream having a feature for this might make the most sense.
Hm, it doesn't seem that the sigmas passed to the sampler completely correspond to what's actually used during sampling. Meh.
I'm here. Commit 4843d59, Used latest commit with #ABS# + same testing workflow as I had in first message. When scheduler is set to my favorite karras + sampler to dpmpp 2s a, all 4 images ([A|B], [B|A], [A|B] #ABS#, [B|A] #ABS#) is the same pixel to pixel.
DPMPP 2S A + Normal scheduler results in [A|B] = [B|A] as well. #ABS# changes nothing.
And even Euler A + Normal scheduler results in [A|B] = [B|A]. #ABS# also changes nothing.
The workflow in the first message? I get very different results for all of those prompts,#ABS# also does change things for me, though not as much
Which version of ComfyUI are you using? Are you sure you're using a recent enough version that timestep ranges actually have an effect?
You need a version of comfyUI newer than 0240946ecff9527a1c39f705ae9d2324612f4cf7 which was committed on July 24, or the schedules will get just completely ignored.
...damn. My version was from 7th July. After update I:
- Did get pretty significant speedup even when using low numbers such as :0.02.
- Didn't get any difference from using #ABS# and not using it (tested Euler a + normal and DPMPP 2s a + karras)
- Finally got the difference between images [A|B] and [B|A].
Suggestion: Add at least one version check into node so it won't start up if version is under required.
I guess I could do that. Oh well, it wasn't wasted effort, at least. This did lead to me discovering that the timesteps ranges don't quite work like I thought they did, which is helpful. I'll close this for now though.
Alternating works differently from A1111 due to how time ranges in ComfyUI work; it's nontrivial to implement something that alternates with every step without basically reimplementing sampling entirely.
I have this implemented in my node. I don't know why you're dealing with lower level details like timesteps when you can hijack ComfyUI's denoiser (which gets run for every sampler) and have access to every step directly.
@shiimizu the simple answer is that initially I didn't even notice that the cfg function can be overridden, and once timestep ranges were introduced, I just went with those instead.
I'm pretty sure the cfg runs too late for what I want to do, anyway, and it doesn't have access to the step number either.
I didn't want to resort to doing anything like counting timesteps, because that likely wouldn't work correctly if people use the Advanced KSampler node and don't start sampling at the first step, or with more advanced samplers like the Restart sampler.