[FR] Attention Interpolation

[Alexander Brown]

Describe what's missing
Basically, this: https://github.com/ljleb/prompt-fusion-extension/wiki/Attention-Interpolation

Additional context
🙏🏻
Not sure if there's a way to accomplish this using the existing interpolation feature.

[asagi4]

I think you can sort of do this already if you combine the MUSimpleWildcard / Jinja nodes from my other repo to generate a [SEQ:...] element with changing weights, ie, you can generate a [SEQ:(a:0.1)(b:0.9):0.1:(a:0.2)(b:0.8):0.2:...] etc. The generated prompt will be long but I think it'll do about the same thing as prompt fusion's interpolation. The MUSimpleWilcard node contains functions to generate sequences like that already, though I admit they aren't the easiest to use.

The existing interpolation feature works on the embedding tensors directly so it's unfortunately not useful.

[Alexander Brown]

Hmmm, this does look like it works, but it's not the prettiest.
attention_interpolation_via_seq.json

[Alexander Brown]

Edited to allow for descending weight

<%- macro step_weight(weight, start, dir) -%>
  <= round(start + (dir * weight), 2) =>
<%- endmacro -%>
<%- macro switch_at(weight, startX, endX) -%>
  <= round(abs(weight) / abs(endX - startX), 2) =>
<%- endmacro -%>

<%- set start = min(__weight-start__, __weight-end__) -%>
<%- set end = max(__weight-start__, __weight-end__) -%>
<%- set dir = 1 if __weight-start__ < __weight-end__ else -1 -%>

[SEQ
<%- for weight in steps(0, end - start, __step-size__) -%>
:(__prompt-segment__:
  <=- step_weight(weight, __weight-start__, dir) -=>):
  <=- switch_at(weight, start, end) -=>
<%- endfor -%>
]

[asagi4]

You can make that a bit simpler with MUSimpleWildcard with its in-prompt variable / macro support, but that does get a bit complex.

I actually didn't know Jinja can do macros; I should probably refactor the "built-in" functions in MUSimpleWildcard to use them more.