Working with multiple characters.

[Arthur Wolf]

I'm working with sd-webui's API + controlnet + sd-webui-cutoff to take black and white line art manga/comics, and color them with the correct colors for the characters that were detected (by gpt4o) in a given panel of a comic/manga.

This works (somewhat) well when there is a single character in the image.

But it doesn't work when there are multiple, it doesn't know which character has what color hair, color clothes, etc.

How do I solve this?

I'm looking for any solution, no matter how difficult, to try to implement. If there are multiple possible ways to attempt this, i'd like to learn about / try all of them.

Any help would be very very welcome.

Thank you