ControlNet training with custom dataset?

[ifsheldon]

Hi! I read your training do. As far as I understand, without hacking the code, we now can only use or implement filters to generate control images on the fly? In my case, my control images are a bit tricky to automatically generate and all of them have been curated by me, which means I need to load a dataset with (reference_image, control_images, text_prompt) tuples. How can I do that?

[Andrew Suter-Morris]

reading through modules\controlnet.py, there is a CannyFilter which should apply on the fly.

edges = [cv2.Canny(x[i].mul(255).permute(1, 2, 0).cpu().numpy().astype(np.uint8), 100, 200) for i in range(len(x))]

This means it does canny with threshold [100,200]. As long as you specify CannyFilter in your yaml file such as:

controlnet_filter: CannyFilter

then you should be set.

[Dominic Rampas]

Hey, yea indeed you would need some manual editing of the code to make it work to use your own pregenerated control images. You would add them to the webdataset and load those as extra keys. Let me know if you get stuck at some point.

[sapkun]

Hey, yea indeed you would need some manual editing of the code to make it work to use your own pregenerated control images. You would add them to the webdataset and load those as extra keys. Let me know if you get stuck at some point.

Hi! I need some example code like this that compatible with this dataset or its webdataset version. Can you show me where to change specificly? thank you!