HyperKohaku
A diffusers based implementation of HyperDreamBooth
the code is based on the HyperDreamBooth implementation in the LyCORIS project.
TODOs
If you have any idea on these todos, just open issues or PRs. Thx!
- Implement in-run validation
- Implement section 4
- Implement better dreambooth training (like mask the face during training)
- Implement a more general hypernetwork (Already in LyCORIS)
HyperDreamBooth
This section is a brief introduction of HyperDreamBooth, I will split hyperdreambooth into 4 sections. And this projection will have 4 corresponding scripts
Section 1: Pre Optimize
Before we start to train the hypernetwork, we should train the lora(lilora) on each identites(instance) first. In this implementation, we took a batch of instances and then do a inner training loop to get the pre optimized weights.
Section 2: train the hypernetwork
Just train it. send the image into hypernetwork, get the weights, apply to unet. Calc the loss based on diffusion loss and weight loss.
Section 3: gen the weight
Use the image of the identity you want to train on, generate the weight from hypernetwork.
Section 4: further finetuning
resume from the generated weight, do a few step training (for about 20~50step)