[Bug]: LoRa doesn't affect images

Question

[Bug]: LoRa doesn't affect images

WheatTail opened this issue 2 months ago · comments

Checklist

The issue exists after disabling all extensions
The issue exists on a clean installation of webui
The issue is caused by an extension, but I believe it is caused by a bug in the webui
The issue exists in the current version of the webui
The issue has not been reported before recently
The issue has been reported before but has not been fixed yet

What happened?

I've downloaded a clean forge (version: f0.0.17v1.8.0rc-latest-276-g29be1da7, latest I believe) and tried to create some images. After a while I found that LoRas doesn't work as intended. And after small digging up I found that they doesn't work at all for me.
The experiment: I made a simple prompt, using mfcgPDXL_v10.safetensors model, sdxl_vae.safetensors vae and FF-Style-Edvard-Munch-32.safetensors LoRa.
The prompt was:
Rainbow Dash, pony, realistic fur, furry, fluffy, semi-anthro, smiling, cute, wings unfolded, <lora:FF-Style-Edvard-Munch-32:1>, style of Edvard Munch, Edvard Munch style, Edvard Munch art, Edvard Munch
Console says:

[LORA] Loaded D:\Stable Diffusion\stable-diffusion-webui-forge\webui\models\Lora\FF-Style-Edvard-Munch-32.safetensors for SDXL-UNet with 722 keys at weight 1.0 (skipped 0 keys)
[LORA] Loaded D:\Stable Diffusion\stable-diffusion-webui-forge\webui\models\Lora\FF-Style-Edvard-Munch-32.safetensors for SDXL-CLIP with 264 keys at weight 1.0 (skipped 0 keys)

Got this nice pic of Rainbow Dash.

But if that's Edvard Munch style - then I'm Her Highness.
After that I removed LoRa from it's folder and hit "generate" again. As expected, I got an error:

loading network D:\Stable Diffusion\stable-diffusion-webui-forge\webui\models\Lora\FF-Style-Edvard-Munch-32.safetensors: FileNotFoundError
Traceback (most recent call last):
  File "D:\Stable Diffusion\stable-diffusion-webui-forge\webui\extensions-builtin\Lora\networks.py", line 56, in load_networks
    net = load_network(name, network_on_disk)
  File "D:\Stable Diffusion\stable-diffusion-webui-forge\webui\extensions-builtin\Lora\networks.py", line 31, in load_network
    net.mtime = os.path.getmtime(network_on_disk.filename)
  File "genericpath.py", line 55, in getmtime
FileNotFoundError: [WinError 2] Не удается найти указанный файл: 'D:\\Stable Diffusion\\stable-diffusion-webui-forge\\webui\\models\\Lora\\FF-Style-Edvard-Munch-32.safetensors'

activating extra network lora with arguments [<modules.extra_networks.ExtraNetworkParams object at 0x000001F4C5A25810>]: ValueError
Traceback (most recent call last):
  File "D:\Stable Diffusion\stable-diffusion-webui-forge\webui\modules_forge\patch_basic.py", line 17, in loader
    result = original_loader(*args, **kwargs)
  File "D:\Stable Diffusion\stable-diffusion-webui-forge\system\python\lib\site-packages\safetensors\torch.py", line 308, in load_file
    with safe_open(filename, framework="pt", device=device) as f:
FileNotFoundError: No such file or directory: "D:\\Stable Diffusion\\stable-diffusion-webui-forge\\webui\\models\\Lora\\FF-Style-Edvard-Munch-32.safetensors"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Stable Diffusion\stable-diffusion-webui-forge\webui\modules\extra_networks.py", line 135, in activate
    extra_network.activate(p, extra_network_args)
  File "D:\Stable Diffusion\stable-diffusion-webui-forge\webui\extensions-builtin\Lora\extra_networks_lora.py", line 43, in activate
    networks.load_networks(names, te_multipliers, unet_multipliers, dyn_dims)
  File "D:\Stable Diffusion\stable-diffusion-webui-forge\webui\extensions-builtin\Lora\networks.py", line 78, in load_networks
    lora_sd = load_lora_state_dict(filename)
  File "D:\Stable Diffusion\stable-diffusion-webui-forge\webui\extensions-builtin\Lora\networks.py", line 18, in load_lora_state_dict
    return load_torch_file(filename, safe_load=True)
  File "D:\Stable Diffusion\stable-diffusion-webui-forge\webui\ldm_patched\modules\utils.py", line 18, in load_torch_file
    sd = safetensors.torch.load_file(ckpt, device=device.type)
  File "D:\Stable Diffusion\stable-diffusion-webui-forge\webui\modules_forge\patch_basic.py", line 33, in loader
    raise ValueError(exp)
ValueError: No such file or directory: "D:\\Stable Diffusion\\stable-diffusion-webui-forge\\webui\\models\\Lora\\FF-Style-Edvard-Munch-32.safetensors"

And generation continued without LoRa loading. As result - I got exactly the same image,
I've tried it with another model (ponyDiffusion v6), another LoRa (PDXL_artist_tags_v2), tried to use non ponyDiffusion nor SDXL models/LoRa (easyfluff_5000DefaultvaeFFR3CS2 plus mlp_queenchrysalis-10). All the time I've got exactly the same images despite was LoRa loaded successfully or not.
I have two extensions (adetailer and regional prompter) but both was installed after I found the problem. Tried to handle it myself, then via google, then reddit, but no avail.

Steps to reproduce the problem

Open webui
Made prompt with LoRa, generate
Remove LoRa from folder
Press "reuse seed from last generation" to get same condition
Generate again

What should have happened?

LoRa should've affect the image.

What browsers do you use to access the UI ?

Mozilla Firefox, Google Chrome

Sysinfo

sysinfo-2024-06-04-21-33.json

Console logs

PS D:\Stable Diffusion\stable-diffusion-webui-forge> .\run.bat
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: f0.0.17v1.8.0rc-latest-276-g29be1da7
Commit hash: 29be1da7cf2b5dccfc70fbdd33eb35c56a31ffb7
Launching Web UI with arguments: --listen
Total VRAM 8192 MB, total RAM 32713 MB
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 2070 : native
Hint: your device supports --pin-shared-memory for potential speed improvements.
Hint: your device supports --cuda-malloc for potential speed improvements.
Hint: your device supports --cuda-stream for potential speed improvements.
VAE dtype: torch.float32
CUDA Stream Activated:  False
Using pytorch cross attention
ControlNet preprocessor location: D:\Stable Diffusion\stable-diffusion-webui-forge\webui\models\ControlNetPreprocessor
[-] ADetailer initialized. version: 24.5.0, num models: 10
Loading weights [529c72f6c3] from D:\Stable Diffusion\stable-diffusion-webui-forge\webui\models\Stable-diffusion\24) mfcgPDXL_v10.safetensors
2024-06-05 00:37:20,342 - ControlNet - INFO - ControlNet UI callback registered.
Running on local URL:  http://0.0.0.0:7860
model_type EPS
UNet ADM Dimension 2816

To create a public link, set `share=True` in `launch()`.
Startup time: 27.6s (prepare environment: 5.1s, import torch: 6.4s, import gradio: 1.5s, setup paths: 1.5s, initialize shared: 0.2s, other imports: 1.1s, load scripts: 5.8s, create ui: 1.2s, gradio launch: 4.5s).
Using pytorch attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using pytorch attention in VAE
extra {'cond_stage_model.clip_l.text_projection', 'cond_stage_model.clip_g.transformer.text_model.embeddings.position_ids', 'cond_stage_model.clip_l.logit_scale'}
Loading VAE weights specified in settings: D:\Stable Diffusion\stable-diffusion-webui-forge\webui\models\VAE\sdxl_vae.safetensors
To load target model SDXLClipModel
Begin to load 1 model
[Memory Management] Current Free GPU Memory (MB) =  7158.99609375
[Memory Management] Model Memory (MB) =  2144.3546981811523
[Memory Management] Minimal Inference Memory (MB) =  1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) =  3990.6413955688477
Moving model(s) has taken 0.65 seconds
Model loaded in 20.3s (load weights from disk: 1.4s, forge instantiate config: 1.5s, forge load real models: 15.8s, load VAE: 0.4s, calculate empty prompt: 1.1s).
2,1 0.2 Horizontal
Regional Prompter Active, Pos tokens : [20, 2, 10, 24], Neg tokens : [33]
[LORA] Loaded D:\Stable Diffusion\stable-diffusion-webui-forge\webui\models\Lora\FF-Style-Edvard-Munch-32.safetensors for SDXL-UNet with 722 keys at weight 1.0 (skipped 0 keys)
[LORA] Loaded D:\Stable Diffusion\stable-diffusion-webui-forge\webui\models\Lora\FF-Style-Edvard-Munch-32.safetensors for SDXL-CLIP with 264 keys at weight 1.0 (skipped 0 keys)
To load target model SDXLClipModel
Begin to load 1 model
Reuse 1 loaded models
[Memory Management] Current Free GPU Memory (MB) =  5360.80859375
[Memory Management] Model Memory (MB) =  0.0
[Memory Management] Minimal Inference Memory (MB) =  1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) =  4336.80859375
Moving model(s) has taken 1.22 seconds
To load target model SDXL
Begin to load 1 model
[Memory Management] Current Free GPU Memory (MB) =  7122.64306640625
[Memory Management] Model Memory (MB) =  4897.086494445801
[Memory Management] Minimal Inference Memory (MB) =  1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) =  1201.5565719604492
Moving model(s) has taken 1.80 seconds
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:27<00:00,  1.11it/s]
To load target model AutoencoderKL███████████████████████████████████████████████████████████████████████████| 30/30 [00:26<00:00,  1.11it/s]
Begin to load 1 model
[Memory Management] Current Free GPU Memory (MB) =  7092.36181640625
[Memory Management] Model Memory (MB) =  319.11416244506836
[Memory Management] Minimal Inference Memory (MB) =  1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) =  5749.247653961182
Moving model(s) has taken 0.35 seconds
Total progress: 100%|████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:27<00:00,  1.08it/s]
2,1 0.2 Horizontal0%|████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:27<00:00,  1.11it/s]

Additional information

No response