Critical Performance Drop in InvokeAI 4.2.x: Image Generation Slows to a Crawl
zero2999 opened this issue · comments
Is there an existing issue for this problem?
- I have searched the existing issues
Operating system
Linux
GPU vendor
Nvidia (CUDA)
GPU model
NVIDIA GeForce RTX 3060 Mobile / Max-Q
GPU VRAM
6GB
Version number
4.2.1
Browser
microsoft edga
Python dependencies
No response
What happened
Hello, I have been a user of InvokeAI since version 3.x, and up until version 4.1.x, it generated images efficiently in both S.D 1.5 and S.D XL. However, after upgrading to version 4.2.x, the performance has deteriorated significantly, with image generation times ranging from 15 to 30 minutes. I reviewed the documentation for potential solutions, but the current recommendation is to perform a clean uninstall and reinstall. I have executed this procedure four times, but the issue persists.
Assuming the problem might be related to a system update, I performed a complete OS reinstallation. Unfortunately, the performance issues remain unchanged, with the application still running sluggishly. I would appreciate any technical assistance or solutions to optimize and expedite the image generation process.
What you expected to happen
I expected image generation in InvokeAI to be fast and efficient, similar to previous versions (up to version 4.1.x), in both S.D 1.5 and S.D XL.
How to reproduce the problem
No response
Additional context
No response
Discord username
No response
Let's check a couple things:
- When the app starts up, in the terminal, we should see the GPU torch is using. The line looks like this:
What does it say for you?
[2024-05-14 13:38:26,759]::[InvokeAI]::INFO --> Using torch device: NVIDIA GeForce RTX 4090
- Please copy and paste your
invokeai.yaml
file so we can review the settings
patchmatch.patch_match: INFO - Compiling and loading c extensions from "/home/ppbg/invokeai/.venv/lib/python3.10/site-packages/patchmatch".
patchmatch.patch_match: ERROR - patchmatch failed to load or compile (Command 'make clean && make' returned non-zero exit status 2.).
patchmatch.patch_match: INFO - Refer to https://invoke-ai.github.io/InvokeAI/installation/060_INSTALL_PATCHMATCH/ for installation instructions.
[2024-05-14 23:56:58,900]::[InvokeAI]::INFO --> Patchmatch not loaded (nonfatal)
[2024-05-14 23:57:01,258]::[InvokeAI]::INFO --> Using torch device: CPU
[2024-05-14 23:57:01,431]::[InvokeAI]::INFO --> cuDNN version: 8902
[2024-05-14 23:57:01,451]::[uvicorn.error]::INFO --> Started server process [51400]
[2024-05-14 23:57:01,451]::[uvicorn.error]::INFO --> Waiting for application startup.
[2024-05-14 23:57:01,451]::[InvokeAI]::INFO --> InvokeAI version 4.2.1
[2024-05-14 23:57:01,452]::[InvokeAI]::INFO --> Root directory = /home/ppbg/invokeai
[2024-05-14 23:57:01,452]::[InvokeAI]::INFO --> Initializing database at /home/ppbg/invokeai/databases/invokeai.db
[2024-05-14 23:57:01,462]::[InvokeAI]::INFO --> Pruned 3 finished queue items
[2024-05-14 23:57:01,477]::[InvokeAI]::INFO --> Cleaned database (freed 0.02MB)
[2024-05-14 23:57:01,478]::[uvicorn.error]::INFO --> Application startup complete.
[2024-05-14 23:57:01,478]::[uvicorn.error]::INFO --> Uvicorn running on http://127.0.0.1:9090 (Press CTRL+C to quit)
And that's even though I chose option 1 for Nvidia during installation. As I mentioned earlier, it used to generate images very quickly, but now it's incredibly slow when generating an image
invokeai.yaml
Internal metadata - do not edit:
schema_version: 4.0.1
Put user settings here - see https://invoke-ai.github.io/InvokeAI/features/CONFIGURATION/:
It seems torch didn't install the correct version of itself or is falling back to CPU. What do you get when you run nvidia-smi
in a terminal?
ppbg@ppbg-IdeaPad-Gaming-3-15ACH6:/.cache$ nvidia-smi0.22.04.1
Command 'nvidia-smi' not found, but can be installed with:
sudo apt install nvidia-utils-390 # version 390.157-0ubuntu0.22.04.2, or
sudo apt install nvidia-utils-418-server # version 418.226.00-0ubuntu5
sudo apt install nvidia-utils-450-server # version 450.248.02-0ubuntu0.22.04.1
sudo apt install nvidia-utils-470 # version 470.223.02-0ubuntu0.22.04.1
sudo apt install nvidia-utils-470-server # version 470.223.02-0ubuntu0.22.04.1
sudo apt install nvidia-utils-525 # version 525.147.05-0ubuntu0.22.04.1
sudo apt install nvidia-utils-525-server # version 525.147.05-0ubuntu0.22.04.1
sudo apt install nvidia-utils-535 # version 535.129.03-0ubuntu0.22.04.1
sudo apt install nvidia-utils-535-server # version 535.129.03-0ubuntu0.22.04.1
sudo apt install nvidia-utils-510 # version 510.60.02-0ubuntu1
sudo apt install nvidia-utils-510-server # version 510.47.03-0ubuntu3
I'm pretty sure you get nvidia-smi
when you install your GPU drivers. Missing or nonfunctional drives could result in the app falling back to CPU, causing generation to be very slow.
Please ensure your GPU drivers are installed - reinstall if they are not - then run nvidia-smi
so we can confirm that your drivers are installed and functioning.