Poor text responsiveness in prompt input box after recent update
gimmic opened this issue · comments
Browser, os, cpu, ram?
Yeah please add some more details, this would really help
I worked on responsiveness really hard here
In my experience, lagginess with the text input box occurs on iOS. It specifically only occurs after generating images, and the effect is more noticable with a bigger batch count. With one image generated, it is still usable, but with more, text appears in the box at a rate <1 character per second, regardless of how quickly you type. This is fixed only by refreshing the page after every generation :/
My issue has also been true for around a week in my experience, so probably not the same problem
This is chrome(Version 105.0.5195.54 (Official Build) (64-bit)) under windows 10. But I am currently unable to reliably replicate at the moment, sorry.
I'm going to guess this was introduced by #421
That code replaces the text of the prompt input box after every single keystroke, even if there's nothing to change or replace, which is really slow - and it involves an exchange with the backend/python before the update happens.
It's also measuring the DOM on every keystroke (if (prompt_input.scrollWidth > prompt_input.clientWidth + 10) which is also a very expensive operation to do so often.
also a very expensive operation
Most of our users run GPU calculations in the gigabites, and CPU intensive stuff as well, I'm sure that "very expensive" is a relative term.
@hlky I did this with python, remember when I showed you an early demo and it was laggy?
Then discussed with Gradio folks, they suggested JS.
This feature was added to support mobile and longer prompts paste, so we won't have overscoll on the input.
And I did really try to lower the performance overheads, but we all have powerful machines, I need a reproduction flow.
oh @ddgond ? maybe it's not related?
It's also measuring the DOM on every keystroke (if (prompt_input.scrollWidth > prompt_input.clientWidth + 10) which is also a very expensive operation to do so often.
I attempted to debug in console what would be causing the delay. It is definitely keystroke-update related.
Rapidly typing in a prompt results in input lag as if the browser was under load(when it isn't)
I'm sure that "very expensive" is a relative term.
It's relative, but you have to remember that all the JS code in the browser runs on a single thread.
The change event handler function is also blocking and synchronous and will prevent other code from running (or even the change that triggered the event from happening) until it finishes. Calls to DOM methods (or even property access of DOM objects) that look simple in the JS code can in fact be longer-running operations that the JS code has to wait for too.
I think the biggest source of lag is that the text is being piped through gradio on every keystroke though.
This is what happens:
- The user presses a key
- The change event fires
- Your custom change handler function runs
- It returns a list of values (including the prompt input's value) back to gradio
- Then gradio sends a message back to the page to tell it to update the prompt input
This issue has always existed on IOS once images have been generated, but this is the first time I've seen the issue happening on PC/Chrome. Generate around 20 images, and it's painfully slow to type on with a high-end CPU.
I would also bet that this issue wouldn't exist if the images could be loaded via files instead of base64 pngs.
Just process the --w 640 --h 640 --steps 40 --cfg_scale 9.0 from prompt on click of generate or on leave of textbox
This issue has always existed on IOS once images have been generated, but this is the first time I've seen the issue happening on PC/Chrome. Generate around 20 images, and it's painfully slow to type on with a high-end CPU.
I would also bet that this issue wouldn't exist if the images could be loaded via files instead of base64 pngs.
Does it remove characters as you type them? I've had this when I implemented the prompt search in python, so it sent every type with an ajax request, but as @nderscore said, now it's with JS.
I really wonder if there's anything else going on here, and obviously don't want to break our users experience for this feature.
Just process the --w 640 --h 640 --steps 40 --cfg_scale 9.0 from prompt on click of generate or on leave of textbox
Don't think that's a good UX tbh, I thought about on submit but there are a host of issues with that.
It's not a network issue, as when I moved to JS, nothing is hitting the network (see on slow 3g simulation here, over the gradio proxy)
CleanShot.2022-09-01.at.13.15.04.mp4
And I've tested on my phone, where previously I did have some stattering, and stattering was removed since this update
I can absolutely confirm it's perfectly smooth for me as well.
If this is presenting as a laggy issue though. Instead of having it check after every space, perhaps have it check once a second or every half second while the box is highlighted, or when the text box is clicked off of, or when the generate button is clicked. That way it's not auto-checking, but it will catch it before a render happens.
EDIT: I just saw it's done after every keystroke. That's could definitely be a bit intensive for some systems, though it shouldn't be that bad. Perhaps just do it after every space? Or like I said above, every half-second/second, and when the box is deselected, and when the generate button is clicked.
This issue has always existed on IOS once images have been generated, but this is the first time I've seen the issue happening on PC/Chrome. Generate around 20 images, and it's painfully slow to type on with a high-end CPU.
I would also bet that this issue wouldn't exist if the images could be loaded via files instead of base64 pngs.Does it remove characters as you type them? I've had this when I implemented the prompt search in python, so it sent every type with an ajax request, but as @nderscore said, now it's with JS.
I really wonder if there's anything else going on here, and obviously don't want to break our users experience for this feature.
Just process the --w 640 --h 640 --steps 40 --cfg_scale 9.0 from prompt on click of generate or on leave of textbox
Don't think that's a good UX tbh, I thought about on submit but there are a host of issues with that.
It's not a network issue, as when I moved to JS, nothing is hitting the network (see on slow 3g simulation here, over the gradio proxy)
CleanShot.2022-09-01.at.13.15.04.mp4
And I've tested on my phone, where previously I did have some stattering, and stattering was removed since this update
In your sample recording, it doesn't look like you have any images generated on screen. Try generating a batch of 20+ images first, and then try to rapidly type in text.
To your question, no I haven't seen any characters get removed as I type. It's just extremely slow.
I can reproduce on Chrome on both Linux and Windows 10, (Ryzen 5800x and Intel 8700K).
Try generating a batch of 20+ images first, and then try to rapidly type in text.
Related: Sygil-Dev/sygil-webui#330
Not a great example, but what is seen. Does not look to be any associated network traffic.
Yeah, I think this may not be related to the text prompt so much as just having all the images on screen. Like issue Sygil-Dev/sygil-webui#330 talks about. Having a lot of images on screen makes everything laggy, even the desktop interface of my PC to a point. It's not terrible, but I have a pretty powerful system, so it might be a lot worse for lower spec people. I recorded a video of this process, hopefully it'll show like it should. I also noticed VRAM requirements must be up ever so slightly, as I actually got a VRAM OOM error when I first tried with the default size settings, which typically doesn't happen. Had to close out of a couple apps on my desktop to get that to work, which I haven't needed to do before now.
EDIT: Well, apparently I couldn't even screen record what happened... So maybe the lagginess/system load is worse than I thought.
See my reply below
System load is nominal during this delay. Often when I see it in chrome, it is a chrome-specific(or backend) problem.
Ok, I just rebooted my system, which fixed the OOM issue, so that may be unrelated. However, the lag with the text box is still present. With a lot of images (25 in my test) on screen in the UI, if I go to backspace on a prompt, it lags hard. Looking at my system monitor, it seems to hit one CPU thread, and I'm on an i9-9900k so I have 16, to 100%. But it also only hits one thread to 100%. It appears the issue at play may be a single threading issue, whether that can be fixed or not. But it's still impressive seeing backspacing a textbox do that.
I also tested with just 1 image on screen and it did not do this, in fact I could not discernibly tell which thread was even handling that. So it seems there is an issue somewhere with how the images are being shown on screen, combined with the new processing for the text prompt. I don't think this is solely related to @altryne new text prompt stuff though, as the interface didn't seem to lag as much previously with a lot of images.
The real culprit is likely Gradio/Svelte, on every keypress there's some component update code running, and the large amount of dynamic base64 png text gets processed on every character typed as a result.
Now which change in this repo in the last 24-48 hours exacerbated that Gradio/Svelte inefficiency, I'm not sure.
For fun: simply removing the input event handler in Chrome Dev tools on the textbox fixes the issue.
The real culprit is likely Gradio/Svelte, on every keypress there's some component update code running, and the large amount of dynamic base64 png text gets processed on every character typed as a result.
That giant text is most likely js code passed from python that is eval()ed on every event.
@altryne you should conform this to the new frontend paradigm and move this code to index.js where it is only eval()ed once on load
ETA: if you're too busy which you probably are, assign this to me
I was working on this while your PR was in progress, and the earlier version was.. not super convinient, so while I waited I moved it to the old way.
There was also another thing, external file doesn't trigger the reload on gradio, so it was a bit more painful to iterate, I wonder what we can do with that.
For fun: simply removing the input event handler in Chrome Dev tools on the textbox fixes the issue.
Will take a look
also tested with just 1 image on screen and it did not do this, in fact I could not discernibly tell which thread was even handling that. So it seems there is an issue somewhere with how the images are being shown on screen, combined with the new processing for the text prompt. I don't think this is solely related to @altryne new text prompt stuff though, as the interface didn't seem to lag as much previously with a lot of images.
Just having them on the screen sitting there does this?? I thought about it during generation time
Will take a look
Just to clarify, the event handler there is needed, I only mentioned that to prove that there's code executing on keypress that causes the issue.
Just having them on the screen sitting there does this?? I thought about it during generation time
Yup, just need them sitting on the screen. I'm not even running the browser on the same computer as my GPU.
@altryne It is absolutely noticeable with the text box. Backspacing in particular causes one CPU thread to cap out at 100%. For others parts of the UI, it's just noticeable there might be a touch of lag, but I don't consider that an issue. But yes, it's not during generation, it's just with 25 images already generated, just sitting in the UI. So something about that, combined with the way the text input works, is causing some serious bottlenecking somewhere. Perhaps you need to try and make the event handler asynchronous? If you haven't done so, that could be why there is an issue, as an event handler like that is a blocking-call in JS, unless it's done with an async/await call.
@JoshuaKimsey just did a full deep dive with heaps and everything.
It's a Gradio issue, even though we do not touch or pass those images at all, or pass them in inputs or outputs, gradio does something with them on change event.
And, i'm not sure what it is, just doing an empty JS function does exactly the same result as doing all the regex and scrollWidth stuff.
4 is a snapshot of 50 loaded images with NO .change at all
3 is with .change and empty JS call
2 is without the scrollWidth code
and
1 is the latest.
cc @nderscore thanks for helping test
@altryne Ooof, that's gonna be tough to work around until Gradio can fix that. I know hlky mentioned you've filed issues there before, this definitely seems like it needs to be reported. They also kinda need to consider making that multi-threaded, if they can, given what I observed with it only ever hitting a single thread on my CPU.
In the meantime, does it do it with any image loaded on the page? Or just images loaded with Gradio? One option, if it's just images loaded with Gradio, is not using Gradio to load the images, but pure JS instead. It won't be the easiest workaround for something like this, but it could work as a stop-gap until the issue can be fixed on Gradio's end.