Passing a ton of data around
opened this issue · comments
It's throwing alot of base64 in and out from Dom land to WASM VM land.
If you use a canvas / webgl on the out it should be more efficient.
Also what about web workers to make it stay interactive.. passing the base64 as a message post is same impact as you have now.
I would also do an edit list. Edit list is where you key up the mutations as a list and in interactive mode it just does the operations on an image 10% of the size.
Then when your happy with the effect the mutations have done you tell it to do it again on the real full size image.
This is what all special effects editors do.
Thanks for your time 😄
If you use a canvas / webgl on the out it should be more efficient.
Yes I know. I have mentioned in my blog post as something to be done. Probably coming weekend ;)
Also what about web workers to make it stay interactive.. passing the base64 as a message post is same impact as you have now.
Not very worried about that now. This is just an experiment to look into WASM. Interactivity comes later.
I would also do an edit list. Edit list is where you key up the mutations as a list and in interactive mode it just does the operations on an image 10% of the size.
Interesting, can you point me to some documentation ?
Hmm .. unfortunately it becomes slower by using canvas/webgl. Because now I have to pass an even bigger img.Pix array to the browser. That alone takes around 1.5s. The canvas rendering just takes 30ms. See my webgl branch
Ignore the inverted image for now. Some webgl issue.
Thanks for the very nice proof of concept. I had a bit more time to experiment with it.
https://github.com/yml/shimmer/pull/1/files
I have slightly modified the worflow to push the sourceImg in the wasm side of the fence and only insert resized (smaller) images in the DOM. This makes the image operation faster it also help to address the issue described above isn't it ?
Can we make sure the webgl option is on a branch too. I also have some ideas to speed it up but am rushed right now for time.
Yes it addresses the Edit List concept nicely.
I think I can add a reactor replay module for playing back onto a large image. Then the user can playback the edits client side or server side.
You can go the full monty and distribute tiles too for each image.
@yml -
I have slightly modified the worflow to push the sourceImg in the wasm side of the fence
Brilliant idea. Can you just modify the code not to resize the target image ? That's going to a different class of optimization which I don't want to do right now. Please send a PR in your convenience. Thanks for your time.
@gedw99 -
Can we make sure the webgl option is on a branch too.
Yep, it already is. https://github.com/agnivade/shimmer/commits/webgl. I would like to apply @yml's patch first to master though.
@agnivade I am not sure exactly which part you want in the PR. I have done 4 things in this branch
- Upload the source image directly in "wasm world"
- resize it
- answer the resized image 2 times as preview and as target
- every time you apply an effect the target image is recomputed starting from the resized images
If you don't work on the resized image you are going to increase the amount of data that is transferred between "js world" and "wasm world". You are source image is going to cross the boundary 2 times instead of one.
Ah I did not think that you would have to preview the image back in the browser. That was dumb of me. I will look into ways we can get the event in browser, as well as DOM. So that we can show the preview directly in browser, as well as get the data in "wasm world" directly. I see that NewEventCallback has no way to bubble the event.
If you don't work on the resized image you are going to increase the amount of data that is transferred between "js world" and "wasm world".
Yes, I know. Resizing the image is a simple way to reduce data transfer. For now, I just want to experiment with basic ways to optimize the workflow without affecting image size or quality. (Like directly writing to a canvas, which did not yield any positive results)
But feel free to keep experimenting and let me know if you find something interesting :)
I was able to load the image directly in wasm as well as preview it in the browser. We simply can make both callbacks live side-by-side. The browser side callback can read from the file and set the img url. The wasm side callback can read and then set the image base64 string.
But most probably this doesn't do what we think it does. i.e. it does not save any data transit from browser to wasm land. The moment we do a ev.Get("target").Get("result").String() call, data gets transferred from browser to wasm. I tested it with uploading big and small images and timed it. There was noticeable lag for big images, which means data is still being transferred.
Conceptually thinking, there is no shared memory here. Anything that needs to be loaded in wasm needs to be initialized first, and therefore the image data needs to be transferred anyways. I don't see any way around it. Happy to be proven wrong.
@agnivade I keep using this ticket to continue the discussion :-)
I went ahead and did more experimentation around image processing : https://github.com/yml/wasmBild
The goal there was to try different application layout. One of the goal was to decouple App from syscall/js. Turn out that struct embedding seems to work well there.
I am curious to hear about your thoughts on the approach taken in the repo above.
Yes, please do report with your findings here :)
It seems like you are injecting html content with a template so that you don't have to change in 2 places when you add a new transformation. Looks like this might be helpful for big apps with lots of moving pieces.
I am however, still interested in performance optimizations. On a cursory look at your code, it seems that you are still getting the .String() of the image base64 and then setting the src attribute of the target image. So the 2 times crossing over of data is still there. I was wondering if there was a way to optimize that.
@agnivade I only insert in the DOM the resized images and all the transformation are applied to the resized images. This makes the app very interactive even if you chain the transformations.
Ideally I would like to to be able to render the app server and client side, sharing the template would be a good first (baby) step.
You are reading the image here - ev.Get("target").Get("result").String()). This crosses the js and wasm boundary. I don't believe it is doing what we think are doing. i.e. it is not doing everything inside wasm land. So as a result, you are crossing the boundary 3 times, not 2 times.
See my comment #1 (comment).
I agree with you that this call transfer the full image resolution from JS to the WASM land. But as far I can tell further manipulation only transfer the resized or the preview images. The preview image is based on the resized.
The original image is then store and access directly in the wasm side.
I am looking forward to see if you manage to find a solution that will reduce the data transferred even more.
Right. I don't have any further ideas as of this moment. Unless we can transfer the data in binary in some way ..
The canvas/webgl approach that I tried needed 32bits for every pixel which bloated the data transfer size even more. We need some way to directly transfer data in binary and read it in the browser.
Did some more investigation. The whole brunt of time is actually going in the image processing. If I just calculate the time it takes to transfer the base64 data, it is actually 2-3ms. The compute time itself is around 200-300ms. I believe data transfer is not the bottleneck here. And the compute entirely happens in wasm which is not under my control.
I will keep looking, but since the data transfer is so tiny part of the entire time, I wonder if it is worth spending more effort in optimizing this.
I wonder if this app could also trigger the behavior describe in that ticket:
The whole brunt of time is actually going in the image processing.
This is why working on the resized image makes such a difference.
As long as the memory is properly garbage collected and that there is no leak, I don't think that there is much improvement that can be done until JS and WASM can share some memory to avoid the copy back and forth.
I wonder if this app could also trigger the behavior describe in that ticket:
Hmm .. I don't think so. I am just passing strings, not slices. It might trigger, but it is certainly not the primary bottleneck in this app.
Agree with your other observations.
Going to close this because data transfer is not the main bottleneck as I see it. We can open further issues to optimize other areas of this. Thank you everyone for your input.