StyleGAN example Processing 4.01A
j3nsykes opened this issue · comments
I appreciate Runway ML Lab isn't as supported now however I'm hoping I might be able to find a solution for teaching purposes.
I am running the StyleGAN example in processing 4.01A (August 9th 2022 release). The example communicates via HTTP successfully. I can see the Runway input interface and preview window update. However, the image is not returned to processing. The error is
Error, disabling runwayDataEvent() null
I have printed the infoEvent details and they are as follows.
{ "outputs": [{ "defaultOutputFormat": "JPEG", "channels": 3, "name": "image", "description": null, "type": "image" }], "inputs": [ { "samplingMean": 0, "default": null, "name": "z", "length": 512, "description": null, "type": "vector", "samplingStd": 0.5 }, { "default": 0.8, "min": 0, "max": 1, "name": "truncation", "description": null, "step": 0.01, "type": "number" } ] }
The example works perfectly in processing 3.5.4 so i don't think it is a message string issue or change on Runway's side. I have tried giving Processing 4 permission access to all files and folders but no change. Other examples run without issue in Processing 4.01 A (eg:GPT2 example that is text based) therefore it is maybe a Base64 issue? Any pointers greatly appreciated as I hope to use this in a teaching example and would rather not request students mix versions of Processing.
Hi @j3nsykes I'd be curious to follow this, while I'm not teaching StyleGAN -> Processing right now, it comes up often for me! Last year I did a bunch of examples with p5.js which I could maybe use to test. I've also switched to some other model hosting services (like hugging face!) but it is trickier to train your own models! If you want to share the full code example I'm happy to take a look! ccing @yining1023 who is teaching a similar class in case she also wants to follow!
Hi @shiffman thanks for replying so quickly and offering to help!
Currently I am just running the examples from the Runway ML Processing library. This one
I've also got some hosted model examples in P5 too but I hoped to show a range of methods that join up workflows and interfaces for teaching purposes. With the advantage of running locally in Processing and perhaps joining up other sketches students have been working on. (they are more practiced in processing than p5 right now). However, I will move back towards P5 if I can't resolve this I think.
Thanks again
Just to update : my P5JS examples that return a base 64 image from Runway also no longer work. Any example that just returns text or numbers (bounding box co-ords for example) still work fine but examples which receive imagery from runway either to P5 or Processing 4.01A no longer work . I would normally think this would be to do with a change in JSON data structure coming from Runway. However, the examples still work in Processing 3.5.4 which doesn't support that theory.
Hi @j3nsykes I just checked and my p5 example is still working. I have it paired with a node server to make the request.
https://runway-ml-template.glitch.me
https://glitch.com/edit/#!/runway-ml-template
I can try to fire up Processing later today and see if this same model will work there!
Interesting! This made me triple check and it materialises my P5JS examples (from the web editor) run in Firefox but not Chrome or Safari. The Debug Inspector is indicating a CORS permission issue which is OK I can work around that for future reference.
Processing 4.0.1A - still no closer
Ah, so looking at the Processing example you linked I believe it's using the OSC connection to the Runway desktop app, is that correct? I bet there's some kind of networking permissions (or firewall settings?) that Processing 4 requires? My p5 examples all use the hosted models and it would be possible to do the same with Processing using the HTTPRequests library directly?
https://github.com/runemadsen/HTTP-Requests-for-Processing
Or am I confused? I using HTTP Requests is helpful, I can see if I can get an example working!
Are you on windows or mac?
Oh its using a local network HTTP request already . Although not via hosted models but via the Network tab interface in Runway Workspaces.
I'm on a Mac and its a new M1 chip which I had thought might also be an issue.
Could still be a permissions issue though
I've printed the runwayData to console that's received into Processing from Runway and I can see it is receiving the Base64 String but throws an error when trying to execute this
try{ PImage result = ModelUtils.fromBase64(base64ImageString); if(result != null){ runwayResult = result; } }catch(Exception e){ e.printStackTrace(); }
Thank you @j3nsykes for reporting/details and @shiffman for the support.
Unfortunately the soonest I can look into this issue is next weekend.
In the meantime, @j3nsykes could you also post the full stack trace error message ?
This library currently only hands basic HTTP requests.
It could be that on newer Macs security is tighter and CORS needs to be handled (maybe https could be enforced) ?
I'm using the okhttp to handle the post requests.
(Maybe in the future I can look at swapping to HTTP-Requests-for-Processing
I don't think this library supports HTTPS / hosted models yet (🤞 hopefully this can remediated soon)).
I will need to get access to a M1 chip mac and need to debug using Postman and see what's going on/what request headers we need to change, etc. (Once a request works with okhttp, Postman should have a snippet generator for okhttp).
Apologies in advance I won't be able to fix this ASAP.
Thanks George .
An update on this now seems to point at the okhttp method of making HTTP requests and permission/security issues. In Processing 4.01A there was no more error information other than Error, disabling runwayDataEvent() null returned in the console.
My colleague had some examples set up for AttGAN and GPT2 using the HTTP-Requests-For-Processing library to communicate directly without the use of the RunwayML Processing lib.
These work without issue. I've since made an additional example for StyleGAN which returns and displays the base64 image without error.
This is still using the local network query not hosted models but a good Processing 4.01A solution. I will post a repo of examples in a few hours so others teaching with these tools can use them.
Templates for HTTP request from Processing 4.01A to RunwayML StyleGAN model. The same methods can be used for all other models by just changing the JSON input and output Strings per model.
Thanks to Catherine Weir who provided a template for the AttGAN model which these templates were adapted from.