Windows compilation
mihaisebea opened this issue · comments
Hi
I want to look into windows compilation (https://github.com/mihaisebea/island)
Let me know if you have any thoughts/sugesstions.
The major issues that i see are
-
api_loader
-
file_watcher
-
job system
-
maybe swapchain since it's made to use x11 display
Other things that come to mind.
- glfw - ideally this should be a submodule since there is no system one on windows
- imgui should be added to the solution instead of cmake building it
- glm should be used directly instead of creating links?
- shader_compiler?
To expand a little more, now that I' ve reached my keyboard - I'm super impressed that you got it to run on Windows, and it would be wonderful if we could find a way to add Windows as a platform with hot reloading and all!
Re: some of the points above:
In terms of priorities, le_api_loader
and file_watcher
would be the amazing to get ported first - these are the key elements for hot-reloading. Both will need some Windows-specific code and there are some windows-specific apis: - the way .dlls are loaded on Windows is subtly different from Linux dynamic linking and loading - and I think on Windows, some methods will need to be annotated in headers with dllexport
and dllimport
directives, if I'm not mistaken.
In terms of job-system: Windows has a really fantastic system API for Fibers/co-routines built-in, if i remember correctly, and I hope we could use this directly instead of an assembly approach for the Windows port. That said, the job system is not yet wired up to many other subsystems, and therefore I would think it can be a lower priority for porting...
I think glfw is already a submodule, it should not need to be installed system-wide, if it expects a system-wide installation that would be good to know, because that would be a bug, and I'd like to track it down ;)
glm yes, i'm not very happy at the moment how this is used - it's also adding a ton of compilation time (because templates). I was thinking of using cglm but glm's syntax is just so nice and I wouldn't want to give that up... I believe the reason for using links was that glm headers depend on each others and reference each other via relative paths - that said, it was one of the first things added to the architecture, and I haven't touched it since. It'd be amazing if there was a more elegant way to include it.
swapchain i think only the direct swapchain makes reference to X11 directly, the other swapchains should be free of linux-specifics, or at least that would be the goal.
imgui i'd be super keen to find a more elegant way to include this - currently, it is compiled during the cmake configuration step of an island app - i was hoping that this be quite ergonomic in terms of making it easy to include in projects - but I don't like that imgui is such a special case, and I'm keen to make this a more "normal" module - you should only have to "pay" for imgui compilation when you're actually using it...
shaderc - I wanted to keep this as an external dependency, mainly because I used to manually compile it and the debug version of the library used to be huge on linux. When it came bundled with the Vulkan SDK a little bit later, it just felt very convenient to use it via the SDK. Are there any particular hiccups on Windows that make it tricky to use it via the SDK?
Some more things that spring to mind re hot-reloading:
There's also an issue with how Windows does file handles, which means that there we can't simply overwrite the "old" version of a library (and dependent files) when re-compiling. Stefan Reinalter of Live++ -fame has some great insights into how to deal with some aspects of this, which look well worth pursuing.
It should be possible and fairly trivial to create a scaled-down-to-the-minimum app which does not do any rendering, but purely focuses on hot-reloading which might help getting this aspect to work nicely on Windows. I think I'll set this up and will link a .zip file here so that it can be used for testing.
It would be fantastic if we could get the same seamless development experience on Windows as on Linux - where you can hot-reload, hit debug, and step through your new code. That said, just being able to compile and run an Island App on Windows, like you demonstrated, would be reaching a fantastic milestone and it makes me really happy to see your proof of concept :)
a windows branch?
Phew, what a long stream-of-consciousness email: forgive me!
If you made it this far, may I re-iterate - I'm super impressed that you got it to run on Windows so quickly, and I'd be super happy to liaise in any way so that we can get a Windows port working with hot-reloading, and everything. Maybe we can start a windows
branch or something? :)
glfw - indeed it's already a submodule. I had a small hickup with it as i probably ran cmake before doing git submodules update and it had a cmakelists file in there that wouldn't let me get the submodule.
glm - at the moment I just changed the include folder to
include_directories(${ISLAND_BASE_DIR}/3rdparty/src/glm/)
And modified the includes to glm/glm.hpp and it seems to work on my side.
swapchain - indeed i put the x11display behind a define and it all seems to compile and work
imgui - next on my list to get it working. I'll keep your comments in mind.
shaderc - i was trying to link with shaderc_shared but it complained about different iterator levels and different c runtimes. I guess I'll leave it static for now.
file_watcher Luckily I do have some experience with it on windows. It's not easy but certainly double.
api_loader Unfortunately I don't have much experience with dynamic libraries on windows but I hope it will be a fun learning experience ;)
windows branch - I'm not sure if it's needed? I would rather discuss the changes and merge them in master branch rather then having a branch that will quickly go obsolete.
I will try to clean up some of the hacks and commit the changes in my fork in case you want to play with it!
Thank you for you kind words and I'm happy you're so excited about having windows support.
That's great to know - can't wait to try things out on my windows machine.
In the meantime, I've managed to set up an isolated test case for hot-reloading (an app which doesen't do anything rendering-wise, but just uses the basic hot-reloading elements). I've packaged it up into this .zip file, which should extract into the apps/examples directory.
Hey @mihaisebea, just a quick question:
I'm working on windows today, and i've got all examples running in x64-Release
using latest visual studio 2019. I'm trying to get the Windows x64-Debug
target to work, but it fails with "unresolved externals" as error message. Are you seeing these errors too, when compiling the x64-Debug
target?
I assume that you passed -Dcmake_build_type=debug for this?
If you don't pass this cmake will generate a debug configuration but without PLUGINS_DYNAMIC set to on.
If you are getting linking errors is probably because PLUGINS_DYNAMIC is set to ON.
I have noticed that too, when starting to support plugins.
Seems we need to do the __dllexport on same variables.
I'll be looking into that at some point.
PLUGINS_DYNAMIC
: Of course!! That's good to know, cheers!
Seems we need to do the __dllexport on same variables:
i'm working on that right now ;)
Update: file_watcher
is now fully ported to Windows, and therefore shader hot-reloading works, too.
I'm tracking Windows port progress in a separate Readme, to keep things together.
Windows hot-reloading is now implemented and as of 3fc16f6 fairly reliable \o/.
@mihaisebea I'd be curious to see what you think on how hot-reloading works ;)
I'm closing this issue as I think the Windows port is in a usable state right now, and I'm tracking progress on some remaining features for the Windows port in the Windows specific readme.
Let's open separate issues should things come up with the Windows port.
Hi @tgfrerer, sorry for the late reply. Been quite busy.
Congratulations for making everything work! Great job!
Shaders hot reload works great.
Code hot reload i haven't manage to make it work. ninja seems to be recompiling but the changes don't seem to be applied. If i recompile the code in visual studio changes are applied so I assume for some reason the dll's are not reloaded.
I would try to use the internal filewatcher for the source as well and stop the update loop, unload the dll's, start a script that recompiled and when it finishes reload the dll's and restart the update loop. But I'm not sure how can this be done though.
interesting! so far, i only had a chance to test with visual studio 2019 - it's possible that hot reloading needs some
extra tricks outside that environment.
i assume you did checkout, and update submodules and did a clean build just in case?
it's a bit harder to pull off hot-reloading on windows because of the peculiar way windows keeps locks on files which are open, especially the PDB files are a nightmare, because vs doesn't release them even if the corresponding .dll has been fully unloaded.
if you could describe your setup (i'm mostly interested in IDE, and Visual Studio version) i can try to reproduce it on my windows machine and see if i can get to the root of it.
i might record a screencast of how i set things up on windows for hot-reloading (not sure that's the most straightforward method of showing how i set up my environment for hot-reloading), what do you think?
This is awesome! I tried it out with the latest and VS 2019 as well, all samples worked as well as a fresh app from the python generator. Both C++ and shaders hotload too :) Only ran into a couple (very) minor issues that I thought I'd list here so far. I'll list them here, though I'm happy to make issues out of these if it helps:
- create_app.py: had to switch the shebang to
#!/usr/bin/env python
for it to work on windows. - hot-reload.ps1 - ran into a problem with it where the build dir had a space in it (VS made it "x64-Debug (default)" for whatever reason)
- imgui example - alt + print screen causes an ex bad access (unknown key -1)
I'm excited to dive deeper into this engine now that it is ported to Windows where I work every day..
@richardeakin that's great to hear!
It would be fantastic to have separate github issues - or maybe pull requests? - for these issues. It seems you found a way to fix most of them on your machine already and it would be great to have these fixes applied :)
Hint for the python project generator: you can use it to take any project as a template by giving it the name of the project you want to duplicate as the -t
(template) parameter, and a matching -T
(template folder) parameter. The template folder needs to be given either absolute, or relative to the scripts
folder...