Giters

tgfrerer / island

🌋🐎 Project Island is an experimental, hot-reloading Vulkan Renderer for Linux and Windows, written in C/C++.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Windows compilation

mihaisebea opened this issue · comments

commented

Hi
I want to look into windows compilation (https://github.com/mihaisebea/island)
Let me know if you have any thoughts/sugesstions.
The major issues that i see are

  • api_loader

  • file_watcher

  • job system

  • maybe swapchain since it's made to use x11 display

Other things that come to mind.

  • glfw - ideally this should be a submodule since there is no system one on windows
  • imgui should be added to the solution instead of cmake building it
  • glm should be used directly instead of creating links?
  • shader_compiler?
commented

So far i ignored the points above and got something running :)

image

commented
commented

To expand a little more, now that I' ve reached my keyboard - I'm super impressed that you got it to run on Windows, and it would be wonderful if we could find a way to add Windows as a platform with hot reloading and all!

Re: some of the points above:

In terms of priorities, le_api_loader and file_watcher would be the amazing to get ported first - these are the key elements for hot-reloading. Both will need some Windows-specific code and there are some windows-specific apis: - the way .dlls are loaded on Windows is subtly different from Linux dynamic linking and loading - and I think on Windows, some methods will need to be annotated in headers with dllexport and dllimport directives, if I'm not mistaken.

In terms of job-system: Windows has a really fantastic system API for Fibers/co-routines built-in, if i remember correctly, and I hope we could use this directly instead of an assembly approach for the Windows port. That said, the job system is not yet wired up to many other subsystems, and therefore I would think it can be a lower priority for porting...

I think glfw is already a submodule, it should not need to be installed system-wide, if it expects a system-wide installation that would be good to know, because that would be a bug, and I'd like to track it down ;)

glm yes, i'm not very happy at the moment how this is used - it's also adding a ton of compilation time (because templates). I was thinking of using cglm but glm's syntax is just so nice and I wouldn't want to give that up... I believe the reason for using links was that glm headers depend on each others and reference each other via relative paths - that said, it was one of the first things added to the architecture, and I haven't touched it since. It'd be amazing if there was a more elegant way to include it.

swapchain i think only the direct swapchain makes reference to X11 directly, the other swapchains should be free of linux-specifics, or at least that would be the goal.

imgui i'd be super keen to find a more elegant way to include this - currently, it is compiled during the cmake configuration step of an island app - i was hoping that this be quite ergonomic in terms of making it easy to include in projects - but I don't like that imgui is such a special case, and I'm keen to make this a more "normal" module - you should only have to "pay" for imgui compilation when you're actually using it...

shaderc - I wanted to keep this as an external dependency, mainly because I used to manually compile it and the debug version of the library used to be huge on linux. When it came bundled with the Vulkan SDK a little bit later, it just felt very convenient to use it via the SDK. Are there any particular hiccups on Windows that make it tricky to use it via the SDK?

Some more things that spring to mind re hot-reloading:

There's also an issue with how Windows does file handles, which means that there we can't simply overwrite the "old" version of a library (and dependent files) when re-compiling. Stefan Reinalter of Live++ -fame has some great insights into how to deal with some aspects of this, which look well worth pursuing.

It should be possible and fairly trivial to create a scaled-down-to-the-minimum app which does not do any rendering, but purely focuses on hot-reloading which might help getting this aspect to work nicely on Windows. I think I'll set this up and will link a .zip file here so that it can be used for testing.

It would be fantastic if we could get the same seamless development experience on Windows as on Linux - where you can hot-reload, hit debug, and step through your new code. That said, just being able to compile and run an Island App on Windows, like you demonstrated, would be reaching a fantastic milestone and it makes me really happy to see your proof of concept :)

a windows branch?

Phew, what a long stream-of-consciousness email: forgive me!

If you made it this far, may I re-iterate - I'm super impressed that you got it to run on Windows so quickly, and I'd be super happy to liaise in any way so that we can get a Windows port working with hot-reloading, and everything. Maybe we can start a windows branch or something? :)

commented

glfw - indeed it's already a submodule. I had a small hickup with it as i probably ran cmake before doing git submodules update and it had a cmakelists file in there that wouldn't let me get the submodule.

glm - at the moment I just changed the include folder to

    include_directories(${ISLAND_BASE_DIR}/3rdparty/src/glm/)

And modified the includes to glm/glm.hpp and it seems to work on my side.

swapchain - indeed i put the x11display behind a define and it all seems to compile and work

imgui - next on my list to get it working. I'll keep your comments in mind.

shaderc - i was trying to link with shaderc_shared but it complained about different iterator levels and different c runtimes. I guess I'll leave it static for now.

file_watcher Luckily I do have some experience with it on windows. It's not easy but certainly double.

api_loader Unfortunately I don't have much experience with dynamic libraries on windows but I hope it will be a fun learning experience ;)

windows branch - I'm not sure if it's needed? I would rather discuss the changes and merge them in master branch rather then having a branch that will quickly go obsolete.

I will try to clean up some of the hacks and commit the changes in my fork in case you want to play with it!

Thank you for you kind words and I'm happy you're so excited about having windows support.

commented

That's great to know - can't wait to try things out on my windows machine.

In the meantime, I've managed to set up an isolated test case for hot-reloading (an app which doesen't do anything rendering-wise, but just uses the basic hot-reloading elements). I've packaged it up into this .zip file, which should extract into the apps/examples directory.

commented

Hey @mihaisebea, just a quick question:

I'm working on windows today, and i've got all examples running in x64-Release using latest visual studio 2019. I'm trying to get the Windows x64-Debug target to work, but it fails with "unresolved externals" as error message. Are you seeing these errors too, when compiling the x64-Debug target?

commented

I assume that you passed -Dcmake_build_type=debug for this?
If you don't pass this cmake will generate a debug configuration but without PLUGINS_DYNAMIC set to on.
If you are getting linking errors is probably because PLUGINS_DYNAMIC is set to ON.
I have noticed that too, when starting to support plugins.
Seems we need to do the __dllexport on same variables.
I'll be looking into that at some point.

commented

PLUGINS_DYNAMIC : Of course!! That's good to know, cheers!

Seems we need to do the __dllexport on same variables:

i'm working on that right now ;)

commented

Update: Windows Debug_x64 targets work now, with dynamic libraries. Working on porting the file watcher to windows now, so that shaders can be hot-reloaded...
bandicam 2020-09-03 13-39-01-080

commented

Update: file_watcher is now fully ported to Windows, and therefore shader hot-reloading works, too.

I'm tracking Windows port progress in a separate Readme, to keep things together.

commented

Windows hot-reloading is now implemented and as of 3fc16f6 fairly reliable \o/.

@mihaisebea I'd be curious to see what you think on how hot-reloading works ;)

I'm closing this issue as I think the Windows port is in a usable state right now, and I'm tracking progress on some remaining features for the Windows port in the Windows specific readme.

Let's open separate issues should things come up with the Windows port.

commented

Hi @tgfrerer, sorry for the late reply. Been quite busy.
Congratulations for making everything work! Great job!
Shaders hot reload works great.
Code hot reload i haven't manage to make it work. ninja seems to be recompiling but the changes don't seem to be applied. If i recompile the code in visual studio changes are applied so I assume for some reason the dll's are not reloaded.
I would try to use the internal filewatcher for the source as well and stop the update loop, unload the dll's, start a script that recompiled and when it finishes reload the dll's and restart the update loop. But I'm not sure how can this be done though.

commented

interesting! so far, i only had a chance to test with visual studio 2019 - it's possible that hot reloading needs some
extra tricks outside that environment.

i assume you did checkout, and update submodules and did a clean build just in case?

it's a bit harder to pull off hot-reloading on windows because of the peculiar way windows keeps locks on files which are open, especially the PDB files are a nightmare, because vs doesn't release them even if the corresponding .dll has been fully unloaded.

if you could describe your setup (i'm mostly interested in IDE, and Visual Studio version) i can try to reproduce it on my windows machine and see if i can get to the root of it.

i might record a screencast of how i set things up on windows for hot-reloading (not sure that's the most straightforward method of showing how i set up my environment for hot-reloading), what do you think?

commented

This is awesome! I tried it out with the latest and VS 2019 as well, all samples worked as well as a fresh app from the python generator. Both C++ and shaders hotload too :) Only ran into a couple (very) minor issues that I thought I'd list here so far. I'll list them here, though I'm happy to make issues out of these if it helps:

  • create_app.py: had to switch the shebang to #!/usr/bin/env python for it to work on windows.
  • hot-reload.ps1 - ran into a problem with it where the build dir had a space in it (VS made it "x64-Debug (default)" for whatever reason)
  • imgui example - alt + print screen causes an ex bad access (unknown key -1)

I'm excited to dive deeper into this engine now that it is ported to Windows where I work every day..

commented

@richardeakin that's great to hear!

It would be fantastic to have separate github issues - or maybe pull requests? - for these issues. It seems you found a way to fix most of them on your machine already and it would be great to have these fixes applied :)

Hint for the python project generator: you can use it to take any project as a template by giving it the name of the project you want to duplicate as the -t (template) parameter, and a matching -T (template folder) parameter. The template folder needs to be given either absolute, or relative to the scripts folder...