CoInitializeEx return value RPC_E_CHANGED_MODE not handled correctly
hrydgard opened this issue · comments
On Windows, in NFD_OpenDialog, COM is initialized like this:
HRESULT coResult = ::CoInitializeEx(NULL,
::COINIT_APARTMENTTHREADED |
::COINIT_DISABLE_OLE1DDE );
Unfortunately, if COM has already been initialized in the process through COINIT_MULTITHREADED, the above call will fail (but COM will still work). See https://docs.microsoft.com/en-us/windows/win32/api/combaseapi/nf-combaseapi-coinitializeex.
And if the above call fails, NFD_OpenFileDialog just bails. This breaks our application.
I can think of three solutions:
- There could be an option to specify COINIT_MULTITHREADED (or that could be the default)
- CoInitialize should happen separately in some kind of NFD_Init, and the failure RPC_E_CHANGED_MODE should be tolerated
- NFD_OpenFileDialog should recognize the RPC_E_CHANGED_MODE error and just roll with it - COM still works if that is returned.
We use this library through https://github.com/saurvs/nfd-rs , but this is easily reproducible by adding a a CoInitializeEx(NULL, COINIT_MULTITHREADED);
call to main() in test_opendialog.c.
Thanks for finding this!
I had the same issue.
I changed the relevant lines in nfd_win.cpp to:
if (autoCoInit.Result() != RPC_E_CHANGED_MODE
&& !SUCCEEDED(autoCoInit.Result()))
{
....
so that it won't bail out if that specific error is returned from the CoInitializeEx
Fix in master branch. Version bumped to 1.1.5.