Win11x64 STATUS_ACCESS_VIOLATION within OrtGetApiBase

Question

Win11x64 STATUS_ACCESS_VIOLATION within OrtGetApiBase

davehorner opened this issue 9 months ago · comments

I'm not sure what to think. I am able to use the library without issue when I use the http server of the application. When I try to use it via the directory, it gives me a (exit code: 0xc0000005, STATUS_ACCESS_VIOLATION) error.

I have used ort successfully in release and debug. I have tried the load-dynamic feature. I've checked all the dependencies look OK for ortruntime and it works. but there are cases where it STATUS_ACCESS_VIOLATIONs.

Exception 0xc0000005 encountered at address 0x7ffbd9b95253: Access violation reading location 0x7ffbad5ed490

OrtGetApiBase (@OrtGetApiBase:249)
OrtSessionOptionsAppendExecutionProvider_CUDA (@OrtSessionOptionsAppendExecutionProvider_CUDA:24955)
OrtSessionOptionsAppendExecutionProvider_CUDA (@OrtSessionOptionsAppendExecutionProvider_CUDA:24955)
void ort::session::impl$5::drop(struct ort::session::SessionPointerHolder *) (c:\Users\dhorner\.cargo\registry\src\index.crates.io-6f17d22bba15001f\ort-1.16.3\src\session.rs:550)
void core::ptr::drop_in_place<ort::session::SessionPointerHolder>(struct ort::session::SessionPointerHolder *) (@core::ptr::drop_in_place<ort::session::SessionPointerHolder>:6)
void alloc::sync::Arc<ort::session::SessionPointerHolder,alloc::alloc::Global>::drop_slow<ort::session::SessionPointerHolder,alloc::alloc::Global>() (@alloc::sync::Arc<T,A>::drop_slow:10)
void alloc::sync::impl$33::drop<ort::session::SessionPointerHolder,alloc::alloc::Global>(struct alloc::sync::Arc<ort::session::SessionPointerHolder,alloc::alloc::Global> *) (@<alloc::sync::Arc<T,A> as core::ops::drop::Drop>::drop:23)
void core::ptr::drop_in_place<alloc::sync::Arc<ort::session::SessionPointerHolder,alloc::alloc::Global> >(struct alloc::sync::Arc<ort::session::SessionPointerHolder,alloc::alloc::Global> *) (@core::ptr::drop_in_place<alloc::sync::Arc<ort::session::SessionPointerHolder>>:6)
void core::ptr::drop_in_place<ort::session::Session>(struct ort::session::Session *) (@core::ptr::drop_in_place<ort::session::Session>:17)
union enum2$<core::task::poll::Poll<enum2$<core::result::Result<tuple$<>,anyhow::Error> > > > rust_background_removal::main::async_block$0(struct core::pin::Pin<ref_mut$<enum2$<rust_background_removal::main::async_block_env$0> > >, struct core::task::wake::Context *) (c:\w\rust-background-removal\src\main.rs:200)
static union enum2$<core::task::poll::Poll<enum2$<core::result::Result<tuple$<>,anyhow::Error> > > > tokio::runtime::park::impl$4::block_on::closure$0<enum2$<rust_background_removal::main::async_block_env$0> >(struct tokio::runtime::park::impl$4::block_on::closure_env$0<enum2$<rust_background_removal::main::async_block_env$0> >) (c:\Users\dhorner\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokio-1.33.0\src\runtime\park.rs:282)
union enum2$<core::result::Result<enum2$<core::result::Result<tuple$<>,anyhow::Error> >,std::thread::local::AccessError> > tokio::runtime::park::CachedParkThread::block_on<enum2$<rust_background_removal::main::async_block_env$0> >(union enum2$<rust_background_removal::main::async_block_env$0>) (c:\Users\dhorner\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokio-1.33.0\src\runtime\park.rs:282)
union enum2$<core::result::Result<enum2$<core::result::Result<tuple$<>,anyhow::Error> >,std::thread::local::AccessError> > tokio::runtime::context::blocking::BlockingRegionGuard::block_on<enum2$<rust_background_removal::main::async_block_env$0> >(union enum2$<rust_background_removal::main::async_block_env$0>) (c:\Users\dhorner\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokio-1.33.0\src\runtime\context\blocking.rs:66)
tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}} (c:\Users\dhorner\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokio-1.33.0\src\runtime\scheduler\multi_thread\mod.rs:87)
union enum2$<core::result::Result<tuple$<>,anyhow::Error> > tokio::runtime::context::runtime::enter_runtime<tokio::runtime::scheduler::multi_thread::impl$0::block_on::closure_env$0<enum2$<rust_background_removal::main::async_block_env$0> >,enum2$<core::result::Result<tuple$<>,anyhow::Error> > >(union enum2$<tokio::runtime::scheduler::Handle> *, bool, struct tokio::runtime::scheduler::multi_thread::impl$0::block_on::closure_env$0<enum2$<rust_background_removal::main::async_block_env$0> >, struct core::panic::location::Location *) (c:\Users\dhorner\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokio-1.33.0\src\runtime\context\runtime.rs:65)
union enum2$<core::result::Result<tuple$<>,anyhow::Error> > tokio::runtime::scheduler::multi_thread::MultiThread::block_on<enum2$<rust_background_removal::main::async_block_env$0> >(union enum2$<tokio::runtime::scheduler::Handle> *, union enum2$<rust_background_removal::main::async_block_env$0>) (c:\Users\dhorner\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokio-1.33.0\src\runtime\scheduler\multi_thread\mod.rs:86)
union enum2$<core::result::Result<tuple$<>,anyhow::Error> > tokio::runtime::runtime::Runtime::block_on<enum2$<rust_background_removal::main::async_block_env$0> >(union enum2$<rust_background_removal::main::async_block_env$0>, struct core::panic::location::Location *) (c:\Users\dhorner\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokio-1.33.0\src\runtime\runtime.rs:350)
static union enum2$<core::result::Result<tuple$<>,anyhow::Error> > rust_background_removal::main() (c:\w\rust-background-removal\src\main.rs:131)
union enum2$<core::result::Result<tuple$<>,anyhow::Error> > core::ops::function::FnOnce::call_once<enum2$<core::result::Result<tuple$<>,anyhow::Error> > (*)(),tuple$<> >( *) (@core::ops::function::FnOnce::call_once:6)
union enum2$<core::result::Result<tuple$<>,anyhow::Error> > std::sys_common::backtrace::__rust_begin_short_backtrace<enum2$<core::result::Result<tuple$<>,anyhow::Error> > (*)(),enum2$<core::result::Result<tuple$<>,anyhow::Error> > >( *) (@std::sys_common::backtrace::__rust_begin_short_backtrace:6)

https://github.com/davehorner/rust-background-removal is the code if that helps. I know not much to go on, but I did try and capture as much detail as possible.

Carson M. · Answer 1 · Sat Dec 23 2023 11:39:18 GMT+0800 (China Standard Time)

cargo run with that repo is working fine for me. How exactly are you using it?

David Horner · Answer 2 · Sat Dec 23 2023 11:49:47 GMT+0800 (China Standard Time)

The --http option runs a server and that works fine. Just running it with no options causes it to do an access violation. --help works fine.
I can debug and it gets to if !args.input_file.is_empty() { the next step hops to the attribute on main

#[tokio::main(worker_threads = 10)]
// #[tokio::main(flavor = "current_thread")]
//#[tokio::main(worker_threads = 1)]

which I had set to 10. I just now tried 1 and current_thread and that did not resolve it.

When I debug, it looks like it is actually doing the access violation after the main return Ok(())! Which might explain why the http server works fine...

Carson M. · Answer 3 · Sat Dec 23 2023 11:55:41 GMT+0800 (China Standard Time)

I cannot reproduce this with/without CUDA, with/without --http, or with any of the configurations of tokio::main you mentioned.

Could you set the environment variable set RUST_LOG="ort=debug"? Maybe that will give some info as to what's happening.

David Horner · Answer 4 · Sat Dec 23 2023 12:02:40 GMT+0800 (China Standard Time)

thanks for trying! This is after set RUST_LOG=ort=debug and running with no options.

2023-12-23T04:01:46.920168Z DEBUG ort::environment: Environment not yet initialized, creating a new one
2023-12-23T04:01:46.945897Z DEBUG ort::environment: Environment created env_ptr="0x12d4c5a3ee0"
2023-12-23T04:01:47.195056Z  INFO apply_execution_providers: ort::execution_providers: Successfully registered `CUDAExecutionProvider`
2023-12-23T04:01:47.728918Z DEBUG drop{self=Environment { env: Mutex { data: EnvironmentSingleton { name: "default", env_ptr: 0x12d4c5a3ee0 }, poisoned: false, .. }, execution_providers: [] }}: ort::environment: Dropping environment global_arc_count=2
2023-12-23T04:01:47.729263Z DEBUG drop{self=Environment { env: Mutex { data: EnvironmentSingleton { name: "default", env_ptr: 0x12d4c5a3ee0 }, poisoned: false, .. }, execution_providers: [] }}: ort::environment: Releasing environment global_arc_count=2
error: process didn't exit successfully: `target\release\rust-background-removal.exe` (exit code: 0xc0000005, STATUS_ACCESS_VIOLATION)

Carson M. · Answer 5 · Sat Dec 23 2023 12:07:21 GMT+0800 (China Standard Time)

The environment being dropped prematurely is either the culprit or a symptom. I don't know why I'm not seeing that on my end, though.

Does it still crash if you remove ort's cuda feature from the Cargo.toml?

David Horner · Answer 6 · Sat Dec 23 2023 12:09:28 GMT+0800 (China Standard Time)

    Finished release [optimized] target(s) in 31.45s
     Running `target\release\rust-background-removal.exe`
2023-12-23T04:08:35.594557Z DEBUG ort::environment: Environment not yet initialized, creating a new one
2023-12-23T04:08:35.620018Z DEBUG ort::environment: Environment created env_ptr="0x1cfc99c8910"
2023-12-23T04:08:35.620428Z DEBUG apply_execution_providers: ort::execution_providers: Execution provider `CUDAExecutionProvider` was not registered because its corresponding Cargo feature is disabled.
2023-12-23T04:08:35.620818Z DEBUG apply_execution_providers: ort::execution_providers: Execution provider `CoreMLExecutionProvider` was not registered because its corresponding Cargo feature is disabled.
2023-12-23T04:08:35.621495Z  INFO apply_execution_providers: ort::execution_providers: Successfully registered `CPUExecutionProvider`
2023-12-23T04:08:35.958028Z DEBUG drop{self=Environment { env: Mutex { data: EnvironmentSingleton { name: "default", env_ptr: 0x1cfc99c8910 }, poisoned: false, .. }, execution_providers: [] }}: ort::environment: Dropping environment global_arc_count=2
2023-12-23T04:08:35.958323Z DEBUG drop{self=Environment { env: Mutex { data: EnvironmentSingleton { name: "default", env_ptr: 0x1cfc99c8910 }, poisoned: false, .. }, execution_providers: [] }}: ort::environment: Releasing environment global_arc_count=2
Error: The system cannot find the path specified. (os error 3)
error: process didn't exit successfully: `target\release\rust-background-removal.exe` (exit code: 1)

it seems cuda is the culprit.

Carson M. · Answer 7 · Sat Dec 23 2023 12:13:12 GMT+0800 (China Standard Time)

Right, I remember running into something similar with CUDA in the past. It could be due to missing zlibwapi.dll or cuDNN, or you have CUDA 12.x installed.

David Horner · Answer 8 · Sat Dec 23 2023 12:15:57 GMT+0800 (China Standard Time)

was careful to install 11 only on this actually. zlibwapi.dll would be a dependency of what? it loads it and works fine in http, it seems to be tear down is off. I will play here some more, now that its running under cpu its not working as I would expect... oh fun.

Carson M. · Answer 9 · Sat Dec 23 2023 12:21:22 GMT+0800 (China Standard Time)

zlibwapi.dll would be a dependency of what?

CUDA, for whatever godforsaken reason 😅

it loads it and works fine in http

In that case, could it be due to threading? Sessions registered with CUDA really don't like being sent across threads and can cause a STATUS_ACCESS_VIOLATION like this. Though, again, I don't know why I can't reproduce it, and it seems like if that were the case it should be the other way around (normal without, crashing with HTTP)

David Horner · Answer 10 · Sat Dec 23 2023 23:16:32 GMT+0800 (China Standard Time)

Well I don't think I have zlibwapi.dll available on the PATH. It's a wild goose chase trying to figure out what I should use for that file. I would expect to see errors about it specifically during the load and not during unload.

2023-12-23T15:05:18.542847Z DEBUG ort::environment: Environment not yet initialized, creating a new one
2023-12-23T15:05:18.567392Z DEBUG ort::environment: Environment created env_ptr="0x152d92ff920"
2023-12-23T15:05:18.877945Z  INFO apply_execution_providers: ort::execution_providers: Successfully registered `CUDAExecutionProvider`
2023-12-23T15:05:19.458356Z DEBUG drop{self=Environment { env: Mutex { data: EnvironmentSingleton { name: "default", env_ptr: 0x152d92ff920 }, poisoned: false, .. }, execution_providers: [] }}: ort::environment: Dropping environment global_arc_count=2
2023-12-23T15:05:19.458628Z DEBUG drop{self=Environment { env: Mutex { data: EnvironmentSingleton { name: "default", env_ptr: 0x152d92ff920 }, poisoned: false, .. }, execution_providers: [] }}: ort::environment: Releasing environment global_arc_count=2
error: process didn't exit successfully: `target\release\rust-background-removal.exe -I """` (exit code: 0xc0000005, STATUS_ACCESS_VIOLATION)

#[tokio::main] operates in the same manner, I thought it was the threads but I just don't know.
We can close this if you want, I don't know what else to do with it, just know its occurring.
It does work though and thank you for the software.