When using tokio and HuggingFaceRemote it breaks dropping the runtime
gadLinux opened this issue · comments
It seems that it's trying to connect to the outside but fails to do it so and it breaks.
[2024-02-08T17:39:28Z INFO main] "Starting assistant"
thread 'main' panicked at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/blocking/shutdown.rs:51:21:
Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context.
stack backtrace:
0: rust_begin_unwind
at /rustc/7ffc697ce10f19447c0ce338428ae4b9bc0c041c/library/std/src/panicking.rs:647:5
1: core::panicking::panic_fmt
at /rustc/7ffc697ce10f19447c0ce338428ae4b9bc0c041c/library/core/src/panicking.rs:72:14
2: tokio::runtime::blocking::shutdown::Receiver::wait
at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/blocking/shutdown.rs:51:21
3: tokio::runtime::blocking::pool::BlockingPool::shutdown
at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/blocking/pool.rs:261:12
4: <tokio::runtime::blocking::pool::BlockingPool as core::ops::drop::Drop>::drop
at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/blocking/pool.rs:278:9
5: core::ptr::drop_in_place<tokio::runtime::blocking::pool::BlockingPool>
at /rustc/7ffc697ce10f19447c0ce338428ae4b9bc0c041c/library/core/src/ptr/mod.rs:507:1
6: core::ptr::drop_in_place<tokio::runtime::runtime::Runtime>
at /rustc/7ffc697ce10f19447c0ce338428ae4b9bc0c041c/library/core/src/ptr/mod.rs:507:1
7: reqwest::blocking::wait::enter
at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.23/src/blocking/wait.rs:76:21
8: reqwest::blocking::wait::timeout
at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.23/src/blocking/wait.rs:13:5
9: reqwest::blocking::client::ClientHandle::new
at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.23/src/blocking/client.rs:1075:15
10: reqwest::blocking::client::ClientBuilder::build
at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.23/src/blocking/client.rs:103:9
11: cached_path::cache::CacheBuilder::build
at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cached-path-0.6.1/src/cache.rs:130:27
12: tokenizers::utils::from_pretrained::from_pretrained
at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.13.4/src/utils/from_pretrained.rs:151:17
13: tokenizers::tokenizer::Tokenizer::from_pretrained
at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokenizers-0.13.4/src/tokenizer/mod.rs:445:30
14: llm_base::tokenizer::TokenizerSource::retrieve
at /home/gaguilar/.cargo/git/checkouts/llm-679a994ae16a26de/00aaf4a/crates/llm-base/src/tokenizer/mod.rs:116:17
15: llm_base::loader::load
at /home/gaguilar/.cargo/git/checkouts/llm-679a994ae16a26de/00aaf4a/crates/llm-base/src/loader.rs:382:21
16: llm::loader::load
at /home/gaguilar/.cargo/git/checkouts/llm-679a994ae16a26de/00aaf4a/crates/llm/src/loader.rs:26:5
17: domain::services::build_model
at ./domain/src/services/mod.rs:48:17
18: <domain::services::AIModelProvider as core::default::Default>::default
at ./domain/src/services/mod.rs:39:20
19: infra::rest::router::build_router::{{closure}}
at ./infra/src/rest/router.rs:50:25
20: main::main::{{closure}}
at ./api/src/bin/main.rs:31:51
21: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/park.rs:281:63
22: tokio::runtime::coop::with_budget
at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/coop.rs:107:5
23: tokio::runtime::coop::budget
at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/coop.rs:73:5
24: tokio::runtime::park::CachedParkThread::block_on
at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/park.rs:281:31
25: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/context/blocking.rs:66:9
26: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/scheduler/multi_thread/mod.rs:87:13
27: tokio::runtime::context::runtime::enter_runtime
at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/context/runtime.rs:65:16
28: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/scheduler/multi_thread/mod.rs:86:9
29: tokio::runtime::runtime::Runtime::block_on
at /home/gaguilar/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/runtime.rs:350:45
30: main::main
at ./api/src/bin/main.rs:39:5
31: core::ops::function::FnOnce::call_once
at /rustc/7ffc697ce10f19447c0ce338428ae4b9bc0c041c/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Hmm, that's quite odd. I don't think this is an issue with llm
in particular; this looks to be more a strange interaction with tokenizers
and your environment. (We don't explicitly do anything with async
- it looks like tokenizers
spins up a Tokio runtime to fetch the file and doesn't clean it up properly).
If this is an issue for your application, I'd suggest fetching the tokenizer or storing it yourself. The remote functionality is provided as a convenience, but it comes with a fair few caveats; this appears to be one of them 😦