Application startup failed
fatualux opened this issue · comments
Hi and thanks for your work.
I opened this issue, and I thought I had solved the problem, but now the application fails again.
I cloned the repository with git clone https://github.com/cheshire-cat-ai/local-cat/, then I ran:
cd local-cat && docker compose up
And what I got is:
WARN[0000] /home/fz/Documents/Github/local-cat/docker-compose.yml: `version` is obsolete
[+] Running 26/26
✔ cheshire-cat-vector-memory Pulled 131.3s
✔ b0a0cf830b12 Pull complete 71.6s
✔ 7bff4bded8c0 Pull complete 77.6s
✔ 7193c073585c Pull complete 78.8s
✔ e0b125224479 Pull complete 118.5s
✔ 836f9676abac Pull complete 119.9s
✔ 28486c15c557 Pull complete 121.5s
✔ 839fce5e19cb Pull complete 123.6s
✔ cheshire-cat-core Pulled 248.7s
✔ f03b40093957 Already exists 0.0s
✔ 05c2151a829c Already exists 0.0s
✔ 94970648c551 Already exists 0.0s
✔ cf2e0f30894f Already exists 0.0s
✔ 661be8ca6397 Already exists 0.0s
✔ 241528821bd8 Pull complete 95.1s
✔ eb1b5b418cb1 Pull complete 96.5s
✔ c3576b7aa0a1 Pull complete 98.3s
✔ 826a460480fb Pull complete 100.0s
✔ 4f4fb700ef54 Pull complete 126.6s
✔ acbeb0db44d4 Pull complete 240.3s
✔ 48d941c7943d Pull complete 242.8s
✔ ebbede783fda Pull complete 244.1s
✔ ollama Pulled 297.8s
✔ a8b1c5f80c2d Pull complete 105.7s
✔ 630d5089a4c7 Pull complete 146.1s
✔ 5cf6c1ae27da Pull complete 292.0s
[+] Running 3/3
✔ Container cheshire_cat_vector_memory Created 1.7s
✔ Container ollama_cat Created 1.6s
✔ Container cheshire_cat_core Created 0.9s
So, images correctly pulled, and containers created.
Then I got some errors:
cheshire_cat_vector_memory | _ _
cheshire_cat_vector_memory | __ _ __| |_ __ __ _ _ __ | |_
cheshire_cat_vector_memory | / _` |/ _` | '__/ _` | '_ \| __|
cheshire_cat_vector_memory | | (_| | (_| | | | (_| | | | | |_
cheshire_cat_vector_memory | \__, |\__,_|_| \__,_|_| |_|\__|
cheshire_cat_vector_memory | |_|
cheshire_cat_vector_memory |
cheshire_cat_vector_memory | Version: 1.9.1, build: 97c107f2
cheshire_cat_vector_memory | Access web UI at http://localhost:6333/dashboard
cheshire_cat_vector_memory |
cheshire_cat_vector_memory | 2024-06-09T21:39:58.286105Z INFO storage::content_manager::consensus::persistent: Initializing new raft state at ./storage/raft_state.json
cheshire_cat_vector_memory | 2024-06-09T21:39:58.860845Z INFO qdrant: Distributed mode disabled
cheshire_cat_vector_memory | 2024-06-09T21:39:58.898804Z INFO qdrant: Telemetry reporting enabled, id: 25966ae8-31c2-4042-a2b7-f52f8a9eb093
cheshire_cat_vector_memory | 2024-06-09T21:39:59.089491Z INFO qdrant::tonic: Qdrant gRPC listening on 6334
cheshire_cat_vector_memory | 2024-06-09T21:39:59.089502Z INFO qdrant::tonic: TLS disabled for gRPC API
cheshire_cat_vector_memory | 2024-06-09T21:39:59.171819Z INFO qdrant::actix: TLS disabled for REST API
cheshire_cat_vector_memory | 2024-06-09T21:39:59.172594Z INFO qdrant::actix: Qdrant HTTP listening on 6333
cheshire_cat_vector_memory | 2024-06-09T21:39:59.172602Z INFO actix_server::builder: Starting 11 workers
cheshire_cat_vector_memory | 2024-06-09T21:39:59.187493Z INFO actix_server::server: Actix runtime found; starting in Actix runtime
ollama_cat | time=2024-06-09T21:40:00.098Z level=INFO source=types.go:71 msg="inference compute" id=GPU-4151ce35-c2d3-a276-826b-00260b538d12 library=cuda compute=7.5 driver=12.4 name="NVIDIA GeForce RTX 2060" total="5.8 GiB" available="3.2 GiB"
cheshire_cat_core | --- Logging error in Loguru Handler #1 ---
cheshire_cat_core | Record was: {'elapsed': datetime.timedelta(seconds=5, microseconds=376415), 'exception': None, 'extra': {}, 'file': (name='embedding.py', path='/usr/local/lib/python3.10/site-packages/fastembed/embedding.py'), 'function': '<module>', 'level': (name='WARNING', no=30, icon='⚠️'), 'line': 7, 'message': 'DefaultEmbedding, FlagEmbedding, JinaEmbedding are deprecated.Use from fastembed import TextEmbedding instead.', 'module': 'embedding', 'name': 'fastembed.embedding', 'process': (id=7, name='MainProcess'), 'thread': (id=136294234482496, name='MainThread'), 'time': datetime(2024, 6, 9, 21, 40, 13, 238897, tzinfo=datetime.timezone(datetime.timedelta(0), 'UTC'))}
cheshire_cat_core | Traceback (most recent call last):
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/loguru/_handler.py", line 184, in emit
cheshire_cat_core | formatted = precomputed_format.format_map(formatter_record)
cheshire_cat_core | KeyError: 'original_name'
cheshire_cat_core | --- End of logging error ---
cheshire_cat_core | --- Logging error in Loguru Handler #1 ---
cheshire_cat_core | Record was: {'elapsed': datetime.timedelta(seconds=1, microseconds=288552), 'exception': None, 'extra': {}, 'file': (name='embedding.py', path='/usr/local/lib/python3.10/site-packages/fastembed/embedding.py'), 'function': '<module>', 'level': (name='WARNING', no=30, icon='⚠️'), 'line': 7, 'message': 'DefaultEmbedding, FlagEmbedding, JinaEmbedding are deprecated.Use from fastembed import TextEmbedding instead.', 'module': 'embedding', 'name': 'fastembed.embedding', 'process': (id=31, name='SpawnProcess-1'), 'thread': (id=132674308892480, name='MainThread'), 'time': datetime(2024, 6, 9, 21, 40, 19, 762763, tzinfo=datetime.timezone(datetime.timedelta(0), 'UTC'))}
cheshire_cat_core | Traceback (most recent call last):
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/loguru/_handler.py", line 184, in emit
cheshire_cat_core | formatted = precomputed_format.format_map(formatter_record)
cheshire_cat_core | KeyError: 'original_name'
cheshire_cat_core | --- End of logging error ---
Finally, the shell returned this:
config.json: 100%|██████████| 742/742 [00:00<00:00, 5.96MB/s]
.gitattributes: 100%|██████████| 1.52k/1.52k [00:00<00:00, 14.2MB/s]
vocab.txt: 100%|██████████| 232k/232k [00:00<00:00, 582kB/s]/s]
ort_config.json: 100%|██████████| 1.27k/1.27k [00:00<00:00, 12.0MB/s]
tokenizer_config.json: 100%|██████████| 1.24k/1.24k [00:00<00:00, 13.3MB/s]
special_tokens_map.json: 100%|██████████| 695/695 [00:00<00:00, 4.87MB/s]
tokenizer.json: 100%|██████████| 711k/711k [00:00<00:00, 882kB/s]]
model_optimized.onnx: 100%|██████████| 668M/668M [03:48<00:00, 2.93MB/s]
Fetching 8 files: 100%|██████████| 8/8 [03:48<00:00, 28.59s/it]
cheshire_cat_core | /usr/local/lib/python3.10/site-packages/qdrant_client/qdrant_remote.py:123: UserWarning: Api key is used with unsecure connection.00:00, 2.89MB/s]
cheshire_cat_core | warnings.warn("Api key is used with unsecure connection.")
cheshire_cat_vector_memory | 2024-06-09T21:44:19.804822Z INFO actix_web::middleware::logger: 172.19.0.4 "GET /collections HTTP/1.1" 200 82 "-" "python-httpx/0.27.0" 0.032041
cheshire_cat_core | [2024-06-09 21:44:19.838] WARNING cat.memory.vector_memory_collection.VectorMemoryCollection.create_collection::104
cheshire_cat_core | "Creating collection 'episodic' ..."
cheshire_cat_vector_memory | 2024-06-09T21:44:19.857125Z INFO storage::content_manager::toc::collection_meta_ops: Deleting collection episodic
cheshire_cat_vector_memory | 2024-06-09T21:44:19.857215Z INFO actix_web::middleware::logger: 172.19.0.4 "DELETE /collections/episodic HTTP/1.1" 200 72 "-" "python-httpx/0.27.0" 0.017867
cheshire_cat_vector_memory | 2024-06-09T21:44:19.877780Z INFO storage::content_manager::toc::collection_meta_ops: Creating collection episodic
cheshire_cat_core | ERROR: Traceback (most recent call last):
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions
cheshire_cat_core | yield
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 233, in handle_request
cheshire_cat_core | resp = self._pool.handle_request(req)
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request
cheshire_cat_core | raise exc from None
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
cheshire_cat_core | response = connection.handle_request(
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
cheshire_cat_core | return self._connection.handle_request(request)
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 143, in handle_request
cheshire_cat_core | raise exc
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 113, in handle_request
cheshire_cat_core | ) = self._receive_response_headers(**kwargs)
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 186, in _receive_response_headers
cheshire_cat_core | event = self._receive_event(timeout=timeout)
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 224, in _receive_event
cheshire_cat_core | data = self._network_stream.read(
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 124, in read
cheshire_cat_core | with map_exceptions(exc_map):
cheshire_cat_core | File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
cheshire_cat_core | self.gen.throw(typ, value, traceback)
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
cheshire_cat_core | raise to_exc(exc) from exc
cheshire_cat_core | httpcore.ReadTimeout: timed out
cheshire_cat_core |
cheshire_cat_core | The above exception was the direct cause of the following exception:
cheshire_cat_core |
cheshire_cat_core | Traceback (most recent call last):
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/qdrant_client/http/api_client.py", line 101, in send_inner
cheshire_cat_core | response = self._client.send(request)
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 914, in send
cheshire_cat_core | response = self._send_handling_auth(
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 942, in _send_handling_auth
cheshire_cat_core | response = self._send_handling_redirects(
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
cheshire_cat_core | response = self._send_single_request(request)
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1015, in _send_single_request
cheshire_cat_core | response = transport.handle_request(request)
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 232, in handle_request
cheshire_cat_core | with map_httpcore_exceptions():
cheshire_cat_core | File "/usr/local/lib/python3.10/contextlib.py", line 153, in __exit__
cheshire_cat_core | self.gen.throw(typ, value, traceback)
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions
cheshire_cat_core | raise mapped_exc(message) from exc
cheshire_cat_core | httpx.ReadTimeout: timed out
cheshire_cat_core |
cheshire_cat_core | During handling of the above exception, another exception occurred:
cheshire_cat_core |
cheshire_cat_core | Traceback (most recent call last):
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 732, in lifespan
cheshire_cat_core | async with self.lifespan_context(app) as maybe_state:
cheshire_cat_core | File "/usr/local/lib/python3.10/contextlib.py", line 199, in __aenter__
cheshire_cat_core | return await anext(self.gen)
cheshire_cat_core | File "/app/cat/main.py", line 33, in lifespan
cheshire_cat_core | app.state.ccat = CheshireCat()
cheshire_cat_core | File "/app/cat/utils.py", line 172, in getinstance
cheshire_cat_core | cls.instances[class_] = class_(*args, **kwargs)
cheshire_cat_core | File "/app/cat/looking_glass/cheshire_cat.py", line 74, in __init__
cheshire_cat_core | self.load_memory()
cheshire_cat_core | File "/app/cat/looking_glass/cheshire_cat.py", line 247, in load_memory
cheshire_cat_core | self.memory = LongTermMemory(vector_memory_config=vector_memory_config)
cheshire_cat_core | File "/app/cat/memory/long_term_memory.py", line 17, in __init__
cheshire_cat_core | self.vectors = VectorMemory(**vector_memory_config)
cheshire_cat_core | File "/app/cat/memory/vector_memory.py", line 37, in __init__
cheshire_cat_core | collection = VectorMemoryCollection(
cheshire_cat_core | File "/app/cat/memory/vector_memory_collection.py", line 49, in __init__
cheshire_cat_core | self.create_db_collection_if_not_exists()
cheshire_cat_core | File "/app/cat/memory/vector_memory_collection.py", line 100, in create_db_collection_if_not_exists
cheshire_cat_core | self.create_collection()
cheshire_cat_core | File "/app/cat/memory/vector_memory_collection.py", line 105, in create_collection
cheshire_cat_core | self.client.recreate_collection(
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/qdrant_client/qdrant_client.py", line 1763, in recreate_collection
cheshire_cat_core | return self._client.recreate_collection(
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/qdrant_client/qdrant_remote.py", line 2262, in recreate_collection
cheshire_cat_core | return self.create_collection(
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/qdrant_client/qdrant_remote.py", line 2233, in create_collection
cheshire_cat_core | result: Optional[bool] = self.http.collections_api.create_collection(
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/qdrant_client/http/api/collections_api.py", line 1180, in create_collection
cheshire_cat_core | return self._build_for_create_collection(
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/qdrant_client/http/api/collections_api.py", line 116, in _build_for_create_collection
cheshire_cat_core | return self.api_client.request(
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/qdrant_client/http/api_client.py", line 74, in request
cheshire_cat_core | return self.send(request, type_)
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/qdrant_client/http/api_client.py", line 91, in send
cheshire_cat_core | response = self.middleware(request, self.send_inner)
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/qdrant_client/http/api_client.py", line 200, in __call__
cheshire_cat_core | return call_next(request)
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/qdrant_client/http/api_client.py", line 103, in send_inner
cheshire_cat_core | raise ResponseHandlingException(e)
cheshire_cat_core | qdrant_client.http.exceptions.ResponseHandlingException: timed out
cheshire_cat_core |
cheshire_cat_core | ERROR: Application startup failed. Exiting.
cheshire_cat_vector_memory | 2024-06-09T21:44:25.746034Z INFO actix_web::middleware::logger: 172.19.0.4 "PUT /collections/episodic HTTP/1.1" 200 71 "-" "python-httpx/0.27.0" 5.886619
The other error I get is:
cheshire_cat_core | ERROR: Traceback (most recent call last):
cheshire_cat_core | File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 732, in lifespan
cheshire_cat_core | async with self.lifespan_context(app) as maybe_state:
cheshire_cat_core | File "/usr/local/lib/python3.10/contextlib.py", line 199, in __aenter__
cheshire_cat_core | return await anext(self.gen)
cheshire_cat_core | File "/app/cat/main.py", line 33, in lifespan
cheshire_cat_core | app.state.ccat = CheshireCat()
cheshire_cat_core | File "/app/cat/utils.py", line 172, in getinstance
cheshire_cat_core | cls.instances[class_] = class_(*args, **kwargs)
cheshire_cat_core | File "/app/cat/looking_glass/cheshire_cat.py", line 74, in __init__
cheshire_cat_core | self.load_memory()
cheshire_cat_core | File "/app/cat/looking_glass/cheshire_cat.py", line 247, in load_memory
cheshire_cat_core | self.memory = LongTermMemory(vector_memory_config=vector_memory_config)
cheshire_cat_core | File "/app/cat/memory/long_term_memory.py", line 17, in __init__
cheshire_cat_core | self.vectors = VectorMemory(**vector_memory_config)
cheshire_cat_core | File "/app/cat/memory/vector_memory.py", line 37, in __init__
cheshire_cat_core | collection = VectorMemoryCollection(
cheshire_cat_core | File "/app/cat/memory/vector_memory_collection.py", line 52, in __init__
cheshire_cat_core | self.check_embedding_size()
cheshire_cat_core | File "/app/cat/memory/vector_memory_collection.py", line 69, in check_embedding_size
cheshire_cat_core | == self.client.get_collection_aliases(self.collection_name)
cheshire_cat_core | IndexError: list index out of range
cheshire_cat_core |
cheshire_cat_core | ERROR: Application startup failed. Exiting.
I cannot figure out where I could have gone wrong.
Thanks in advance for your support
Hi,
This is the first time seeing this problem, but:
- Loggeru Errors can be ignored (are generated from libraries)
- Seems an http timeout from qdrant, try to start each container starting from qdrant
warnings.warn("Api key is used with unsecure connection.")from this warning, I need to ask if you set any .env file with a qdrant cloud key. Maybe the qdrant library is too old to handle the cloud- if the error persists, I think there is a problem with qdrant container and your system (from the other issue, the cat with
docker runworks flawlessly)
Hi and thanks for your time.
- Perfect, so they are not involved in this specific issue.
- I am new to Docker, I do not know how to do that, so I will study and try to follow your tip
- I did not set any .env file with a qdrant cloud key, I do not know qdrant, I have only read about what it is, but did not go into detail (I plan to do it in the future)
- It is very likely it is a problem with my system and qdrant container, infact I managed to get it working, by following the steps listed below.
Steps to reproduce:
-
I deleted inactive containers with
docker rm -v $(docker ps -a -q -f status=exited) -
I deleted all the images running
docker image prune -a -
I deleted the repository and cloned it again.
-
I restarted my system.
-
I cd into local-cat's directory, then I ran
docker compose up, and now the whole thing seems to be just working fine.
Thanks again for your support, I hope this can be helpful to other users.
What linux kernel are you using? maybe is better to open an issue on qdrant repo
Hi, sorry for the latency.
SYSTEM INFORMATION:
Linux
6.9.4-arch1-1 x86_64
if I encounter the same error again I will open an issue on qdrant repo, thanks for the tip.