use route query engine : http pool time out error
waveBoom opened this issue · comments
Question Validation
- I have searched both the documentation and discord for an answer.
Question
when i use the route query engine
code is
`summary_tool = QueryEngineTool.from_defaults(
query_engine=summary_query_engine,
description=(
"Useful for summarization questions related to the eassy or the file on "
" What I Worked On."
),
)
vector_tool = QueryEngineTool.from_defaults(
query_engine=vector_query_engine,
description=(
"Useful for retrieving specific context using vectors or Embedding from the essay or the file on What"
" I Worked On."
),
)
keyword_tool = QueryEngineTool.from_defaults(
query_engine=vector_query_engine,
description=(
"Useful for retrieving specific context using keywords from the essay or the file on What"
" I Worked On."
),
)
query_engine = RouterQueryEngine(
selector=LLMMultiSelector.from_defaults(),
query_engine_tools=[
summary_tool,
vector_tool,
keyword_tool
],
)`
when i upload epub file to qa
alaways have error : time out
all log below :
openai.APITimeoutError: Request timed out. ERROR:server.py:handle_mentions:An error occurred: Traceback (most recent call last): File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/llama_index/core/async_utils.py", line 29, in asyncio_run loop = asyncio.get_event_loop() File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/events.py", line 642, in get_event_loop raise RuntimeError('There is no current event loop in thread %r.' RuntimeError: There is no current event loop in thread 'ThreadPoolExecutor-0_0'. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/anyio/_core/_tasks.py", line 115, in fail_after yield cancel_scope File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/httpcore/_synchronization.py", line 103, in wait await self._anyio_event.wait() File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1651, in wait await self._event.wait() File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/locks.py", line 226, in wait await fut asyncio.exceptions.CancelledError: Cancelled by cancel scope 120054fd0 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions yield File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/httpcore/_synchronization.py", line 103, in wait await self._anyio_event.wait() File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 137, in __exit__ self.gen.throw(typ, value, traceback) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/anyio/_core/_tasks.py", line 118, in fail_after raise TimeoutError TimeoutError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions yield File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request resp = await self._pool.handle_async_request(req) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 234, in handle_async_request raise exc File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 227, in handle_async_request connection = await status.wait_for_connection(timeout=timeout) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 34, in wait_for_connection await self._connection_acquired.wait(timeout=timeout) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/httpcore/_synchronization.py", line 103, in wait await self._anyio_event.wait() File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 137, in __exit__ self.gen.throw(typ, value, traceback) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions raise to_exc(exc) httpcore.PoolTimeout The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/openai/_base_client.py", line 1514, in _request response = await self._client.send( File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/httpx/_client.py", line 1620, in send response = await self._send_handling_auth( File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/httpx/_client.py", line 1648, in _send_handling_auth response = await self._send_handling_redirects( File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/httpx/_client.py", line 1685, in _send_handling_redirects response = await self._send_single_request(request) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/httpx/_client.py", line 1722, in _send_single_request response = await transport.handle_async_request(request) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request resp = await self._pool.handle_async_request(req) File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 137, in __exit__ self.gen.throw(typ, value, traceback) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions raise mapped_exc(message) from exc httpx.PoolTimeout The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Users/bobo/practise/myGPTReader/app/server.py", line 252, in bot_process gpt_response, total_llm_model_tokens, total_embedding_model_tokens = future.result(timeout=300) File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 445, in result return self.__get_result() File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result raise self._exception File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 52, in run result = self.fn(*self.args, **self.kwargs) File "/Users/bobo/practise/myGPTReader/app/gpt.py", line 305, in get_answer_from_llama_file_route_engine answer = query_engine.query(dialog_messages) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/llama_index/core/instrumentation/dispatcher.py", line 274, in wrapper result = func(*args, **kwargs) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/llama_index/core/base/base_query_engine.py", line 53, in query query_result = self._query(str_or_query_bundle) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/llama_index/core/query_engine/router_query_engine.py", line 183, in _query responses.append(selected_query_engine.query(query_bundle)) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/llama_index/core/instrumentation/dispatcher.py", line 274, in wrapper result = func(*args, **kwargs) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/llama_index/core/base/base_query_engine.py", line 53, in query query_result = self._query(str_or_query_bundle) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/llama_index/core/instrumentation/dispatcher.py", line 274, in wrapper result = func(*args, **kwargs) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/llama_index/core/query_engine/retriever_query_engine.py", line 190, in _query response = self._response_synthesizer.synthesize( File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/llama_index/core/instrumentation/dispatcher.py", line 274, in wrapper result = func(*args, **kwargs) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/llama_index/core/response_synthesizers/base.py", line 242, in synthesize response_str = self.get_response( File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/llama_index/core/response_synthesizers/tree_summarize.py", line 236, in get_response return self.get_response( File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/llama_index/core/response_synthesizers/tree_summarize.py", line 207, in get_response summary_responses = run_async_tasks(tasks) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/llama_index/core/async_utils.py", line 66, in run_async_tasks outputs: List[Any] = asyncio_run(_gather()) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/llama_index/core/async_utils.py", line 32, in asyncio_run return asyncio.run(coro) File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete return future.result() File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/llama_index/core/async_utils.py", line 64, in _gather return await asyncio.gather(*tasks_to_execute) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/llama_index/core/instrumentation/dispatcher.py", line 307, in async_wrapper result = await func(*args, **kwargs) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/llama_index/core/llms/llm.py", line 526, in apredict chat_response = await self.achat(messages) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/llama_index/core/llms/callbacks.py", line 73, in wrapped_async_llm_chat f_return_val = await f(_self, messages, **kwargs) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/llama_index/llms/openai/base.py", line 575, in achat return await achat_fn(messages, **kwargs) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/tenacity/_asyncio.py", line 88, in async_wrapped return await fn(*args, **kwargs) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/tenacity/_asyncio.py", line 47, in __call__ do = self.iter(retry_state=retry_state) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/tenacity/__init__.py", line 325, in iter raise retry_exc.reraise() File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/tenacity/__init__.py", line 158, in reraise raise self.last_attempt.result() File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 438, in result return self.__get_result() File "/usr/local/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result raise self._exception File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/tenacity/_asyncio.py", line 50, in __call__ result = await fn(*args, **kwargs) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/llama_index/llms/openai/base.py", line 620, in _achat response = await aclient.chat.completions.create( File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/openai/resources/chat/completions.py", line 1159, in create return await self._post( File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/openai/_base_client.py", line 1782, in post return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls) File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/openai/_base_client.py", line 1485, in request return await self._request( File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/openai/_base_client.py", line 1538, in _request return await self._retry_request( File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/openai/_base_client.py", line 1607, in _retry_request return await self._request( File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/openai/_base_client.py", line 1538, in _request return await self._retry_request( File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/openai/_base_client.py", line 1607, in _retry_request return await self._request( File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/openai/_base_client.py", line 1523, in _request return await self._retry_request( File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/openai/_base_client.py", line 1607, in _retry_request return await self._request( File "/Users/bobo/practise/myGPTReader/venv/lib/python3.9/site-packages/openai/_base_client.py", line 1533, in _request raise APITimeoutError(request=request) from err openai.APITimeoutError: Request timed out.
Hey @waveBoom! Good to see you back. 😄 Hope everything else is going smoothly for you. I'm diving into your issue now and will get back to you with a full diagnosis shortly. Hang tight!
before my summary query engine is :
summary_query_engine = summary_index.as_query_engine( response_mode="tree_summarize" use_async=True, )
when i change the summary_query_engine to :
httpclient = DefaultHttpxClient() llm_tree_summary = llmOpenAi(temperature=0, model=model_name, http_client=httpclient) service_context_tree_summary = ServiceContext.from_defaults(llm=llm_tree_summary) summary_index = SummaryIndex.from_documents(documents,service_context=service_context_tree_summary) summary_query_engine = summary_index.as_query_engine( response_mode="tree_summarize" ),
the error does not exist
i dont konw the reason that cause this error
The timeout error you encountered when uploading epub files and using the route query engine was likely due to the asynchronous handling of HTTP requests within the LlamaIndex framework. This framework uses httpx.AsyncClient for efficient, non-blocking I/O operations, which can be particularly sensitive to operations that take a significant amount of time, such as processing large files.
Switching to a custom HTTP client (DefaultHttpxClient) and adjusting the configuration of your summary query engine helped resolve the timeout issue. This change provided you with enhanced control over HTTP client parameters, including timeouts, connection pooling, and retries. Such adjustments are crucial for optimizing the handling of network-related issues and improving the overall performance of your application, especially when dealing with large files or complex operations that require more time to process.
In essence, the resolution came from better managing the asynchronous request handling by using a custom HTTP client, which allowed for more tailored configuration to suit the specific needs of your application and the network conditions it operates under.
Details
Sources
- llama-index-integrations/embeddings/llama-index-embeddings-azure-openai/tests/test_azure_openai.py
- llama-index-integrations/llms/llama-index-llms-llamafile/llama_index/llms/llamafile/base.py
- llama-index-legacy/llama_index/legacy/ingestion/pipeline.py
- llama-index-networks/llama_index/networks/network/query_engine.py
- llama-index-integrations/embeddings/llama-index-embeddings-llamafile/llama_index/embeddings/llamafile/base.py
- llama-index-integrations/llms/llama-index-llms-azure-openai/tests/test_azure_openai.py
- llama-index-core/llama_index/core/readers/file/base.py
- llama-index-core/llama_index/core/response_synthesizers/type.py
- llama-index-core/llama_index/core/response_synthesizers/tree_summarize.py
- llama-index-integrations/readers/llama-index-readers-file/llama_index/readers/file/epub/base.py
- llama-index-legacy/llama_index/legacy/readers/file/epub_reader.py
- llama-index-legacy/llama_index/legacy/response_synthesizers/tree_summarize.py
- llama-index-core/tests/indices/response/test_tree_summarize.py
About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
To continue the conversation, mention @dosu.
before code :
from llama_index.llms.openai import OpenAI as llmOpenAi
llm = llmOpenAi(temperature=0, model=model_name)
service_context = ServiceContext.from_defaults(llm=llm)
index = VectorStoreIndex.from_documents(documents, service_context=service_context)
summary_index = SummaryIndex.from_documents(documents,service_context=service_context)
keyword_index = SimpleKeywordTableIndex.from_documents(documents,service_context=service_context)vector_query_engine = vector_index.as_query_engine(text_qa_template=prompt, similarity_top_k=5)
summary_query_engine = summary_index.as_query_engine(
response_mode="tree_summarize"
use_async=True,
)
summary_tool = QueryEngineTool.from_defaults(
query_engine=summary_query_engine,
description=(
"Useful for summarization questions related to the eassy or the file on "
" What I Worked On."
),
)
vector_tool = QueryEngineTool.from_defaults(
query_engine=vector_query_engine,
description=(
"Useful for retrieving specific context using vectors or Embedding from the essay or the file on What"
" I Worked On."
),
)
keyword_tool = QueryEngineTool.from_defaults(
query_engine=vector_query_engine,
description=(
"Useful for retrieving specific context using keywords from the essay or the file on What"
" I Worked On."
),
)
query_engine = RouterQueryEngine(
selector=LLMMultiSelector.from_defaults(),
query_engine_tools=[
summary_tool,
vector_tool,
keyword_tool
],
)
answer = query_engine.query(query_messages)
by default , i see the code default limits ls :
DEFAULT_CONNECTION_LIMITS = httpx.Limits(max_connections=1000, max_keepalive_connections=100)
then i change code is
httpclient = DefaultHttpxClient()
llm_tree_summary = llmOpenAi(temperature=0, model=model_name, http_client=httpclient)
service_context_tree_summary = ServiceContext.from_defaults(llm=llm_tree_summary)
summary_index = SummaryIndex.from_documents(documents,service_context=service_context_tree_summary) summary_query_engine = summary_index.as_query_engine( response_mode="tree_summarize" )
The modified content is ：
summary_query_engine changed from 'use_async=True ' to 'use_async=False'
and http_client is DefaultHttpxClient
but i see the DefaultHttpxClient 's Limits is also :
DEFAULT_CONNECTION_LIMITS = httpx.Limits(max_connections=1000, max_keepalive_connections=100)
（Same as when http_client is not set）
！！！So I think the reason why no errors occurred later was not because http_client was set, but because asynchronous was changed to synchronous？
Please help me solve this confusion:
What happened after setting to sync that caused the error to disappear
Switching from asynchronous to synchronous processing likely resolved the timeout errors you were experiencing with the LlamaIndex framework when uploading and querying epub files. This change leads to a more predictable and stable environment for handling file operations, which is particularly beneficial for tasks that require consistent and sequential processing, like reading and indexing large files.
In asynchronous processing, while it allows for non-blocking I/O operations, it introduces complexity in error handling and resource management. This complexity can lead to timeouts, especially when dealing with large files or under heavy load, due to potential mismanagement of the event loop or resources.
On the other hand, synchronous processing waits for each operation to complete before moving on to the next, offering more predictable control over resource management and operation sequencing. This method is simpler to manage and debug, ensuring that operations complete in a defined order and reducing the likelihood of timeout errors during intensive file operations.
Therefore, the absence of errors after switching to synchronous processing is not merely due to setting a custom HTTP client but primarily because of the shift to synchronous execution. This approach mitigates the risk of timeouts by ensuring more stable and predictable file and network operations within the LlamaIndex framework.
Details
Sources
- [Feature Request]: improve support for async in llama_index
- llama-index-core/llama_index/core/readers/file/base.py
- llama-index-integrations/readers/llama-index-readers-web/llama_index/readers/web/async_web/base.py
- llama-index-integrations/llms/llama-index-llms-azure-openai/tests/test_azure_openai.py
- llama-index-core/llama_index/core/indices/common_tree/base.py
- llama-index-legacy/llama_index/legacy/llms/base.py
- llama-index-integrations/embeddings/llama-index-embeddings-llamafile/llama_index/embeddings/llamafile/base.py
- llama-index-core/llama_index/core/llms/callbacks.py
- llama-index-core/llama_index/core/evaluation/batch_runner.py
About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
To continue the conversation, mention @dosu.