Error if ddgs is run in non-main thread . Happens only in v4.4.1

Question

Error if ddgs is run in non-main thread . Happens only in v4.4.1

kspviswa opened this issue 5 months ago · comments

I reproduced this error in standalone fashion. Please use the gist available here : https://gist.github.com/kspviswa/6efe09b937af61315cec2f585eea60f5 .

As explained if ddgs is used in a different thread, this issue is seen

.venv/lib/python3.12/site-packages/duckduckgo_search/duckduckgo_search.py", line 16, in __init__
    if asyncio.get_event_loop().is_running():
       ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/events.py", line 698, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'Thread-1 (do_Search)'.

If I comment line 10-12 and direct call do_Search() as in line 13, then no issues.

deedy5 · Answer 1 · Sun Feb 11 2024 15:21:21 GMT+0800 (China Standard Time)

Thanks for discovering the problem.
Fixed in v4.4.2

Viz · Answer 2 · Mon Feb 12 2024 01:01:44 GMT+0800 (China Standard Time)

Based on my testing, search is functional, but unfortunately prior issues do remain.

Seeing this in console:

Task was destroyed but it is pending!
task: <Task cancelling name='Task-2' coro=<AsyncCurl._force_timeout() done, defined at .venv/lib/python3.12/site-packages/curl_cffi/aio.py:164> wait_for=<Future cancelled>>

Also seeing the old packaging issue with flet

Traceback (most recent call last):
  File "<string>", line 40, in <module>
  File "<frozen runpy>", line 229, in run_module
  File "<frozen runpy>", line 88, in _run_code
  File "/var/folders/rd/nsmb4_7n0lx599jrv5sl21400000gn/T/serious_python_temprJG3pv/main.py", line 3, in <module>
  File "/var/folders/rd/nsmb4_7n0lx599jrv5sl21400000gn/T/serious_python_temprJG3pv/prompt.py", line 3, in <module>
  File "/var/folders/rd/nsmb4_7n0lx599jrv5sl21400000gn/T/serious_python_temprJG3pv/search.py", line 5, in <module>
  File "/var/folders/rd/nsmb4_7n0lx599jrv5sl21400000gn/T/serious_python_temprJG3pv/__pypackages__/pydantic/v1/main.py", line 339, in __init__
  File "/var/folders/rd/nsmb4_7n0lx599jrv5sl21400000gn/T/serious_python_temprJG3pv/__pypackages__/pydantic/v1/main.py", line 1100, in validate_model
  File "/var/folders/rd/nsmb4_7n0lx599jrv5sl21400000gn/T/serious_python_temprJG3pv/__pypackages__/langchain_community/utilities/duckduckgo_search.py", line 33, in validate_environment
  File "/var/folders/rd/nsmb4_7n0lx599jrv5sl21400000gn/T/serious_python_temprJG3pv/__pypackages__/duckduckgo_search/__init__.py", line 10, in <module>
  File "/var/folders/rd/nsmb4_7n0lx599jrv5sl21400000gn/T/serious_python_temprJG3pv/__pypackages__/duckduckgo_search/duckduckgo_search.py", line 14, in <module>
  File "/var/folders/rd/nsmb4_7n0lx599jrv5sl21400000gn/T/serious_python_temprJG3pv/__pypackages__/docstring_inheritance/__init__.py", line 89, in __init__
  File "/var/folders/rd/nsmb4_7n0lx599jrv5sl21400000gn/T/serious_python_temprJG3pv/__pypackages__/docstring_inheritance/__init__.py", line 75, in __init__
  File "/var/folders/rd/nsmb4_7n0lx599jrv5sl21400000gn/T/serious_python_temprJG3pv/__pypackages__/docstring_inheritance/class_docstrings_inheritor.py", line 88, in inherit_docstrings
  File "/var/folders/rd/nsmb4_7n0lx599jrv5sl21400000gn/T/serious_python_temprJG3pv/__pypackages__/docstring_inheritance/class_docstrings_inheritor.py", line 136, in _inherit_attrs_docstrings
  File "/var/folders/rd/nsmb4_7n0lx599jrv5sl21400000gn/T/serious_python_temprJG3pv/__pypackages__/docstring_inheritance/docstring_inheritors/bases/inheritor.py", line 115, in inherit
  File "/var/folders/rd/nsmb4_7n0lx599jrv5sl21400000gn/T/serious_python_temprJG3pv/__pypackages__/docstring_inheritance/docstring_inheritors/bases/inheritor.py", line 127, in _inherit
  File "/var/folders/rd/nsmb4_7n0lx599jrv5sl21400000gn/T/serious_python_temprJG3pv/__pypackages__/docstring_inheritance/docstring_inheritors/bases/inheritor.py", line 268, in _inherit_sections
  File "/var/folders/rd/nsmb4_7n0lx599jrv5sl21400000gn/T/serious_python_temprJG3pv/__pypackages__/docstring_inheritance/docstring_inheritors/bases/inheritor.py", line 334, in _filter_args_section
  File "/var/folders/rd/nsmb4_7n0lx599jrv5sl21400000gn/T/serious_python_temprJG3pv/__pypackages__/docstring_inheritance/docstring_inheritors/bases/inheritor.py", line 209, in _warn
  File "./inspect.py", line 1244, in getsourcelines
  File "./inspect.py", line 1081, in findsource
OSError: could not get source code

deedy5 · Answer 3 · Mon Feb 12 2024 01:17:27 GMT+0800 (China Standard Time)

Are you sure you've updated the package?
Сheck the version:

ddgs version

Viz · Answer 4 · Mon Feb 12 2024 01:19:17 GMT+0800 (China Standard Time)

Yes..

(.venv) ☁  pyOllaMx [main] ⚡  pip show duckduckgo_search 
Name: duckduckgo_search
Version: 4.4.2
Summary: Search for words, documents, images, news, maps and text translation using the DuckDuckGo.com search engine.
Home-page: 
Author: deedy5
Author-email: 
License: MIT License
Location: /.venv/lib/python3.12/site-packages
Requires: click, curl-cffi, lxml, nest-asyncio
Required-by:

(.venv) ☁  pyOllaMx [main] ⚡  ddgs version
4.4.2

deedy5 · Answer 5 · Mon Feb 12 2024 01:22:12 GMT+0800 (China Standard Time)

Show me a code example of how I can reproduce the errors

Viz · Answer 6 · Mon Feb 12 2024 01:25:05 GMT+0800 (China Standard Time)

@deedy5 - For the async warning issue, you can use the same example I quoted in the gist : https://gist.github.com/kspviswa/6efe09b937af61315cec2f585eea60f5 .

For me, it does prints the results, but also with that warning. see below :

[{'title': 'What is DuckDuckGo and should you be using it? - Trusted Reviews', 'href': 'https://www.trustedreviews.com/explainer/what-is-duckduckgo-3969972', 'body': 'DuckDuckGo is a search engine that claims to protect your online privacy by not collecting or tracking your data. It also offers the same functionality as Google, but with more privacy-friendly features. Learn more about its history, mission and benefits.'}, {'title': 'About DuckDuckGo', 'href': 'https://duckduckgo.com/about', 'body': "We are the independent Internet privacy company for anyone who's tired of being tracked online and wants an easy solution. Our free, go-to browser comes with over a dozen powerful privacy protections built-in, including our search engine that replaces Google and doesn't track your search history."}, {'title': "DuckDuckGo: What to Know About Google Search's Privacy-Focused Rival", 'href': 'https://www.cnet.com/tech/services-and-software/duckduckgo-what-to-know-about-google-searchs-privacy-focused-rival/', 'body': "DuckDuckGo is a search engine that's available as both a mobile browser app and a desktop extension, each aimed at allowing you to browse the internet without companies gobbling up your personal..."}, {'title': 'How to use DuckDuckGo: Everything you need to know', 'href': 'https://www.tomsguide.com/how-to/how-to-use-duckduckgo', 'body': "What DuckDuckGo is, and isn't, good for Comments (2) (Image credit: Ascannio/Shutterstock) If privacy is what you're after and you want to get away from using Google, DuckDuckGo is a great..."}, {'title': 'DuckDuckGo - Wikipedia', 'href': 'https://en.wikipedia.org/wiki/DuckDuckGo', 'body': 'DuckDuckGo is an American software company that offers a number of software products oriented towards helping people protect their privacy online, [6] most notably, a private search engine, a tracker-blocking browser extension, email protection, and app tracking protection.'}]
Task was destroyed but it is pending!
task: <Task cancelling name='Task-2' coro=<AsyncCurl._force_timeout() done, defined at .venv/lib/python3.12/site-packages/curl_cffi/aio.py:164> wait_for=<Future cancelled>>

Viz · Answer 7 · Mon Feb 12 2024 01:26:56 GMT+0800 (China Standard Time)

For the flet app startup issue, I can upload the packaged app for you to review if you want. Let me know!

deedy5 · Answer 8 · Mon Feb 12 2024 01:42:23 GMT+0800 (China Standard Time)

https://gist.github.com/kspviswa/6efe09b937af61315cec2f585eea60f5
I see the messages.
flet app
Check duckduckgo_search version in flet app.
The docstring_inheritance package was removed in 4.4.1, but persist in traceback.

deedy5 · Answer 9 · Mon Feb 12 2024 03:24:58 GMT+0800 (China Standard Time)

I created PR to resolve problems with messages like Task was destroyed but it is pending!
yifeikong/curl_cffi#245

Viz · Answer 10 · Mon Feb 12 2024 06:05:00 GMT+0800 (China Standard Time)

https://gist.github.com/kspviswa/6efe09b937af61315cec2f585eea60f5
I see the messages.

flet app
Check duckduckgo_search version in flet app.
The docstring_inheritance package was removed in 4.4.1, but persist in traceback.

It works. flet pack command respects what is there in requirements.txt. When I tested, I didn't update the reqs. Now it is working perfectly fine 👍 . Thank you very much for fixing this.

Michael Doyle · Answer 11 · Fri Feb 16 2024 04:54:25 GMT+0800 (China Standard Time)

I still run into this issue when just installing ddgs and using gradio. Seems to be a collision with uvicorn, asyncio, etc. but this isn't really my expertise:

Gradio version: 4.16.0
DDGS version: 4.4.3

Code:

import duckduckgo_search
import gradio as gr


def predict(message: str, history: list[str]) -> str:
    return "hello"


gr.ChatInterface(predict).launch()

Output:

Exception in thread Thread-4 (run):
Traceback (most recent call last):
  File "/Users/mdoyle/miniconda3/envs/ma/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/Users/mdoyle/miniconda3/envs/ma/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/mdoyle/miniconda3/envs/ma/lib/python3.12/site-packages/uvicorn/server.py", line 62, in run
    return asyncio.run(self.serve(sockets=sockets))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdoyle/miniconda3/envs/ma/lib/python3.12/site-packages/nest_asyncio.py", line 26, in run
    loop = asyncio.get_event_loop()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdoyle/miniconda3/envs/ma/lib/python3.12/site-packages/nest_asyncio.py", line 40, in _get_event_loop
    loop = events.get_event_loop_policy().get_event_loop()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mdoyle/miniconda3/envs/ma/lib/python3.12/asyncio/events.py", line 698, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'Thread-4 (run)'.

I don't get this when I don't import duckduckgo_search.

deedy5 · Answer 12 · Fri Feb 16 2024 15:51:46 GMT+0800 (China Standard Time)

@doyled-it
nest-asyncio is just a temporary measure to get around the curl-cffi bug
try to use v4.4.1

pip install duckduckgo_search==4.4.1

Michael Doyle · Answer 13 · Tue Feb 20 2024 03:19:54 GMT+0800 (China Standard Time)

@doyled-it nest-asyncio is just a temporary measure to get around the curl-cffi bug try to use v4.4.1
pip install duckduckgo_search==4.4.1

That worked. Also, the issue with nest_asyncio looks to be fixed in more recent versions according to the maintainer.