oobabooga / text-generation-webui

A Gradio web UI for Large Language Models.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Python completions example with streaming Error.

TheMeIonGod opened this issue · comments

Describe the bug

When trying to use text stream via the API the Host receives it and responds but the example code fails to display it showing the error below. The Non-streaming API works perfectly. Tried with GGUF and EXL2 models (Just in case)

Is there an existing issue for this?

  • I have searched the existing issues

Reproduction

Make sure you are running TGW with API enabled and load a model.
Copy Python completions example with streaming code.
Make sure sseclient is installed
Run said code Using python -u scriptname.py
Error.

Screenshot

Screenshot 2024-10-21 013325
Screenshot 2024-10-21 013358

Logs

Traceback (most recent call last):
  File "E:\AI\LLM-API\Text-Streaming\venv\lib\site-packages\urllib3\util\url.py", line 425, in parse_url
    host, port = _HOST_PORT_RE.match(host_port).groups()  # type: ignore[union-attr]
AttributeError: 'NoneType' object has no attribute 'groups'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "E:\AI\LLM-API\Text-Streaming\venv\lib\site-packages\requests\models.py", line 433, in prepare_url
    scheme, auth, host, port, path, query, fragment = parse_url(url)
  File "E:\AI\LLM-API\Text-Streaming\venv\lib\site-packages\urllib3\util\url.py", line 451, in parse_url
    raise LocationParseError(source_url) from e
urllib3.exceptions.LocationParseError: Failed to parse: <Response [200]>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:\AI\LLM-API\Text-Streaming\API-Stream.py", line 21, in <module>
    client = sseclient.SSEClient(stream_response)
  File "E:\AI\LLM-API\Text-Streaming\venv\lib\site-packages\sseclient.py", line 48, in __init__
    self._connect()
  File "E:\AI\LLM-API\Text-Streaming\venv\lib\site-packages\sseclient.py", line 56, in _connect
    self.resp = requester.get(self.url, stream=True, **self.requests_kwargs)
  File "E:\AI\LLM-API\Text-Streaming\venv\lib\site-packages\requests\api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "E:\AI\LLM-API\Text-Streaming\venv\lib\site-packages\requests\api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "E:\AI\LLM-API\Text-Streaming\venv\lib\site-packages\requests\sessions.py", line 575, in request
    prep = self.prepare_request(req)
  File "E:\AI\LLM-API\Text-Streaming\venv\lib\site-packages\requests\sessions.py", line 484, in prepare_request
    p.prepare(
  File "E:\AI\LLM-API\Text-Streaming\venv\lib\site-packages\requests\models.py", line 367, in prepare
    self.prepare_url(url, params)
  File "E:\AI\LLM-API\Text-Streaming\venv\lib\site-packages\requests\models.py", line 435, in prepare_url
    raise InvalidURL(*e.args)
requests.exceptions.InvalidURL: Failed to parse: <Response [200]>

System Info

Windows 10, Nvidia 4080.