Python completions example with streaming Error.
TheMeIonGod opened this issue · comments
TheMelonGod commented
Describe the bug
When trying to use text stream via the API the Host receives it and responds but the example code fails to display it showing the error below. The Non-streaming API works perfectly. Tried with GGUF and EXL2 models (Just in case)
Is there an existing issue for this?
- I have searched the existing issues
Reproduction
Make sure you are running TGW with API enabled and load a model.
Copy Python completions example with streaming code.
Make sure sseclient is installed
Run said code Using python -u scriptname.py
Error.
Screenshot
Logs
Traceback (most recent call last):
File "E:\AI\LLM-API\Text-Streaming\venv\lib\site-packages\urllib3\util\url.py", line 425, in parse_url
host, port = _HOST_PORT_RE.match(host_port).groups() # type: ignore[union-attr]
AttributeError: 'NoneType' object has no attribute 'groups'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "E:\AI\LLM-API\Text-Streaming\venv\lib\site-packages\requests\models.py", line 433, in prepare_url
scheme, auth, host, port, path, query, fragment = parse_url(url)
File "E:\AI\LLM-API\Text-Streaming\venv\lib\site-packages\urllib3\util\url.py", line 451, in parse_url
raise LocationParseError(source_url) from e
urllib3.exceptions.LocationParseError: Failed to parse: <Response [200]>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "E:\AI\LLM-API\Text-Streaming\API-Stream.py", line 21, in <module>
client = sseclient.SSEClient(stream_response)
File "E:\AI\LLM-API\Text-Streaming\venv\lib\site-packages\sseclient.py", line 48, in __init__
self._connect()
File "E:\AI\LLM-API\Text-Streaming\venv\lib\site-packages\sseclient.py", line 56, in _connect
self.resp = requester.get(self.url, stream=True, **self.requests_kwargs)
File "E:\AI\LLM-API\Text-Streaming\venv\lib\site-packages\requests\api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "E:\AI\LLM-API\Text-Streaming\venv\lib\site-packages\requests\api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "E:\AI\LLM-API\Text-Streaming\venv\lib\site-packages\requests\sessions.py", line 575, in request
prep = self.prepare_request(req)
File "E:\AI\LLM-API\Text-Streaming\venv\lib\site-packages\requests\sessions.py", line 484, in prepare_request
p.prepare(
File "E:\AI\LLM-API\Text-Streaming\venv\lib\site-packages\requests\models.py", line 367, in prepare
self.prepare_url(url, params)
File "E:\AI\LLM-API\Text-Streaming\venv\lib\site-packages\requests\models.py", line 435, in prepare_url
raise InvalidURL(*e.args)
requests.exceptions.InvalidURL: Failed to parse: <Response [200]>
System Info
Windows 10, Nvidia 4080.