Error while loading knowledge base from url
suresh-saini opened this issue · comments
Describe the bug
When I add knowledge base from url it adds few pages after that it keeps rolling but no further progress.
It shows few error on backend logs
To Reproduce
Steps to reproduce the behavior:
- Clone the repository
- Get the servers up using
make install
- Open the portal http://localhost:8888
- Create a copilot
- Go to 'Knowledge Base'
- Click on 'Add Data Source->URL(s)'
- Enter url as
https://gamechange.com
- Click on 'Add'
Expected behavior
All the pages/subpages (around 40 in total) should have been added.
Desktop (please complete the following information):
- OS: Ubuntu
- Browser: Chrome
- Version: 120.0.6099.199 (Official Build) (64-bit)
Smartphone (please complete the following information):
NA
Additional context
below are the backend log entries where I see few errors
opencopilot-llm-server-1 | 172.30.0.10 - - [16/Jan/2024 07:10:50] "POST /backend/copilot/ HTTP/1.0" 200 -
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:10:50 +0000] "POST /backend/copilot/ HTTP/1.1" 200 628 "http://localhost:8888/create/copilot" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-llm-server-1 | 172.30.0.10 - - [16/Jan/2024 07:10:50] "GET /backend/actions/bot/71175cae-da82-489f-900f-42607113fa6c HTTP/1.0" 200 -
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:10:50 +0000] "GET /backend/actions/bot/71175cae-da82-489f-900f-42607113fa6c HTTP/1.1" 200 3 "http://localhost:8888/create/copilot" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:10:57 +0000] "GET /copilot/71175cae-da82-489f-900f-42607113fa6c?_rsc=14gk8 HTTP/1.1" 200 188 "http://localhost:8888/create/copilot" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:11:00 +0000] "GET /copilot/71175cae-da82-489f-900f-42607113fa6c?_rsc=5xwpy HTTP/1.1" 200 7094 "http://localhost:8888/create/copilot" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-llm-server-1 | 172.30.0.10 - - [16/Jan/2024 07:11:00] "GET /backend/copilot/71175cae-da82-489f-900f-42607113fa6c HTTP/1.0" 200 -
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:11:00 +0000] "GET /backend/copilot/71175cae-da82-489f-900f-42607113fa6c HTTP/1.1" 200 640 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:11:00 +0000] "GET /copilot/71175cae-da82-489f-900f-42607113fa6c/workflow?_rsc=1arzg HTTP/1.1" 200 185 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:11:00 +0000] "GET /copilot/71175cae-da82-489f-900f-42607113fa6c/settings?_rsc=1arzg HTTP/1.1" 200 198 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:11:00 +0000] "GET /copilot/71175cae-da82-489f-900f-42607113fa6c/conversations?_rsc=1arzg HTTP/1.1" 200 201 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:11:00 +0000] "GET /copilot/71175cae-da82-489f-900f-42607113fa6c/knowledge?_rsc=1arzg HTTP/1.1" 200 199 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-llm-server-1 | 172.30.0.10 - - [16/Jan/2024 07:11:01] "GET /socket.io/?EIO=4&transport=polling&t=OqH6_Ng HTTP/1.1" 200 -
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:11:01 +0000] "GET /socket.io/?EIO=4&transport=polling&t=OqH6_Ng HTTP/1.1" 200 108 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-llm-server-1 | 172.30.0.10 - - [16/Jan/2024 07:11:01] "POST /socket.io/?EIO=4&transport=polling&t=OqH6_PI&sid=UDfunb5pCvKU-BSjAAAA HTTP/1.1" 200 -
opencopilot-llm-server-1 | 172.30.0.10 - - [16/Jan/2024 07:11:01] "GET /socket.io/?EIO=4&transport=polling&t=OqH6_PM&sid=UDfunb5pCvKU-BSjAAAA HTTP/1.1" 200 -
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:11:01 +0000] "POST /socket.io/?EIO=4&transport=polling&t=OqH6_PI&sid=UDfunb5pCvKU-BSjAAAA HTTP/1.1" 200 12 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:11:01 +0000] "GET /socket.io/?EIO=4&transport=polling&t=OqH6_PM&sid=UDfunb5pCvKU-BSjAAAA HTTP/1.1" 200 43 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-llm-server-1 | 172.30.0.10 - - [16/Jan/2024 07:11:01] "GET /backend/chat/init HTTP/1.0" 200 -
opencopilot-llm-server-1 | 172.30.0.10 - - [16/Jan/2024 07:11:01] "GET /socket.io/?EIO=4&transport=polling&t=OqH6_Qu&sid=UDfunb5pCvKU-BSjAAAA HTTP/1.1" 200 -
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:11:01 +0000] "GET /backend/chat/init HTTP/1.1" 200 91 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:11:01 +0000] "GET /socket.io/?EIO=4&transport=polling&t=OqH6_Qu&sid=UDfunb5pCvKU-BSjAAAA HTTP/1.1" 200 11 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-llm-server-1 | 172.30.0.10 - - [16/Jan/2024 07:11:01] "GET /backend/chat/init HTTP/1.0" 200 -
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:11:01 +0000] "GET /backend/chat/init HTTP/1.1" 200 91 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:11:03 +0000] "GET /copilot/71175cae-da82-489f-900f-42607113fa6c/workflow?_rsc=1vw3t HTTP/1.1" 200 1979 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:11:03 +0000] "GET /random_icons_2.svg HTTP/1.1" 304 0 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c/workflow" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-llm-server-1 | 172.30.0.10 - - [16/Jan/2024 07:11:03] "GET /backend/flows/bot/71175cae-da82-489f-900f-42607113fa6c HTTP/1.0" 200 -
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:11:03 +0000] "GET /backend/flows/bot/71175cae-da82-489f-900f-42607113fa6c HTTP/1.1" 200 3 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c/workflow" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:11:05 +0000] "GET /copilot/71175cae-da82-489f-900f-42607113fa6c/knowledge?_rsc=136t3 HTTP/1.1" 200 3430 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c/workflow" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-llm-server-1 | 172.30.0.10 - - [16/Jan/2024 07:11:05] "GET /backend/data_sources/b/71175cae-da82-489f-900f-42607113fa6c HTTP/1.0" 200 -
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:11:05 +0000] "GET /backend/data_sources/b/71175cae-da82-489f-900f-42607113fa6c HTTP/1.1" 200 36 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c/knowledge" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-llm-server-1 | 172.30.0.10 - - [16/Jan/2024 07:11:15] "GET /backend/data_sources/b/71175cae-da82-489f-900f-42607113fa6c HTTP/1.0" 200 -
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:11:15 +0000] "GET /backend/data_sources/b/71175cae-da82-489f-900f-42607113fa6c HTTP/1.1" 200 36 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c/knowledge" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-workers-1 | [2024-01-16 07:11:20,941: INFO/MainProcess] Task workers.tasks.web_crawl.web_crawl[efa251ad-1594-4400-b353-cd1b706ead54] received
opencopilot-workers-1 | [2024-01-16 07:11:20,950: WARNING/ForkPoolWorker-7] Received: https://gamechange.com, 71175cae-da82-489f-900f-42607113fa6c
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:11:20 +0000] "POST /backend/uploads/file/ingest HTTP/1.1" 200 41 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c/knowledge" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-llm-server-1 | 172.30.0.10 - - [16/Jan/2024 07:11:20] "POST /backend/uploads/file/ingest HTTP/1.0" 200 -
opencopilot-selenium-1 | 07:11:21.822 INFO [LocalDistributor.newSession] - Session request received by the Distributor:
opencopilot-selenium-1 | [Capabilities {acceptInsecureCerts: true, browserName: firefox, moz:debuggerAddress: true, pageLoadStrategy: normal}]
opencopilot-selenium-1 | 07:11:25.434 WARN [SeleniumManager.lambda$runCommand$1] - Exception managing firefox: Unable to discover proper geckodriver version in offline mode
opencopilot-llm-server-1 | 172.30.0.10 - - [16/Jan/2024 07:11:25] "GET /backend/data_sources/b/71175cae-da82-489f-900f-42607113fa6c HTTP/1.0" 200 -
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:11:25 +0000] "GET /backend/data_sources/b/71175cae-da82-489f-900f-42607113fa6c HTTP/1.1" 200 359 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c/knowledge" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-llm-server-1 | 172.30.0.10 - - [16/Jan/2024 07:11:35] "GET /backend/data_sources/b/71175cae-da82-489f-900f-42607113fa6c HTTP/1.0" 200 -
opencopilot-nginx-1 | 172.30.0.1 - - [16/Jan/2024:07:11:35 +0000] "GET /backend/data_sources/b/71175cae-da82-489f-900f-42607113fa6c HTTP/1.1" 200 359 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c/knowledge" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-selenium-1 | 07:11:51.712 INFO [LocalNode.newSession] - Session created by the Node. Id: 8e6a396c-c5d3-458e-a66a-4121b110a53f, Caps: Capabilities {acceptInsecureCerts: true, browserName: firefox, browserVersion: 119.0, moz:accessibilityChecks: false, moz:buildID: 20231019122658, moz:debuggerAddress: 127.0.0.1:21699, moz:geckodriverVersion: 0.33.0, moz:headless: false, moz:platformVersion: 6.5.0-14-generic, moz:processID: 249, moz:profile: /tmp/rust_mozprofileJD2MMj, moz:shutdownTimeout: 60000, moz:webdriverClick: true, moz:windowless: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), se:bidiEnabled: false, se:cdp: ws://172.30.0.5:4444/sessio..., se:cdpVersion: 85.0, se:noVncPort: 7900, se:vnc: ws://172.30.0.5:4444/sessio..., se:vncEnabled: true, se:vncLocalAddress: ws://172.30.0.5:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify}
opencopilot-selenium-1 | 07:11:51.768 INFO [LocalDistributor.newSession] - Session created by the Distributor. Id: 8e6a396c-c5d3-458e-a66a-4121b110a53f
opencopilot-selenium-1 | Caps: Capabilities {acceptInsecureCerts: true, browserName: firefox, browserVersion: 119.0, moz:accessibilityChecks: false, moz:buildID: 20231019122658, moz:debuggerAddress: 127.0.0.1:21699, moz:geckodriverVersion: 0.33.0, moz:headless: false, moz:platformVersion: 6.5.0-14-generic, moz:processID: 249, moz:profile: /tmp/rust_mozprofileJD2MMj, moz:shutdownTimeout: 60000, moz:webdriverClick: true, moz:windowless: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), se:bidiEnabled: false, se:cdp: ws://172.30.0.5:4444/sessio..., se:cdpVersion: 85.0, se:noVncPort: 7900, se:vnc: ws://172.30.0.5:4444/sessio..., se:vncEnabled: true, se:vncLocalAddress: ws://172.30.0.5:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify}
opencopilot-qdrant-1 | 2024-01-16T07:12:13.063651Z INFO actix_web::middleware::logger: 172.30.0.7 "PUT /collections/knowledgebase/points?wait=true HTTP/1.1" 200 92 "-" "python-httpx/0.25.0" 0.343635
opencopilot-workers-1 | [2024-01-16 07:12:13,068: INFO/ForkPoolWorker-7] HTTP Request: PUT http://qdrant:6333/collections/knowledgebase/points?wait=true "HTTP/1.1 200 OK"
opencopilot-qdrant-1 | 2024-01-16T07:12:15.284872Z INFO actix_web::middleware::logger: 172.30.0.7 "PUT /collections/knowledgebase/points?wait=true HTTP/1.1" 200 91 "-" "python-httpx/0.25.0" 0.049263
opencopilot-workers-1 | [2024-01-16 07:12:15,285: INFO/ForkPoolWorker-7] HTTP Request: PUT http://qdrant:6333/collections/knowledgebase/points?wait=true "HTTP/1.1 200 OK"
opencopilot-qdrant-1 | 2024-01-16T07:12:17.125837Z INFO actix_web::middleware::logger: 172.30.0.7 "PUT /collections/knowledgebase/points?wait=true HTTP/1.1" 200 91 "-" "python-httpx/0.25.0" 0.065460
opencopilot-workers-1 | [2024-01-16 07:12:17,126: INFO/ForkPoolWorker-7] HTTP Request: PUT http://qdrant:6333/collections/knowledgebase/points?wait=true "HTTP/1.1 200 OK"
opencopilot-qdrant-1 | 2024-01-16T07:12:19.221687Z INFO actix_web::middleware::logger: 172.30.0.7 "PUT /collections/knowledgebase/points?wait=true HTTP/1.1" 200 91 "-" "python-httpx/0.25.0" 0.113664
opencopilot-workers-1 | [2024-01-16 07:12:19,222: INFO/ForkPoolWorker-7] HTTP Request: PUT http://qdrant:6333/collections/knowledgebase/points?wait=true "HTTP/1.1 200 OK"
opencopilot-qdrant-1 | 2024-01-16T07:12:21.388306Z INFO actix_web::middleware::logger: 172.30.0.7 "PUT /collections/knowledgebase/points?wait=true HTTP/1.1" 200 91 "-" "python-httpx/0.25.0" 0.064480
opencopilot-workers-1 | [2024-01-16 07:12:21,390: INFO/ForkPoolWorker-7] HTTP Request: PUT http://qdrant:6333/collections/knowledgebase/points?wait=true "HTTP/1.1 200 OK"
opencopilot-selenium-1 | 07:12:25.081 WARN [ExternalProcess$Builder.lambda$start$0] - failed to copy the output of process 239
opencopilot-selenium-1 | java.io.IOException: Stream closed
opencopilot-selenium-1 | at java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:176)
opencopilot-selenium-1 | at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:342)
opencopilot-selenium-1 | at java.base/java.io.InputStream.transferTo(InputStream.java:704)
opencopilot-selenium-1 | at org.openqa.selenium.os.ExternalProcess$Builder.lambda$start$0(ExternalProcess.java:209)
opencopilot-selenium-1 | at java.base/java.lang.Thread.run(Thread.java:829)
opencopilot-selenium-1 | 07:12:25.202 INFO [LocalSessionMap.lambda$new$0] - Deleted session from local Session Map, Id: 8e6a396c-c5d3-458e-a66a-4121b110a53f
opencopilot-selenium-1 | 07:12:25.205 INFO [GridModel.release] - Releasing slot for session id 8e6a396c-c5d3-458e-a66a-4121b110a53f
opencopilot-selenium-1 | 07:12:25.207 INFO [SessionSlot.stop] - Stopping session 8e6a396c-c5d3-458e-a66a-4121b110a53f
opencopilot-workers-1 | [2024-01-16 07:12:25,223: INFO/ForkPoolWorker-7] Task workers.tasks.web_crawl.web_crawl[efa251ad-1594-4400-b353-cd1b706ead54] succeeded in 64.27358841899877s: None
opencopilot-llm-server-1 | {"error": "<NotFound '404: Not Found'>", "event": "Internal Server Error", "level": "error", "timestamp": "2024-01-16T07:13:41.405414Z", "exception": [{"exc_type": "NotFound", "exc_value": "404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.", "syntax_error": null, "is_cause": false, "frames": [{"filename": "/usr/local/lib/python3.9/site-packages/flask/app.py", "lineno": 867, "name": "full_dispatch_request", "line": "", "locals": {"self": "<Flask 'app'>", "rv": "None", "e": "<NotFound '404: Not Found'>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/flask/app.py", "lineno": 841, "name": "dispatch_request", "line": "", "locals": {"self": "<Flask 'app'>", "req": "<Request 'http://llm-server:8002/healthcheck' [GET]>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/flask/app.py", "lineno": 450, "name": "raise_routing_exception", "line": "", "locals": {"self": "<Flask 'app'>", "request": "<Request 'http://llm-server:8002/healthcheck' [GET]>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/flask/ctx.py", "lineno": 353, "name": "match_request", "line": "", "locals": {"self": "<RequestContext 'http://llm-server:8002/healthcheck' [GET] of app>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/werkzeug/routing/map.py", "lineno": 624, "name": "match", "line": "", "locals": {"self": "<werkzeug.routing.map.MapAdapter object at 0x7fa01dee1bb0>", "path_info": "/healthcheck", "method": "GET", "return_rule": "True", "query_args": "{}", "websocket": "False", "domain_part": "", "path_part": "/healthcheck"}}]}]}
opencopilot-llm-server-1 | 172.30.0.8 - - [16/Jan/2024 07:13:41] "GET /healthcheck HTTP/1.1" 500 -
opencopilot-llm-server-1 | {"error": "<NotFound '404: Not Found'>", "event": "Internal Server Error", "level": "error", "timestamp": "2024-01-16T07:18:42.053351Z", "exception": [{"exc_type": "NotFound", "exc_value": "404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.", "syntax_error": null, "is_cause": false, "frames": [{"filename": "/usr/local/lib/python3.9/site-packages/flask/app.py", "lineno": 867, "name": "full_dispatch_request", "line": "", "locals": {"self": "<Flask 'app'>", "rv": "None", "e": "<NotFound '404: Not Found'>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/flask/app.py", "lineno": 841, "name": "dispatch_request", "line": "", "locals": {"self": "<Flask 'app'>", "req": "<Request 'http://llm-server:8002/healthcheck' [GET]>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/flask/app.py", "lineno": 450, "name": "raise_routing_exception", "line": "", "locals": {"self": "<Flask 'app'>", "request": "<Request 'http://llm-server:8002/healthcheck' [GET]>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/flask/ctx.py", "lineno": 353, "name": "match_request", "line": "", "locals": {"self": "<RequestContext 'http://llm-server:8002/healthcheck' [GET] of app>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/werkzeug/routing/map.py", "lineno": 624, "name": "match", "line": "", "locals": {"self": "<werkzeug.routing.map.MapAdapter object at 0x7fa01de0d3a0>", "path_info": "/healthcheck", "method": "GET", "return_rule": "True", "query_args": "{}", "websocket": "False", "domain_part": "", "path_part": "/healthcheck"}}]}]}
opencopilot-llm-server-1 | 172.30.0.8 - - [16/Jan/2024 07:18:42] "GET /healthcheck HTTP/1.1" 500 -
opencopilot-llm-server-1 | {"error": "<NotFound '404: Not Found'>", "event": "Internal Server Error", "level": "error", "timestamp": "2024-01-16T07:23:42.557059Z", "exception": [{"exc_type": "NotFound", "exc_value": "404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.", "syntax_error": null, "is_cause": false, "frames": [{"filename": "/usr/local/lib/python3.9/site-packages/flask/app.py", "lineno": 867, "name": "full_dispatch_request", "line": "", "locals": {"self": "<Flask 'app'>", "rv": "None", "e": "<NotFound '404: Not Found'>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/flask/app.py", "lineno": 841, "name": "dispatch_request", "line": "", "locals": {"self": "<Flask 'app'>", "req": "<Request 'http://llm-server:8002/healthcheck' [GET]>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/flask/app.py", "lineno": 450, "name": "raise_routing_exception", "line": "", "locals": {"self": "<Flask 'app'>", "request": "<Request 'http://llm-server:8002/healthcheck' [GET]>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/flask/ctx.py", "lineno": 353, "name": "match_request", "line": "", "locals": {"self": "<RequestContext 'http://llm-server:8002/healthcheck' [GET] of app>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/werkzeug/routing/map.py", "lineno": 624, "name": "match", "line": "", "locals": {"self": "<werkzeug.routing.map.MapAdapter object at 0x7fa03c48a400>", "path_info": "/healthcheck", "method": "GET", "return_rule": "True", "query_args": "{}", "websocket": "False", "domain_part": "", "path_part": "/healthcheck"}}]}]}
@suresh-saini please take a pull from main
and run
docker compose up -d --build --no-deps llm-server
docker compose up -d --build --no-deps worker
Alternatively you can run make install
or make install-arm
Thanks @codebanesr. It proceeded further.
But now it stuck after loading 15 pages. Though I don't see any error in backend logs this time
Is there a setting that is restricting the max number of pages to be indexed.
Is this config or is it hard-coded?
Is there a way to configure OpenCopilot to index all pages?
you can open adminer, then edit chatbot_setting
table.
On the other hand if you want to change the total number of pages being crawled globally for any new copilot, you should change the following environment variable
MAX_PAGES_TO_CRAWL=100
For a list of all configurable parameters:
https://github.com/openchatai/OpenCopilot/blob/main/llm-server/utils/llm_consts.py
cc: @suresh-saini ⬆️ - editing the MAX_PAGES_TO_CRAWL
will allow you to increase beyond 15 items.
Thanks @codebanesr
Awesome, thanks so much!
I appreciate your quick response.
This is similar to the OpenChat issue where I mentioned that I updated the ./backend-server/app/Http/Listeners/StartRecursiveCrawler.php
to increase the max amount of pages OpenChat can index. I was unsure how OpenChat compared to OpenCopilot; thanks for your message. It has helped me understand how OpenCopilot works.
I will close this issue now, feel free to reopen if something has not been answered
Just curious to learn if an alternative for updating the database, as suggested above, can be to update the python file instead?
vi llm-server/models/repository/copilot_settings.py
For example, update the get_chatbot_settings
function's limit
argument from 100
to 1000
:
def get_chatbot_settings(skip: int = 0, limit: int = 1000):