Giters

openchatai / OpenCopilot

🤖 🔥 Language-to-actions engine

Home Page:https://opencopilot.so

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Error while loading knowledge base from url

suresh-saini opened this issue · comments

commented

Describe the bug
When I add knowledge base from url it adds few pages after that it keeps rolling but no further progress.

It shows few error on backend logs

To Reproduce
Steps to reproduce the behavior:

  1. Clone the repository
  2. Get the servers up using make install
  3. Open the portal http://localhost:8888
  4. Create a copilot
  5. Go to 'Knowledge Base'
  6. Click on 'Add Data Source->URL(s)'
  7. Enter url as https://gamechange.com
  8. Click on 'Add'

Expected behavior
All the pages/subpages (around 40 in total) should have been added.

Screenshots
Screenshot from 2024-01-16 16-02-31

image

Desktop (please complete the following information):

  • OS: Ubuntu
  • Browser: Chrome
  • Version: 120.0.6099.199 (Official Build) (64-bit)

Smartphone (please complete the following information):
NA

Additional context
below are the backend log entries where I see few errors

opencopilot-llm-server-1  | 172.30.0.10 - - [16/Jan/2024 07:10:50] "POST /backend/copilot/ HTTP/1.0" 200 -
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:10:50 +0000] "POST /backend/copilot/ HTTP/1.1" 200 628 "http://localhost:8888/create/copilot" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-llm-server-1  | 172.30.0.10 - - [16/Jan/2024 07:10:50] "GET /backend/actions/bot/71175cae-da82-489f-900f-42607113fa6c HTTP/1.0" 200 -
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:10:50 +0000] "GET /backend/actions/bot/71175cae-da82-489f-900f-42607113fa6c HTTP/1.1" 200 3 "http://localhost:8888/create/copilot" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:10:57 +0000] "GET /copilot/71175cae-da82-489f-900f-42607113fa6c?_rsc=14gk8 HTTP/1.1" 200 188 "http://localhost:8888/create/copilot" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:11:00 +0000] "GET /copilot/71175cae-da82-489f-900f-42607113fa6c?_rsc=5xwpy HTTP/1.1" 200 7094 "http://localhost:8888/create/copilot" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-llm-server-1  | 172.30.0.10 - - [16/Jan/2024 07:11:00] "GET /backend/copilot/71175cae-da82-489f-900f-42607113fa6c HTTP/1.0" 200 -
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:11:00 +0000] "GET /backend/copilot/71175cae-da82-489f-900f-42607113fa6c HTTP/1.1" 200 640 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:11:00 +0000] "GET /copilot/71175cae-da82-489f-900f-42607113fa6c/workflow?_rsc=1arzg HTTP/1.1" 200 185 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:11:00 +0000] "GET /copilot/71175cae-da82-489f-900f-42607113fa6c/settings?_rsc=1arzg HTTP/1.1" 200 198 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:11:00 +0000] "GET /copilot/71175cae-da82-489f-900f-42607113fa6c/conversations?_rsc=1arzg HTTP/1.1" 200 201 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:11:00 +0000] "GET /copilot/71175cae-da82-489f-900f-42607113fa6c/knowledge?_rsc=1arzg HTTP/1.1" 200 199 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-llm-server-1  | 172.30.0.10 - - [16/Jan/2024 07:11:01] "GET /socket.io/?EIO=4&transport=polling&t=OqH6_Ng HTTP/1.1" 200 -
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:11:01 +0000] "GET /socket.io/?EIO=4&transport=polling&t=OqH6_Ng HTTP/1.1" 200 108 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-llm-server-1  | 172.30.0.10 - - [16/Jan/2024 07:11:01] "POST /socket.io/?EIO=4&transport=polling&t=OqH6_PI&sid=UDfunb5pCvKU-BSjAAAA HTTP/1.1" 200 -
opencopilot-llm-server-1  | 172.30.0.10 - - [16/Jan/2024 07:11:01] "GET /socket.io/?EIO=4&transport=polling&t=OqH6_PM&sid=UDfunb5pCvKU-BSjAAAA HTTP/1.1" 200 -
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:11:01 +0000] "POST /socket.io/?EIO=4&transport=polling&t=OqH6_PI&sid=UDfunb5pCvKU-BSjAAAA HTTP/1.1" 200 12 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:11:01 +0000] "GET /socket.io/?EIO=4&transport=polling&t=OqH6_PM&sid=UDfunb5pCvKU-BSjAAAA HTTP/1.1" 200 43 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-llm-server-1  | 172.30.0.10 - - [16/Jan/2024 07:11:01] "GET /backend/chat/init HTTP/1.0" 200 -
opencopilot-llm-server-1  | 172.30.0.10 - - [16/Jan/2024 07:11:01] "GET /socket.io/?EIO=4&transport=polling&t=OqH6_Qu&sid=UDfunb5pCvKU-BSjAAAA HTTP/1.1" 200 -
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:11:01 +0000] "GET /backend/chat/init HTTP/1.1" 200 91 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:11:01 +0000] "GET /socket.io/?EIO=4&transport=polling&t=OqH6_Qu&sid=UDfunb5pCvKU-BSjAAAA HTTP/1.1" 200 11 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-llm-server-1  | 172.30.0.10 - - [16/Jan/2024 07:11:01] "GET /backend/chat/init HTTP/1.0" 200 -
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:11:01 +0000] "GET /backend/chat/init HTTP/1.1" 200 91 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:11:03 +0000] "GET /copilot/71175cae-da82-489f-900f-42607113fa6c/workflow?_rsc=1vw3t HTTP/1.1" 200 1979 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:11:03 +0000] "GET /random_icons_2.svg HTTP/1.1" 304 0 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c/workflow" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-llm-server-1  | 172.30.0.10 - - [16/Jan/2024 07:11:03] "GET /backend/flows/bot/71175cae-da82-489f-900f-42607113fa6c HTTP/1.0" 200 -
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:11:03 +0000] "GET /backend/flows/bot/71175cae-da82-489f-900f-42607113fa6c HTTP/1.1" 200 3 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c/workflow" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:11:05 +0000] "GET /copilot/71175cae-da82-489f-900f-42607113fa6c/knowledge?_rsc=136t3 HTTP/1.1" 200 3430 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c/workflow" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-llm-server-1  | 172.30.0.10 - - [16/Jan/2024 07:11:05] "GET /backend/data_sources/b/71175cae-da82-489f-900f-42607113fa6c HTTP/1.0" 200 -
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:11:05 +0000] "GET /backend/data_sources/b/71175cae-da82-489f-900f-42607113fa6c HTTP/1.1" 200 36 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c/knowledge" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-llm-server-1  | 172.30.0.10 - - [16/Jan/2024 07:11:15] "GET /backend/data_sources/b/71175cae-da82-489f-900f-42607113fa6c HTTP/1.0" 200 -
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:11:15 +0000] "GET /backend/data_sources/b/71175cae-da82-489f-900f-42607113fa6c HTTP/1.1" 200 36 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c/knowledge" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-workers-1     | [2024-01-16 07:11:20,941: INFO/MainProcess] Task workers.tasks.web_crawl.web_crawl[efa251ad-1594-4400-b353-cd1b706ead54] received
opencopilot-workers-1     | [2024-01-16 07:11:20,950: WARNING/ForkPoolWorker-7] Received: https://gamechange.com, 71175cae-da82-489f-900f-42607113fa6c
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:11:20 +0000] "POST /backend/uploads/file/ingest HTTP/1.1" 200 41 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c/knowledge" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-llm-server-1  | 172.30.0.10 - - [16/Jan/2024 07:11:20] "POST /backend/uploads/file/ingest HTTP/1.0" 200 -
opencopilot-selenium-1    | 07:11:21.822 INFO [LocalDistributor.newSession] - Session request received by the Distributor: 
opencopilot-selenium-1    |  [Capabilities {acceptInsecureCerts: true, browserName: firefox, moz:debuggerAddress: true, pageLoadStrategy: normal}]
opencopilot-selenium-1    | 07:11:25.434 WARN [SeleniumManager.lambda$runCommand$1] - Exception managing firefox: Unable to discover proper geckodriver version in offline mode
opencopilot-llm-server-1  | 172.30.0.10 - - [16/Jan/2024 07:11:25] "GET /backend/data_sources/b/71175cae-da82-489f-900f-42607113fa6c HTTP/1.0" 200 -
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:11:25 +0000] "GET /backend/data_sources/b/71175cae-da82-489f-900f-42607113fa6c HTTP/1.1" 200 359 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c/knowledge" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-llm-server-1  | 172.30.0.10 - - [16/Jan/2024 07:11:35] "GET /backend/data_sources/b/71175cae-da82-489f-900f-42607113fa6c HTTP/1.0" 200 -
opencopilot-nginx-1       | 172.30.0.1 - - [16/Jan/2024:07:11:35 +0000] "GET /backend/data_sources/b/71175cae-da82-489f-900f-42607113fa6c HTTP/1.1" 200 359 "http://localhost:8888/copilot/71175cae-da82-489f-900f-42607113fa6c/knowledge" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
opencopilot-selenium-1    | 07:11:51.712 INFO [LocalNode.newSession] - Session created by the Node. Id: 8e6a396c-c5d3-458e-a66a-4121b110a53f, Caps: Capabilities {acceptInsecureCerts: true, browserName: firefox, browserVersion: 119.0, moz:accessibilityChecks: false, moz:buildID: 20231019122658, moz:debuggerAddress: 127.0.0.1:21699, moz:geckodriverVersion: 0.33.0, moz:headless: false, moz:platformVersion: 6.5.0-14-generic, moz:processID: 249, moz:profile: /tmp/rust_mozprofileJD2MMj, moz:shutdownTimeout: 60000, moz:webdriverClick: true, moz:windowless: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), se:bidiEnabled: false, se:cdp: ws://172.30.0.5:4444/sessio..., se:cdpVersion: 85.0, se:noVncPort: 7900, se:vnc: ws://172.30.0.5:4444/sessio..., se:vncEnabled: true, se:vncLocalAddress: ws://172.30.0.5:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify}
opencopilot-selenium-1    | 07:11:51.768 INFO [LocalDistributor.newSession] - Session created by the Distributor. Id: 8e6a396c-c5d3-458e-a66a-4121b110a53f 
opencopilot-selenium-1    |  Caps: Capabilities {acceptInsecureCerts: true, browserName: firefox, browserVersion: 119.0, moz:accessibilityChecks: false, moz:buildID: 20231019122658, moz:debuggerAddress: 127.0.0.1:21699, moz:geckodriverVersion: 0.33.0, moz:headless: false, moz:platformVersion: 6.5.0-14-generic, moz:processID: 249, moz:profile: /tmp/rust_mozprofileJD2MMj, moz:shutdownTimeout: 60000, moz:webdriverClick: true, moz:windowless: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), se:bidiEnabled: false, se:cdp: ws://172.30.0.5:4444/sessio..., se:cdpVersion: 85.0, se:noVncPort: 7900, se:vnc: ws://172.30.0.5:4444/sessio..., se:vncEnabled: true, se:vncLocalAddress: ws://172.30.0.5:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify}
opencopilot-qdrant-1      | 2024-01-16T07:12:13.063651Z  INFO actix_web::middleware::logger: 172.30.0.7 "PUT /collections/knowledgebase/points?wait=true HTTP/1.1" 200 92 "-" "python-httpx/0.25.0" 0.343635    
opencopilot-workers-1     | [2024-01-16 07:12:13,068: INFO/ForkPoolWorker-7] HTTP Request: PUT http://qdrant:6333/collections/knowledgebase/points?wait=true "HTTP/1.1 200 OK"
opencopilot-qdrant-1      | 2024-01-16T07:12:15.284872Z  INFO actix_web::middleware::logger: 172.30.0.7 "PUT /collections/knowledgebase/points?wait=true HTTP/1.1" 200 91 "-" "python-httpx/0.25.0" 0.049263    
opencopilot-workers-1     | [2024-01-16 07:12:15,285: INFO/ForkPoolWorker-7] HTTP Request: PUT http://qdrant:6333/collections/knowledgebase/points?wait=true "HTTP/1.1 200 OK"
opencopilot-qdrant-1      | 2024-01-16T07:12:17.125837Z  INFO actix_web::middleware::logger: 172.30.0.7 "PUT /collections/knowledgebase/points?wait=true HTTP/1.1" 200 91 "-" "python-httpx/0.25.0" 0.065460    
opencopilot-workers-1     | [2024-01-16 07:12:17,126: INFO/ForkPoolWorker-7] HTTP Request: PUT http://qdrant:6333/collections/knowledgebase/points?wait=true "HTTP/1.1 200 OK"
opencopilot-qdrant-1      | 2024-01-16T07:12:19.221687Z  INFO actix_web::middleware::logger: 172.30.0.7 "PUT /collections/knowledgebase/points?wait=true HTTP/1.1" 200 91 "-" "python-httpx/0.25.0" 0.113664    
opencopilot-workers-1     | [2024-01-16 07:12:19,222: INFO/ForkPoolWorker-7] HTTP Request: PUT http://qdrant:6333/collections/knowledgebase/points?wait=true "HTTP/1.1 200 OK"
opencopilot-qdrant-1      | 2024-01-16T07:12:21.388306Z  INFO actix_web::middleware::logger: 172.30.0.7 "PUT /collections/knowledgebase/points?wait=true HTTP/1.1" 200 91 "-" "python-httpx/0.25.0" 0.064480    
opencopilot-workers-1     | [2024-01-16 07:12:21,390: INFO/ForkPoolWorker-7] HTTP Request: PUT http://qdrant:6333/collections/knowledgebase/points?wait=true "HTTP/1.1 200 OK"
opencopilot-selenium-1    | 07:12:25.081 WARN [ExternalProcess$Builder.lambda$start$0] - failed to copy the output of process 239
opencopilot-selenium-1    | java.io.IOException: Stream closed
opencopilot-selenium-1    |     at java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:176)
opencopilot-selenium-1    |     at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:342)
opencopilot-selenium-1    |     at java.base/java.io.InputStream.transferTo(InputStream.java:704)
opencopilot-selenium-1    |     at org.openqa.selenium.os.ExternalProcess$Builder.lambda$start$0(ExternalProcess.java:209)
opencopilot-selenium-1    |     at java.base/java.lang.Thread.run(Thread.java:829)
opencopilot-selenium-1    | 07:12:25.202 INFO [LocalSessionMap.lambda$new$0] - Deleted session from local Session Map, Id: 8e6a396c-c5d3-458e-a66a-4121b110a53f
opencopilot-selenium-1    | 07:12:25.205 INFO [GridModel.release] - Releasing slot for session id 8e6a396c-c5d3-458e-a66a-4121b110a53f
opencopilot-selenium-1    | 07:12:25.207 INFO [SessionSlot.stop] - Stopping session 8e6a396c-c5d3-458e-a66a-4121b110a53f
opencopilot-workers-1     | [2024-01-16 07:12:25,223: INFO/ForkPoolWorker-7] Task workers.tasks.web_crawl.web_crawl[efa251ad-1594-4400-b353-cd1b706ead54] succeeded in 64.27358841899877s: None
opencopilot-llm-server-1  | {"error": "<NotFound '404: Not Found'>", "event": "Internal Server Error", "level": "error", "timestamp": "2024-01-16T07:13:41.405414Z", "exception": [{"exc_type": "NotFound", "exc_value": "404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.", "syntax_error": null, "is_cause": false, "frames": [{"filename": "/usr/local/lib/python3.9/site-packages/flask/app.py", "lineno": 867, "name": "full_dispatch_request", "line": "", "locals": {"self": "<Flask 'app'>", "rv": "None", "e": "<NotFound '404: Not Found'>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/flask/app.py", "lineno": 841, "name": "dispatch_request", "line": "", "locals": {"self": "<Flask 'app'>", "req": "<Request 'http://llm-server:8002/healthcheck' [GET]>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/flask/app.py", "lineno": 450, "name": "raise_routing_exception", "line": "", "locals": {"self": "<Flask 'app'>", "request": "<Request 'http://llm-server:8002/healthcheck' [GET]>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/flask/ctx.py", "lineno": 353, "name": "match_request", "line": "", "locals": {"self": "<RequestContext 'http://llm-server:8002/healthcheck' [GET] of app>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/werkzeug/routing/map.py", "lineno": 624, "name": "match", "line": "", "locals": {"self": "<werkzeug.routing.map.MapAdapter object at 0x7fa01dee1bb0>", "path_info": "/healthcheck", "method": "GET", "return_rule": "True", "query_args": "{}", "websocket": "False", "domain_part": "", "path_part": "/healthcheck"}}]}]}
opencopilot-llm-server-1  | 172.30.0.8 - - [16/Jan/2024 07:13:41] "GET /healthcheck HTTP/1.1" 500 -
opencopilot-llm-server-1  | {"error": "<NotFound '404: Not Found'>", "event": "Internal Server Error", "level": "error", "timestamp": "2024-01-16T07:18:42.053351Z", "exception": [{"exc_type": "NotFound", "exc_value": "404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.", "syntax_error": null, "is_cause": false, "frames": [{"filename": "/usr/local/lib/python3.9/site-packages/flask/app.py", "lineno": 867, "name": "full_dispatch_request", "line": "", "locals": {"self": "<Flask 'app'>", "rv": "None", "e": "<NotFound '404: Not Found'>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/flask/app.py", "lineno": 841, "name": "dispatch_request", "line": "", "locals": {"self": "<Flask 'app'>", "req": "<Request 'http://llm-server:8002/healthcheck' [GET]>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/flask/app.py", "lineno": 450, "name": "raise_routing_exception", "line": "", "locals": {"self": "<Flask 'app'>", "request": "<Request 'http://llm-server:8002/healthcheck' [GET]>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/flask/ctx.py", "lineno": 353, "name": "match_request", "line": "", "locals": {"self": "<RequestContext 'http://llm-server:8002/healthcheck' [GET] of app>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/werkzeug/routing/map.py", "lineno": 624, "name": "match", "line": "", "locals": {"self": "<werkzeug.routing.map.MapAdapter object at 0x7fa01de0d3a0>", "path_info": "/healthcheck", "method": "GET", "return_rule": "True", "query_args": "{}", "websocket": "False", "domain_part": "", "path_part": "/healthcheck"}}]}]}
opencopilot-llm-server-1  | 172.30.0.8 - - [16/Jan/2024 07:18:42] "GET /healthcheck HTTP/1.1" 500 -
opencopilot-llm-server-1  | {"error": "<NotFound '404: Not Found'>", "event": "Internal Server Error", "level": "error", "timestamp": "2024-01-16T07:23:42.557059Z", "exception": [{"exc_type": "NotFound", "exc_value": "404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.", "syntax_error": null, "is_cause": false, "frames": [{"filename": "/usr/local/lib/python3.9/site-packages/flask/app.py", "lineno": 867, "name": "full_dispatch_request", "line": "", "locals": {"self": "<Flask 'app'>", "rv": "None", "e": "<NotFound '404: Not Found'>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/flask/app.py", "lineno": 841, "name": "dispatch_request", "line": "", "locals": {"self": "<Flask 'app'>", "req": "<Request 'http://llm-server:8002/healthcheck' [GET]>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/flask/app.py", "lineno": 450, "name": "raise_routing_exception", "line": "", "locals": {"self": "<Flask 'app'>", "request": "<Request 'http://llm-server:8002/healthcheck' [GET]>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/flask/ctx.py", "lineno": 353, "name": "match_request", "line": "", "locals": {"self": "<RequestContext 'http://llm-server:8002/healthcheck' [GET] of app>"}}, {"filename": "/usr/local/lib/python3.9/site-packages/werkzeug/routing/map.py", "lineno": 624, "name": "match", "line": "", "locals": {"self": "<werkzeug.routing.map.MapAdapter object at 0x7fa03c48a400>", "path_info": "/healthcheck", "method": "GET", "return_rule": "True", "query_args": "{}", "websocket": "False", "domain_part": "", "path_part": "/healthcheck"}}]}]}
commented

@suresh-saini please take a pull from main
and run

docker compose up -d --build --no-deps llm-server
docker compose up -d --build --no-deps worker

Alternatively you can run make install or make install-arm

commented

Thanks @codebanesr. It proceeded further.

But now it stuck after loading 15 pages. Though I don't see any error in backend logs this time
Screenshot from 2024-02-05 17-08-52

commented

Is there a setting that is restricting the max number of pages to be indexed.
Is this config or is it hard-coded?
Is there a way to configure OpenCopilot to index all pages?

commented

you can open adminer, then edit chatbot_setting table.
Screenshot 2024-02-06 at 10 54 22 AM
Screenshot 2024-02-06 at 10 54 43 AM

On the other hand if you want to change the total number of pages being crawled globally for any new copilot, you should change the following environment variable
MAX_PAGES_TO_CRAWL=100

For a list of all configurable parameters:
https://github.com/openchatai/OpenCopilot/blob/main/llm-server/utils/llm_consts.py

commented

cc: @suresh-saini ⬆️ - editing the MAX_PAGES_TO_CRAWL will allow you to increase beyond 15 items.

Thanks @codebanesr
Awesome, thanks so much!
I appreciate your quick response.

This is similar to the OpenChat issue where I mentioned that I updated the ./backend-server/app/Http/Listeners/StartRecursiveCrawler.php to increase the max amount of pages OpenChat can index. I was unsure how OpenChat compared to OpenCopilot; thanks for your message. It has helped me understand how OpenCopilot works.

commented

I will close this issue now, feel free to reopen if something has not been answered

commented

Just curious to learn if an alternative for updating the database, as suggested above, can be to update the python file instead?

vi llm-server/models/repository/copilot_settings.py

For example, update the get_chatbot_settings function's limit argument from 100 to 1000:

  def get_chatbot_settings(skip: int = 0, limit: int = 1000):