Firecrawl API Key not working, getting 500 error in Dify
Edison-A-N opened this issue · comments
Self Checks
- This is only for bug report, if you would like to ask a question, please head to Discussions.
- I have searched for existing issues search for existing issues, including closed ones.
- I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
- 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
- Please do not modify this template :) and fill in all the required fields.
Dify version
Cloud or Self Hosted
Self Hosted (Docker)
Steps to reproduce
Enter the correct API Key in Firecrawl
Add the Firecrawl API Key in Difi Agent Knowledge
Observe if Difi throws a 500 error
✔️ Expected Behavior
No response
❌ Actual Behavior
worker logs:
api_1 | 2024-06-30 09:07:14,849.849 ERROR [Dummy-156] [app.py:838] - Exception on /console/api/api-key-auth/data-source/binding [POST]
api_1 | Traceback (most recent call last):
api_1 | File "/app/api/controllers/console/auth/data_source_bearer_auth.py", line 51, in post
api_1 | ApiKeyAuthService.create_provider_auth(current_user.current_tenant_id, args)
api_1 | File "/app/api/services/auth/api_key_auth_service.py", line 21, in create_provider_auth
api_1 | auth_result = ApiKeyAuthFactory(args['provider'], args['credentials']).validate_credentials()
api_1 | File "/app/api/services/auth/api_key_auth_factory.py", line 14, in validate_credentials
api_1 | return self.auth.validate_credentials()
api_1 | File "/app/api/services/auth/firecrawl.py", line 37, in validate_credentials
api_1 | self._handle_error(response)
api_1 | File "/app/api/services/auth/firecrawl.py", line 54, in _handle_error
api_1 | error_message = json.loads(response.text).get('error', 'Unknown error occurred')
api_1 | File "/usr/local/lib/python3.10/json/__init__.py", line 346, in loads
api_1 | return _default_decoder.decode(s)
api_1 | File "/usr/local/lib/python3.10/json/decoder.py", line 337, in decode
api_1 | obj, end = self.raw_decode(s, idx=_w(s, 0).end())
api_1 | File "/usr/local/lib/python3.10/json/decoder.py", line 355, in raw_decode
api_1 | raise JSONDecodeError("Expecting value", s, err.value) from None
api_1 | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
api_1 |
api_1 | During handling of the above exception, another exception occurred:
api_1 |
api_1 | Traceback (most recent call last):
api_1 | File "/app/api/.venv/lib/python3.10/site-packages/flask/app.py", line 880, in full_dispatch_request
api_1 | rv = self.dispatch_request()
api_1 | File "/app/api/.venv/lib/python3.10/site-packages/flask/app.py", line 865, in dispatch_request
api_1 | return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
api_1 | File "/app/api/.venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper
api_1 | resp = resource(*args, **kwargs)
api_1 | File "/app/api/.venv/lib/python3.10/site-packages/flask/views.py", line 110, in view
api_1 | return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return]
api_1 | File "/app/api/.venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request
api_1 | resp = meth(*args, **kwargs)
api_1 | File "/app/api/controllers/console/setup.py", line 74, in decorated
api_1 | return view(*args, **kwargs)
api_1 | File "/app/api/libs/login.py", line 91, in decorated_view
api_1 | return current_app.ensure_sync(func)(*args, **kwargs)
api_1 | File "/app/api/controllers/console/wraps.py", line 21, in decorated
api_1 | return view(*args, **kwargs)
api_1 | File "/app/api/controllers/console/auth/data_source_bearer_auth.py", line 53, in post
api_1 | raise ApiKeyAuthFailedError(str(e))
api_1 | controllers.console.auth.error.ApiKeyAuthFailedError: 500 Internal Server Error: Expecting value: line 1 column 1 (char 0)
To resolve the 500 Internal Server Error when using the Firecrawl API Key in Dify, follow these steps:
- Check the API Key: Ensure that the API key is correctly provided and valid.
- Verify the Base URL: Make sure the
base_urlis correct and the Firecrawl service is up and running. - Inspect the Error Message: The error message indicates a JSON decoding issue, which suggests that the response from the Firecrawl API might not be in JSON format. Check the response from the Firecrawl API to understand the root cause.
The _handle_error method in the FirecrawlApp class captures the error message from the response and raises an exception with it. Here is the relevant part of the code:
def _handle_error(self, response, action):
error_message = response.json().get('error', 'Unknown error occurred')
raise Exception(f'Failed to {action}. Status code: {response.status_code}. Error: {error_message}')Additionally, ensure that Firecrawl is properly configured in Dify:
-
Navigate to the Firecrawl Configuration Section:
- Go to the section where you can configure Firecrawl in Dify.
-
Enter the API Key:
- Enter your API key from firecrawl.dev.
-
Configure Firecrawl Options:
- Set various options such as crawl sub-pages, limit, max depth, exclude paths, include only paths, and extract only main content.
-
Run Firecrawl:
- Once configured, run Firecrawl by clicking the "运行" button.
If the error persists, you may need to contact Firecrawl support for further assistance.
For more details, you can refer to the Dify documentation: Firecrawl docs [1][2][3].
This problem cannot be reproduced. Please confirm whether your local connection to Firecrawl can be successful， thanks @Edison-A-N
Thank you. It seems I only need to fill in the Key, and don't need to fill in the base_url. This way it should work successfully. Thank you very much.