Boris-code / feaplat

爬虫管理系统,支持集群,弹性伸缩。支持运行feapder、scrapy、selenium、playwright等各种框架及脚本

Home Page:https://feapder.com/#/feapder_platform/feaplat

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

执行任务后,查看实例异常

Foreverpy opened this issue · comments

启动任务,查看实例时异常
查看容器feapder_backend的执行日志,发现后端执行错误

INFO:     10.0.1.8:34674 - "GET /feapder/task/get_all_tag?project_id=2 HTTP/1.1" 200 OK
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/python3/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py", line 398, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/usr/local/python3/lib/python3.6/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/python3/lib/python3.6/site-packages/fastapi/applications.py", line 201, in __call__
    await super().__call__(scope, receive, send)  # pragma: no cover
  File "/usr/local/python3/lib/python3.6/site-packages/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/python3/lib/python3.6/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/usr/local/python3/lib/python3.6/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/python3/lib/python3.6/site-packages/starlette/middleware/cors.py", line 86, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/usr/local/python3/lib/python3.6/site-packages/starlette/middleware/cors.py", line 142, in simple_response
    await self.app(scope, receive, send)
  File "/usr/local/python3/lib/python3.6/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/usr/local/python3/lib/python3.6/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/python3/lib/python3.6/site-packages/starlette/routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/python3/lib/python3.6/site-packages/starlette/routing.py", line 227, in handle
    await self.app(scope, receive, send)
  File "/usr/local/python3/lib/python3.6/site-packages/starlette/routing.py", line 41, in app
    response = await func(request)
  File "/usr/local/python3/lib/python3.6/site-packages/fastapi/routing.py", line 202, in app
    dependant=dependant, values=values, is_coroutine=is_coroutine
  File "/usr/local/python3/lib/python3.6/site-packages/fastapi/routing.py", line 148, in run_endpoint_function
    return await dependant.call(**values)
  File "<frozen views.task_instance_view>", line 94, in get_task_instance
  File "task_instance.py", line 40, in core.task_instance.get_task_instance
AttributeError: 'NoneType' object has no attribute 'get'
INFO:     10.0.1.8:34678 - "POST /feapder/task_instance/get_task_instance HTTP/1.1" 500 Internal Server Error
commented

电脑系统是苹果m1么

电脑系统是苹果m1么

爬虫项目编写是在windows系统,feaplat在centos服务器上部署的;
单个服务器是没有问题的,我通过docker swarm添加了一台服务器后,再次启动任务就无法正常启动,查看实例页面也无法正常返回数据,查看feapder_backend容器的日志发现有上述的异常日志出现。

commented

那可能是节点的镜像没拉下来,在后添加的服务器上执行
docker pull registry.cn-hangzhou.aliyuncs.com/feapderd/feapder:2.2

好的,我顺便了解一下,在有多个worker服务器的情况下,feaplat平台能否支持任务启动时指定某台服务器启动,我在添加任务和启动任务过程中都没有发现此选项。在实际使用中,各个服务器的运行环境可能有差异,某些采集任务可能需要指定的服务器运行。

commented

不可以
feaplat里的任务是在docker里运行,没运行环境不一致一说
至于服务器资源的不同,feaplat会自己调度,负载均衡。

好的,了解了,感谢