got 500 code with node-crawler but curl and axios works as expected.
LOGANLEEE opened this issue · comments
here is the Target site url : http://www.todayhumor.co.kr/board/list.php?table=bestofbest&page=1
and got response body is this
`
<title>500 Internal Server Error</title>Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, [email protected] and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.
<script data-cfasync="false" src="/cdn-cgi/scripts/5c5dd728/cloudflare-static/email-decode.min.js"></script> `curl & axios works fine.
I have no idea where start to debug.
Maybe you could get work by setting user agent?
Maybe you could get work by setting user agent?
I will try thank you.