Bug: MISP does not send a UA with (at least feed related) HTTP requests, preventing requests from traversing Cloudflare

[UFOSmuggler]

Actual behavior

See here in gitter: https://matrix.to/#/!obmOUUhIzVdqeMFLvM:gitter.im/$YwdlcpytGVdaVg6m6CT5UrNNetswMYP-2W945Nch7es?via=gitter.im&via=matrix.org&via=matrix.circl.lu

When fetching feeds, MISP does not seem to send a UA header. Here is the output of a feed request hitting nc:

$ nc -l -p 1111

GET / HTTP/1.1
Host: 192.168.5.100:1111
Accept-Encoding: deflate, gzip, br
Accept: application/json, text/plain, text/*
MISP-version: 2.4.193
MISP-uuid: 29d3eae9-8963-4080-ac8e-46c4a6867682
commit: 2b28ec0c6a812467a026cba31834e113b99ebd38

For some feeds, for example PhishTank, the feed is behind a CDN or protective service such as Cloudflare.

Here is an example of attempting to request this feed without a UA header:

$ curl -v -H 'User-Agent:' -H 'Accept-Encoding: identity' -H 'Accept: application/json, text/plain, text/*' -H 'MISP-version: 2.4.193' -H 'MISP-uuid: 29d3eae9-8963-4080-ac8e-46c4a6867682' -H 'commit: 2b28ec0c6a812467a026cba31834e113b99ebd38' https://data.phishtank.com/data/online-valid.csv -o -

...snip...
> GET /data/online-valid.csv HTTP/2
> Host: data.phishtank.com
> accept-encoding: identity
> accept: application/json, text/plain, text/*
> misp-version: 2.4.193
> misp-uuid: 29d3eae9-8963-4080-ac8e-46c4a6867682
> commit: 2b28ec0c6a812467a026cba31834e113b99ebd38
> 
...snip...
< HTTP/2 403 
< date: Tue, 11 Jun 2024 22:31:26 GMT
< content-type: text/html; charset=UTF-8
< x-frame-options: SAMEORIGIN
< referrer-policy: same-origin
< cache-control: max-age=15
< expires: Tue, 11 Jun 2024 22:31:41 GMT
< set-cookie: __cf_bm=5dIAcO5CK3fhIA.TbkrUc3.ugaWzIxBfqbaerrElBbA-1718145086-1.0.1.1-j_wkxHIAXDzJTJny6B4h89yZW0o21UtfTC5QjrkBG67wxw8HoZXOaZGo.Nl1Ct_p5GSgLTgUcX_.hQA6NwWrHQ; path=/; expires=Tue, 11-Jun-24 23:01:26 GMT; domain=.phishtank.com; HttpOnly; Secure
< vary: Accept-Encoding
< strict-transport-security: max-age=15552000
< server: cloudflare
...snip...
            <p data-translate="blocked_why_detail">This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.</p>

If I simply add the header "user-agent: MISP 2.4.193" all of a sudden, this works fine.

In MISP, adding a user-agent header into the "additional headers" box in the feed definition causes this feed to work, but I view this as a workaround, I think the fix is adding a UA header to all requests.

Given that over time more and more things will use Cloudflare and similar services, which will filter anomalous things like missing expected headers, it makes sense to simply add a UA header.

Expected behavior

MISP should send a UA header.

Steps to reproduce

1. Try to fetch the PhishTank feed from MISP, observe error.log.
2. Try with unset UA header: curl -v -H 'User-Agent:' -H 'Accept-Encoding: identity' -H 'Accept: application/json, text/plain, text/*' -H 'MISP-version: 2.4.193' -H 'MISP-uuid: 29d3eae9-8963-4080-ac8e-46c4a6867682' -H 'commit: 2b28ec0c6a812467a026cba31834e113b99ebd38' https://data.phishtank.com/data/online-valid.csv -o -
3. Try with set UA header: curl -v -H 'User-Agent: MISP 2.4.193' -H 'Accept-Encoding: identity' -H 'Accept: application/json, text/plain, text/*' -H 'MISP-version: 2.4.193' -H 'MISP-uuid: 29d3eae9-8963-4080-ac8e-46c4a6867682' -H 'commit: 2b28ec0c6a812467a026cba31834e113b99ebd38' https://data.phishtank.com/data/online-valid.csv -o -

Version

2.4.193

Operating System

misp-docker:latest

Operating System version

misp-docker:latest

PHP version

misp-docker:latest

Browser

Chrome, curl

Browser version

No response

Relevant log output

No response

Extra attachments

No response

Code of Conduct

• I agree to follow this project's Code of Conduct