NVD API sometimes returns gibberish

[Arnout Engelen]

After a number of successful fetches I sometimes see:

ERROR
io.github.jeremylong.nvdlib.NvdApiException: Failed to parse JSON starting with: "??`I?%&/m?{J?J??t?`$??@???????iG#)?*??eVe]f@??????{????{????;?N'????\fdl??J???!?"
	at io.github.jeremylong.nvdlib.NvdCveApi.next(NvdCveApi.java:294)

I haven't verified whether the NVD API is really returning gibberish or that there's something going wrong on the application side. I did check that the response has an application/json content type as usual.

[Jeremy Long]

I'm gong to bet we can do better here:

https://github.com/jeremylong/vuln-tools/blob/ee5de12fb479516db20529ec4a41d86d4a82a52b/nvd-lib/src/main/java/io/github/jeremylong/nvdlib/NvdCveApi.java#L286-L288

Instead of just using the getBodyBytes() we should validate it is correct. In cases that I have seen the getBody() return null - there was actual JSON in the getBodyBytes()...

[Arnout Engelen]

I added a log statement and when it happened again the jibberish came from getBodyText.

[Jeremy Long]

so we'll need to put a retry if we get gibberish... love stable APIs.

[meha]

I also faced this issue recently. Adding screenshot of the error with the nvd mirroring.

Would it be good to add some retry logic in the library to resolve it?
For now I was just thinking on the lines of retry based on JsonParseException. But we can add more conditions if needed

[Jeremy Long]

while the underlying HTTP request will return garbage - the updated library will retry automatically.