invalid character '<' looking for beginning of value
ammmze opened this issue · comments
I doubt this is something we can resolve here (aside from maybe adding some retry mechanism), mostly just trying to figure out if this is a broader issue than just me. I'm not sure if this is something triggered by the rapid fire calls from this provider, or somehow related to my vyos install. But I frequently get errors ("invalid character '<' looking for beginning of value") that seem to go away with enough retries. With some debugging I found that the vyos api is serving up an HTTP 502 Bad Gateway (with HTML body...hence the <). Looking around in logs in vyos, it looks like vyos is segfaulting. I was running on an old 1.4 rolling image from June last year, but get the same thing after upgrading to the latest version. I'm just experimenting with vyos before I go to put it into place on my network. Have you happened to run across this?
Feb 12 21:36:16 kernel: vyos-http-api-s[6134]: segfault at 56b5 ip 00007fbe8f4f0d76 sp 00007fbe8a200508 error 4 in libc-2.31.so[7fbe8f478000+15a000] likely on CPU 1 (core 1, socket 0)
Feb 12 21:36:16 kernel: Code: 0f 1f 40 00 66 0f ef c0 66 0f ef c9 66 0f ef d2 66 0f ef db 48 89 f8 48 89 f9 48 81 e1 ff 0f 00 00 48 81 f9 cf 0f 00 00 77 6a <f3> 0f 6f 20 66 0f 74 e0 66 0f d7 d4 85 d2 74 04 0f bc c2 c3 48 83
Feb 12 21:36:16 systemd[1]: vyos-http-api.service: Main process exited, code=killed, status=11/SEGV
Seems like it's an issue with concurrent requests. I am able to reproduce the 502's with autocannon.
$ npx autocannon --duration 10 --connections 2 --renderStatusCodes --method POST --form '{"data": {"type":"text", "value": "{\"op\": \"showConfig\", \"path\": []}"}, "key": {"type":"text","value":"my-secret-key"}}' 'https://vyos:8443/retrieve'
Running 10s test @ https://vyos:8443/retrieve
2 connections
┌─────────┬──────┬───────┬───────┬────────┬──────────┬──────────┬────────┐
│ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │
├─────────┼──────┼───────┼───────┼────────┼──────────┼──────────┼────────┤
│ Latency │ 4 ms │ 50 ms │ 92 ms │ 107 ms │ 46.58 ms │ 24.93 ms │ 365 ms │
└─────────┴──────┴───────┴───────┴────────┴──────────┴──────────┴────────┘
┌───────────┬─────────┬─────────┬────────┬────────┬────────┬─────────┬─────────┐
│ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │
├───────────┼─────────┼─────────┼────────┼────────┼────────┼─────────┼─────────┤
│ Req/Sec │ 245 │ 245 │ 426 │ 557 │ 431.6 │ 83.69 │ 245 │
├───────────┼─────────┼─────────┼────────┼────────┼────────┼─────────┼─────────┤
│ Bytes/Sec │ 76.9 kB │ 76.9 kB │ 134 kB │ 175 kB │ 136 kB │ 26.3 kB │ 76.9 kB │
└───────────┴─────────┴─────────┴────────┴────────┴────────┴─────────┴─────────┘
┌──────┬───────┐
│ Code │ Count │
├──────┼───────┤
│ 502 │ 4316 │
└──────┴───────┘
Req/Bytes counts sampled once per second.
# of samples: 10
0 2xx responses, 4316 non 2xx responses
4k requests in 10.02s, 1.36 MB read
I posted a topic in Vyos forums about it...we'll see what they say. But I am still curious to see if you are seeing this issue too.
I haven't seen this issue before, but creating an issue on Phabricator is probably the best place for this, given it's an error on their side.
Sounds good, must be something with my setup then. Thank you!
FYI ... in case anyone else runs into this, I brought it up on their Phabricator and confirmed it was an issue and in my testing it has been resolved with the rolling image vyos-1.4-rolling-202302231931-amd64.