Crash when trying to report Kennel API error

[Rachel Evans]

https://zendesk.slack.com/archives/C81T33XTK/p1692346298884879

[129, 138] in /Users/revans/git/github.com/grosser/kennel/lib/kennel/api.rb
   129:         if !response.success? && (response.status != 404 || !ignore_404)
   130:           message = +"Error #{response.status} during #{method.upcase} #{path}\n"
   131:           message << "request:\n#{JSON.pretty_generate(body)}\nresponse:\n" if body
   132:           require 'byebug'
   133:           byebug
=> 134:           message << response.body
   135:           raise message
   136:         end
   137: 
   138:         if response.body.empty?
(byebug) message.encoding
#<Encoding:UTF-8>
(byebug) response.body.encoding
#<Encoding:ASCII-8BIT>
(byebug) message + response.body
*** Encoding::CompatibilityError Exception: incompatible character encodings: UTF-8 and ASCII-8BIT

nil
(byebug) message + String.new(response.body, encoding: message.encoding)
"Error 400 during POST /api/v1/monitor\nrequest:\n{\n  \"name\": \"text-matching-ml-service - (Production) Error Budget Alert on SLO🔒\",\n  \"type\": \"slo alert\",\n  \"query\": \"error_budget(\\\"57f99d0335815223a57b7f978fe62d33\\\").o (etc etc etc)

[Michael Grosser]

was unable to reproduce

[Rachel Evans]

Try this in a zendesk/kennel checkout:

git checkout 3face58c1315d553b21a070fe148a179c8997994
bundle exec rake kennel:update_datadog PROJECT=text_matching_ml_service KENNEL_MARKER_TEXT=grosser-kennel-test

With then a simple patch to kennel-lib to stop on the exception in question:

[132, 141] in /Users/revans/git/github.com/grosser/kennel/lib/kennel/api.rb
   132:           begin
   133:             message << response.body
   134:           rescue StandardError
   135:             require 'byebug'
   136:             byebug
=> 137:             raise
   138:           end
   139:           raise message
   140:         end
   141: 
(byebug) message.encoding
#<Encoding:UTF-8>
(byebug) response.body.encoding
#<Encoding:ASCII-8BIT>
(byebug) response.body
"{\"errors\":[\"The value provided for parameter 'query' is invalid: Alerting on monitor based SLOs currently supports metric monitors. Monitor text-matching-ml-service - (SLO monitor) TMS Error Rate\xF0\x9F\x94\x92 (129905222) is invalid.\"]}"
(byebug) response.headers
{"date"=>"Mon, 21 Aug 2023 04:26:08 GMT", "content-type"=>"application/json", "transfer-encoding"=>"chunked", "connection"=>"keep-alive", "x-frame-options"=>"SAMEORIGIN", "content-security-policy"=>"frame-ancestors 'self'; report-uri https://logs.browser-intake-datadoghq.com/api/v2/logs?dd-api-key=REDACTED&dd-evp-origin=content-security-policy&ddsource=csp-report&ddtags=site%3Adatadoghq.com", "vary"=>"Accept-Encoding", "x-ratelimit-limit"=>"500", "x-ratelimit-period"=>"10", "x-ratelimit-remaining"=>"499", "x-ratelimit-reset"=>"2", "x-ratelimit-name"=>"post_monitors", "x-content-type-options"=>"nosniff", "strict-transport-security"=>"max-age=31536000; includeSubDomains; preload"}
(byebug) 

[Rachel Evans]

... which makes me think this might actually be a datadog bug, perhaps – because there's no response character set mentioned in the headers.

message << response.body.encode(message.encoding, invalid: :replace, undef: :replace) might be a sufficient (plus tests) defence / fix.

[Michael Grosser]

yeah, was able to reproduce ... seems to happen only when creating mon+slo in one go which seems like another bug :/
that fix looks about right too 👍

[Michael Grosser]

I'll make the PR/test, thx!

[Michael Grosser]

can reproduce when creating monitor for a monitor slo
forcing the encoding works 👍
will open a dd issue too so they are aware
... no bug in kennel though, it figured out the ordering just fine