AWS 404 response with XML-encoded body causes JSON parsing exception

[Scott Bale]

This issue was uncovered while reproducing #174.

Dependencies

I was able to reproduce #174 both with the dependencies listed there and the following:

:deps    {org.clojure/clojure       {:mvn/version "1.11.1"}
          org.clojure/core.async    {:mvn/version "1.5.648"}
          org.clojure/tools.logging {:mvn/version "1.2.4"}
          org.clojure/data.json     {:mvn/version "2.4.0"}
          org.clojure/data.xml      {:mvn/version "0.2.0-alpha6"}
          com.cognitect/http-client {:mvn/version "1.0.115"}}

Description with failing test case

Here is a simplification of the #174 steps to reproduce:

(def kvs 
 (aws/client 
  {:api :kinesisvideo 
  :region "eu-west-1" 
  :credentials-provider (cred/profile-credentials-provider "devpl2")}))

; get the arn of the first stream in the KVSs
(def stream-arn
  (-> (aws/invoke kvs {:op :ListStreams :request {}})
      :StreamInfoList
      (first)
      :StreamARN))

;; Create a KV Media client which defaults to the wrong endpoint without an :endpoint-override

(def kvmedia 
 (aws/client 
  {:api :kinesis-video-media 
   :region "eu-west-1" 
   :credentials-provider (cred/profile-credentials-provider "devpl2")}))

;; invoke the API
(aws/invoke kvmedia {:op :GetMedia
                     :request {:StartSelector {:StartSelectorType "EARLIEST"}
                               :StreamARN stream-arn}})

Stack traces

Here is an abbreviated stack trace, see #174 for full stack trace.

{:cognitect.anomalies/category :cognitect.anomalies/fault
 :cognitect.aws.client/throwable
 #error
  {:cause "JSON error (unexpected character): <"
   :trace
   [[clojure.data.json$_read invokeStatic "json.clj"
     230]
    [clojure.data.json$_read invoke "json.clj" 181]
    [clojure.data.json$read invokeStatic "json.clj" 276]
    [clojure.data.json$read doInvoke "json.clj" 232]
    [clojure.lang.RestFn applyTo "RestFn.java" 139]
    [clojure.core$apply invokeStatic "core.clj" 667]
    [clojure.core$apply invoke "core.clj" 660]
    [clojure.data.json$read_str invokeStatic "json.clj"
     282]
    [clojure.data.json$read_str doInvoke "json.clj" 278]
    [clojure.lang.RestFn invoke "RestFn.java" 439]
    [cognitect.aws.protocols.common$json_parse_error
     invokeStatic "common.clj" 42]
    [cognitect.aws.protocols.common$json_parse_error
     invoke "common.clj" 40]
    [cognitect.aws.protocols.rest$parse_http_response
     invokeStatic "rest.clj" 261]
    [cognitect.aws.protocols.rest$parse_http_response
     invoke "rest.clj" 249]
    ...

Note the JSON error (unexpected character): <". The response from AWS is a 404, the body is the string <UnknownOperationException/> which, being XML instead of the expected/requested JSON, causes the json-parse-error function to throw an uncaught exception.

In other words, although this operation is expected to fail because of the lack of :endpoint-override, the response body is unexpectedly formatted as XML instead of JSON.

This issue is for potentially making the error response parsing more robust in the face of behavior like this.

[Philipp Meier (Billy)]

I want to point out that the http server can ultimately decide what encoding to use when creating a respose. In general it's advised to consider at least the "content-type" of every response to determine how to interpret the http body. (See https://www.rfc-editor.org/rfc/rfc7231#section-3.4.1)

[David Chelimsky]

I want to point out that the http server can ultimately decide what encoding to use when creating a respose. In general it's advised to consider at least the "content-type" of every response to determine how to interpret the http body. (See https://www.rfc-editor.org/rfc/rfc7231#section-3.4.1)

@phmeier-nubank in this case, we send "accept" "application/json" and don't get back any "content-type" header.