illegal_argument_exception when using AWS

[mcade]

Hi Ryan

I'm using AWS elasticsearch version 6.4. When I try to create an index, I get the following error:

Elastic.Index.create("post")
{:error, 400,
 %{
   "error" => %{
     "reason" => "request [/elastic_dev_post] contains unrecognized parameters: [X-Amz-Algorithm], [X-Amz-Credential], [X-Amz-Date], [X-Amz-Expires], [X-Amz-Signature], [X-Amz-SignedHeaders]",
     "root_cause" => [
       %{
         "reason" => "request [/elastic_dev_post] contains unrecognized parameters: [X-Amz-Algorithm], [X-Amz-Credential], [X-Amz-Date], [X-Amz-Expires], [X-Amz-Signature], [X-Amz-SignedHeaders]",
         "type" => "illegal_argument_exception"
       }
     ],
     "type" => "illegal_argument_exception"
   },
   "status" => 400
 }}

Here are the parameters being sent to AWSAuth:
AWSAuth.sign_url("xxxx", "xxxxx", "put", %URI{authority: "search-blog-dev-sdgsergdsgfdg23423.us-east-1.es.amazonaws.com", fragment: nil, host: "search-blog-dev-sdgsergdsgfdg23423.us-east-1.es.amazonaws.com", path: "/elastic_dev_post", port: 443, query: nil, scheme: "https", userinfo: nil}, "us-east-1", "es", %{"Content-Type" => "application/json"}, ~N[2019-03-01 04:36:31.902975], [])

Which creates a URL like this:
https://search-blog-dev-sdgsergdsgfdg23423.us-east-1.es.amazonaws.com/elastic_dev_post?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=XXXX%2Fus-east-1%2Fes%2Faws4_request&X-Amz-Date=20190301T120111Z&X-Amz-Expires=86400&X-Amz-Signature=xxxxxXXXXXXXxxxxxx&X-Amz-SignedHeaders=Content-Type%3Bhost

Only thing I could find on google was this post of someone getting the same error message but it got no replies. I feel like if I'm the only person getting this error when using the library with AWS I must have something configured wrong?

Here's my config:

config :elastic,
  base_url: "https://search-blog-dev-sdgsergdsgfdg23423.us-east-1.es.amazonaws.com",
  index_prefix: "elastic",
  use_mix_env: true,
  aws: %{
    enabled: true,
    access_key_id: System.get_env("AWS_ACCESS_KEY_ID"),
    secret_access_key: System.get_env("AWS_SECRET_ACCESS_KEY"),
    region: "us-east-1"
  }

Any help is appreciated, thanks

[Ryan Bigg]

Just looked at this today briefly. I can replicate this issue:

{:error, 400,
 %{
   "error" => %{
     "reason" => "request [/elastic_dev_post] contains unrecognized parameters: [X-Amz-Algorithm], [X-Amz-Credential], [X-Amz-Date], [X-Amz-Expires], [X-Amz-Signature], [X-Amz-SignedHeaders]",
     "root_cause" => [
       %{
         "reason" => "request [/elastic_dev_post] contains unrecognized parameters: [X-Amz-Algorithm], [X-Amz-Credential], [X-Amz-Date], [X-Amz-Expires], [X-Amz-Signature], [X-Amz-SignedHeaders]",
         "type" => "illegal_argument_exception"
       }
     ],
     "type" => "illegal_argument_exception"
   },
   "status" => 400
 }}

I am not sure yet what causes it. I can take a look again tomorrow morning.

[Ryan Bigg]

I suspect that AWS no longer accepts passing these parameters through in the query URL, but now instead prefers them in headers. I think aws_auth does it via params, not headers. I might need to change Elastic... but I will investigate first. I think elastic needs to use AWSAuth.sign_authorization_header.

[Ryan Bigg]

Could you please try using the branch from #24? That seems to fix my issues. I am hesitant to merge it as I do not know if it will break earlier AWS ES versions yet. I will check on a few tomorrow.

[Ryan Bigg]

As I mentioned on #24, this change seems to work for me. It would be great to get a 2nd opinion from you before I go and release a new hex package though. If I don't hear from you by next week, then I will do one anyway.

[Ryan Bigg]

Changed my mind this morning and had a burst of confidence, so released 3.5.0 with this fix.

[mcade]

The change checks out, works for me now! Thanks for digging into it