Giters

okigan / awscurl

curl-like access to AWS resources with AWS Signature Version 4 request signing.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[FEATURE] enhance awscurl support create/send multipart POST REST request (from file)

opened this issue · comments

commented

Versions

  • OS :
    • cURL => via Windows 10
    • awscurl => via gitlab runner (Docker)
  • OpenSearch 2.11

What's my problem ?
First of all, this closed issue : opensearch-project/opensearch-cli#76
Seems to be close to my problem, but I'm not using opensearch-cli curl. Only awscurl

I try to migrate dashboards objects from one opensearch cluster to another one.
To do that I'm using api provided by opensearch : /_dashboards/api/saved_objects/_import?overwrite=true
But I get errors when I'm trying to post my objects.

How that's done ?
I'm exporting the objects via the opensearch UI (From my first cluster). Dashboards Management > Saved objects > Export (when my dashboard is selected). And I get my objects in a .ndjson file.
After that i'm posting my .ndjson file with awscurl :

Without Content Type header

awscurl --service es --region ${region} -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \
  -H 'osd-xsrf: true' \
  -d '${dashboard_content}'
{
   "statusCode":415,
   "error":"Unsupported Media Type",
   "message":"Unsupported Media Type"
}

Without Boundary

awscurl --service es --region ${region} -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \
  -H 'Content-Type: multipart/form-data' \
  -H 'osd-xsrf: true' \
  -d '${dashboard_content}'
{
   "statusCode":400,
   "error":"Bad Request",
   "message":"Invalid content-type header: multipart missing boundary"
}

With a random boundary

awscurl --service es --region ${region} -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \
  -H 'Content-Type: multipart/form-data; boundary=---------BOUNDARY;' \
  -H 'osd-xsrf: true' \
  -d '${dashboard_content}'

image

image

By only using curl, and don't put Content Type header, it works fine.

curl -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \
  -H "osd-xsrf: true"
  -F "file=@data/test_dashboard.ndjson"
{
   "successCount":6,
   "success":true,
   "successResults":[
      {
         "type":"index-pattern",
         "id":"610267e0-7f21-11ee-80e6-370892a64263",
         "meta":{
            "title":"documents-metadata-v2",
            "icon":"indexPatternApp"
         },
         "overwrite":true
      },
      {
         "type":"visualization",
         "id":"da95cc90-7f22-11ee-9e31-e14f5ec8461b",
         "meta":{
            "title":"count by cat_doc",
            "icon":"visualizeApp"
         },
         "overwrite":true
      },
      {
         "type":"index-pattern",
         "id":"d3d7af60-4c81-11e8-b3d7-01146121b73d",
         "meta":{
            "title":"opensearch_dashboards_sample_data_flights",
            "icon":"indexPatternApp"
         },
         "overwrite":true
      },
      {
         "type":"search",
         "id":"571aaf70-4c88-11e8-b3d7-01146121b73d",
         "meta":{
            "title":"[Flights] Flight Log",
            "icon":"discoverApp"
         },
         "overwrite":true
      },
      {
         "type":"visualization",
         "id":"94003870-dbb5-11ee-81ba-b7ae96977aa8",
         "meta":{
            "title":"test_object_dashboard",
            "icon":"visualizeApp"
         },
         "overwrite":true
      },
      {
         "type":"dashboard",
         "id":"2d7aceb0-7f23-11ee-9e31-e14f5ec8461b",
         "meta":{
            "title":"test_dashboard",
            "icon":"dashboardApp"
         },
         "overwrite":true
      }
   ]
}
commented
commented

Yes, cURL doesn't need to explicit the Content Type.
Here's the cURL with verbose option :

curl -v -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \
-H "osd-xsrf: true" \
-F "file=@data/test_dashboard.ndjson"

And the result :

Note: Unnecessary use of -X or --request, POST is already inferred.
* Uses proxy env variable no_proxy == '****'
*   Trying [::1]:8080...
*   Trying ${domain_ip}:8080...
* Connected to localhost (127.0.0.1) port 8080
> POST /_dashboards/api/saved_objects/_import?overwrite=true HTTP/1.1
> Host: ${domain_url}
> User-Agent: curl/8.4.0
> Accept: */*
> osd-xsrf: true
> Content-Length: 40384
> Content-Type: multipart/form-data; boundary=------------------------wOWPtrGc1TrXmDPpX5GfNt
>
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Cache-Control: private, no-cache, no-store, must-revalidate
< Content-Length: 947
< Content-Type: application/json; charset=utf-8
< Date: Tue, 12 Mar 2024 09:44:45 GMT
< Osd-Name: osd
< Set-Cookie: security_authentication_saml1=Fe26.2**838c01d554bcb2b7d4e5ead20dafc498fcdd74e1faf6b74c5d1348a2326b713c*D_5paBXbChAFcjTpZ00nLg*YUmGokrp_eV_EEv_2S0NZecOyi1Lwhrgas5VLH69YDC6nBJDau20hsaz8ipoTcUPxMS2AxQoGXqv5bUMo8sY8ulz783TWCck7iibWpBdYn_FvqXpBoCu9C23MAD-v6CDe7CJ5ch2KczWBZhnI-jc_z53Lz-Dt62h2PQ1OAAq7-He5QR7Jx-lg87XpaWpCL2A0-uAvfASBUoZ7ui2IvPHuLmw5myZ5acmFZsxpgg9QLrRKlBlUvpNRz0-rG6EZY2uI1UoVRnsaYFLUkRKe3YOZYSCsvLWxGDi6XBEUfdLH9bl3WSV1u0eDdAEvt0qViIcp8Njt0_iT1iyQiHR6ju-Xg**8def9b476d52a739f02c24f8ed8783b8585d01177dcd04e624d2ec17b0a17c61*nVfuzR87kXvFQPaesZZZfY1aPWiT87yLITvEMUJOj5s; Secure; HttpOnly; Path=/_dashboards
< Set-Cookie: security_authentication_saml2=; Max-Age=0; Expires=Thu, 01 Jan 1970 00:00:00 GMT; Secure; HttpOnly; Path=/_dashboards
< Set-Cookie: security_authentication_saml3=; Max-Age=0; Expires=Thu, 01 Jan 1970 00:00:00 GMT; Secure; HttpOnly; Path=/_dashboards
< Set-Cookie: security_authentication=Fe26.2**59193956774e2ff6b56087734d72a99abd64c27811b759e7c10e85f995ae4792*CFtdPl74zukFptAGTWbAXw*bHObA_QMNUiaQI8V471pQhS_Z2GRkmQuy4-G6e5ycbyBSGZ_8W6ThTG7f3SlTgtQLD5AmqSa6mGKeZ5j6JJZoY0zTX2T5Zimkr8JWEljMBIlFOK_G-SSNmEtZPVuKHUfbKOVH-4_6WlW6DVVQO7rGYe9dnm615ssNvTuzFQNJ8BTZ5HVXutfXhMrOzipcs1nh4A2K6PaZnbT-RNpS823WT_JVmdfNcrHXj2xWfjYbHQLC2gv5FTcOHAMd0yPCDS0**9e47d862c46786b7011a75ddbde639601e554e9a30ca09366ede38184d642392*vsovYjoNWeD0yiX3QUmj3ryV1V0IJTuejNc3rBMOLP8; Secure; HttpOnly; Path=/_dashboards
< Strict-Transport-Security: max-age=31536000
< X-Amzn-Requestid: a56e2412-d6f8-4119-84ac-9eead1de5731
<
{"successCount":6,"success":true,"successResults":[{"type":"index-pattern","id":"610267e0-7f21-11ee-80e6-370892a64263","meta":{"title":"documents-metadata-v2","icon":"indexPatternApp"},"overwrite":true},{"type":"visualization","id":"da95cc90-7f22-11ee-9e31-e14f5ec8461b","meta":{"title":"count by cat_doc","icon":"visualizeApp"},"overwrite":true},{"type":"index-pattern","id":"d3d7af60-4c81-11e8-b3d7-01146121b73d","meta":{"title":"opensearch_dashboards_sample_data_flights","icon":"indexPatternApp"},"overwrite":true},{"type":"search","id":"571aaf70-4c88-11e8-b3d7-01146121b73d","meta":{"title":"[Flights] Flight Log","icon":"discoverApp"},"overwrite":true},{"type":"visualization","id":"94003870-dbb5-11ee-81ba-b7ae96977aa8","meta":{"title":"test_object_dashboard","icon":"visualizeApp"},"overwrite":true},{"type":"dashboard","id":"2d7aceb0-7f23-11ee-9e31-e14f5ec8461b","meta":{"title":"test_dashboard","icon":"dashboardApp"},"overwrite":true}]}

* Connection #0 to host ${domain}left intact
```^

As we can see, cURL putting itself the Content Type to multipart/form-data and setting the boundary to my data.
commented
commented

@llasherme try out this branch, his are the relevant steps:

git checkout feature/194
make venv
source venv/bin/activate
./run.sh -v -X POST -F name=@README.md https://httpbin.org/post

once that works try with your ES cluster

commented

@llasherme interesting, try latest please (de473d8)

commented