[FEATURE] enhance awscurl support create/send multipart POST REST request (from file)
opened this issue · comments
Versions
- OS :
- cURL => via Windows 10
- awscurl => via gitlab runner (Docker)
- OpenSearch 2.11
What's my problem ?
First of all, this closed issue : opensearch-project/opensearch-cli#76
Seems to be close to my problem, but I'm not using opensearch-cli curl. Only awscurl
I try to migrate dashboards objects from one opensearch cluster to another one.
To do that I'm using api provided by opensearch : /_dashboards/api/saved_objects/_import?overwrite=true
But I get errors when I'm trying to post my objects.
How that's done ?
I'm exporting the objects via the opensearch UI (From my first cluster). Dashboards Management > Saved objects > Export (when my dashboard is selected). And I get my objects in a .ndjson file.
After that i'm posting my .ndjson file with awscurl :
Without Content Type header
awscurl --service es --region ${region} -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \
-H 'osd-xsrf: true' \
-d '${dashboard_content}'
{
"statusCode":415,
"error":"Unsupported Media Type",
"message":"Unsupported Media Type"
}
Without Boundary
awscurl --service es --region ${region} -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \
-H 'Content-Type: multipart/form-data' \
-H 'osd-xsrf: true' \
-d '${dashboard_content}'
{
"statusCode":400,
"error":"Bad Request",
"message":"Invalid content-type header: multipart missing boundary"
}
With a random boundary
awscurl --service es --region ${region} -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \
-H 'Content-Type: multipart/form-data; boundary=---------BOUNDARY;' \
-H 'osd-xsrf: true' \
-d '${dashboard_content}'
By only using curl, and don't put Content Type header, it works fine.
curl -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \
-H "osd-xsrf: true"
-F "file=@data/test_dashboard.ndjson"
{
"successCount":6,
"success":true,
"successResults":[
{
"type":"index-pattern",
"id":"610267e0-7f21-11ee-80e6-370892a64263",
"meta":{
"title":"documents-metadata-v2",
"icon":"indexPatternApp"
},
"overwrite":true
},
{
"type":"visualization",
"id":"da95cc90-7f22-11ee-9e31-e14f5ec8461b",
"meta":{
"title":"count by cat_doc",
"icon":"visualizeApp"
},
"overwrite":true
},
{
"type":"index-pattern",
"id":"d3d7af60-4c81-11e8-b3d7-01146121b73d",
"meta":{
"title":"opensearch_dashboards_sample_data_flights",
"icon":"indexPatternApp"
},
"overwrite":true
},
{
"type":"search",
"id":"571aaf70-4c88-11e8-b3d7-01146121b73d",
"meta":{
"title":"[Flights] Flight Log",
"icon":"discoverApp"
},
"overwrite":true
},
{
"type":"visualization",
"id":"94003870-dbb5-11ee-81ba-b7ae96977aa8",
"meta":{
"title":"test_object_dashboard",
"icon":"visualizeApp"
},
"overwrite":true
},
{
"type":"dashboard",
"id":"2d7aceb0-7f23-11ee-9e31-e14f5ec8461b",
"meta":{
"title":"test_dashboard",
"icon":"dashboardApp"
},
"overwrite":true
}
]
}
Yes, cURL doesn't need to explicit the Content Type.
Here's the cURL with verbose option :
curl -v -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \
-H "osd-xsrf: true" \
-F "file=@data/test_dashboard.ndjson"
And the result :
Note: Unnecessary use of -X or --request, POST is already inferred.
* Uses proxy env variable no_proxy == '****'
* Trying [::1]:8080...
* Trying ${domain_ip}:8080...
* Connected to localhost (127.0.0.1) port 8080
> POST /_dashboards/api/saved_objects/_import?overwrite=true HTTP/1.1
> Host: ${domain_url}
> User-Agent: curl/8.4.0
> Accept: */*
> osd-xsrf: true
> Content-Length: 40384
> Content-Type: multipart/form-data; boundary=------------------------wOWPtrGc1TrXmDPpX5GfNt
>
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Cache-Control: private, no-cache, no-store, must-revalidate
< Content-Length: 947
< Content-Type: application/json; charset=utf-8
< Date: Tue, 12 Mar 2024 09:44:45 GMT
< Osd-Name: osd
< Set-Cookie: security_authentication_saml1=Fe26.2**838c01d554bcb2b7d4e5ead20dafc498fcdd74e1faf6b74c5d1348a2326b713c*D_5paBXbChAFcjTpZ00nLg*YUmGokrp_eV_EEv_2S0NZecOyi1Lwhrgas5VLH69YDC6nBJDau20hsaz8ipoTcUPxMS2AxQoGXqv5bUMo8sY8ulz783TWCck7iibWpBdYn_FvqXpBoCu9C23MAD-v6CDe7CJ5ch2KczWBZhnI-jc_z53Lz-Dt62h2PQ1OAAq7-He5QR7Jx-lg87XpaWpCL2A0-uAvfASBUoZ7ui2IvPHuLmw5myZ5acmFZsxpgg9QLrRKlBlUvpNRz0-rG6EZY2uI1UoVRnsaYFLUkRKe3YOZYSCsvLWxGDi6XBEUfdLH9bl3WSV1u0eDdAEvt0qViIcp8Njt0_iT1iyQiHR6ju-Xg**8def9b476d52a739f02c24f8ed8783b8585d01177dcd04e624d2ec17b0a17c61*nVfuzR87kXvFQPaesZZZfY1aPWiT87yLITvEMUJOj5s; Secure; HttpOnly; Path=/_dashboards
< Set-Cookie: security_authentication_saml2=; Max-Age=0; Expires=Thu, 01 Jan 1970 00:00:00 GMT; Secure; HttpOnly; Path=/_dashboards
< Set-Cookie: security_authentication_saml3=; Max-Age=0; Expires=Thu, 01 Jan 1970 00:00:00 GMT; Secure; HttpOnly; Path=/_dashboards
< Set-Cookie: security_authentication=Fe26.2**59193956774e2ff6b56087734d72a99abd64c27811b759e7c10e85f995ae4792*CFtdPl74zukFptAGTWbAXw*bHObA_QMNUiaQI8V471pQhS_Z2GRkmQuy4-G6e5ycbyBSGZ_8W6ThTG7f3SlTgtQLD5AmqSa6mGKeZ5j6JJZoY0zTX2T5Zimkr8JWEljMBIlFOK_G-SSNmEtZPVuKHUfbKOVH-4_6WlW6DVVQO7rGYe9dnm615ssNvTuzFQNJ8BTZ5HVXutfXhMrOzipcs1nh4A2K6PaZnbT-RNpS823WT_JVmdfNcrHXj2xWfjYbHQLC2gv5FTcOHAMd0yPCDS0**9e47d862c46786b7011a75ddbde639601e554e9a30ca09366ede38184d642392*vsovYjoNWeD0yiX3QUmj3ryV1V0IJTuejNc3rBMOLP8; Secure; HttpOnly; Path=/_dashboards
< Strict-Transport-Security: max-age=31536000
< X-Amzn-Requestid: a56e2412-d6f8-4119-84ac-9eead1de5731
<
{"successCount":6,"success":true,"successResults":[{"type":"index-pattern","id":"610267e0-7f21-11ee-80e6-370892a64263","meta":{"title":"documents-metadata-v2","icon":"indexPatternApp"},"overwrite":true},{"type":"visualization","id":"da95cc90-7f22-11ee-9e31-e14f5ec8461b","meta":{"title":"count by cat_doc","icon":"visualizeApp"},"overwrite":true},{"type":"index-pattern","id":"d3d7af60-4c81-11e8-b3d7-01146121b73d","meta":{"title":"opensearch_dashboards_sample_data_flights","icon":"indexPatternApp"},"overwrite":true},{"type":"search","id":"571aaf70-4c88-11e8-b3d7-01146121b73d","meta":{"title":"[Flights] Flight Log","icon":"discoverApp"},"overwrite":true},{"type":"visualization","id":"94003870-dbb5-11ee-81ba-b7ae96977aa8","meta":{"title":"test_object_dashboard","icon":"visualizeApp"},"overwrite":true},{"type":"dashboard","id":"2d7aceb0-7f23-11ee-9e31-e14f5ec8461b","meta":{"title":"test_dashboard","icon":"dashboardApp"},"overwrite":true}]}
* Connection #0 to host ${domain}left intact
```^
As we can see, cURL putting itself the Content Type to multipart/form-data and setting the boundary to my data.
@llasherme try out this branch, his are the relevant steps:
git checkout feature/194
make venv
source venv/bin/activate
./run.sh -v -X POST -F name=@README.md https://httpbin.org/post
once that works try with your ES cluster
@llasherme interesting, try latest please (de473d8)