CFHTTP and encoding
LiQz0r opened this issue · comments
Hey,
I'm trying to post some JSON data (which contains letters like 'æ', 'ø', and 'å') to a REST endpoint using CFHTTP. The code works in Adobe's ColdFusion and the data is passed as expected. However, using OpenBD (tested on both 3.1 and 3.2) the data doesn't seem to be encoded in any way before being posted. I need these characters (among other special characters as well) to be encoded to avoid problems in other parts of the system that are outside of my control.
I'm not interested in a solution involving something like replace(var, "ø", "encoded ø", "ALL") as that is too specific/hardcoded for my taste.
I've made a simple example to demonstrate my problem.
client.cfm
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Client</title>
</head>
<body>
<form action="server.cfm" method="post" accept-charset="UTF-8">
<input type="text" name="test" id="test" title="" />
<button type="submit">Test</button>
</form>
</body>
</html>server.cfm
<cfprocessingdirective pageEncoding="utf-8" />
<cfset myStruct = { "values": { "value1": #form['test']# } } />
<cfset serializedStruct = serializeJSON(myStruct) />
<cfhttp url="[myRestServer]/test" method="post" result="result" charset="utf-8">
<cfhttpparam type="header" name="Content-Type" value="application/json; charset=utf-8" />
<cfhttpparam type="body" value="#serializedStruct#" />
</cfhttp>
<cfdump var=#result# />TestResource.java
import org.json.JSONException;
import org.json.JSONObject;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import javax.inject.Singleton;
import javax.ws.rs.Consumes;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
@Singleton
public class TestResource {
private static final Logger logger = LoggerFactory.getLogger(TestResource.class);
@Path("/test")
@POST
@Consumes(MediaType.APPLICATION_JSON)
public Response test(String body) throws JSONException {
JSONObject jsonObject = new JSONObject(body);
logger.debug("json: {}", jsonObject.toString());
return Response.ok().build();
}
}When I enter "æøå" in the client form above I get the following results:
Adobe ColdFusion: json: {"values":{"value1":"æøå"}}
OpenBD: json: {"values":{"value1":"???"}}
Both examples run on the same Tomcat 8.5.32 installation.
Is this a bug or is this the intended behavior?
I tried it like this,
client.cfm
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Client</title>
</head>
<body>
<form action="server.cfm" method="post" accept-charset="UTF-8">
<input type="text" name="test" />
<button type="submit">Test</button>
</form>
</body>
</html>
server.cfm
mystruct = {};
mystruct.values = {};
mystruct.values.value1 = form.test;
params = [
{ "type" : "header", "name" : "Content-Type", "value" : "application/json" },
{ "type" : "body", "value" : serializeJSON(mystruct) }
];
HTTP(
method:"POST",
url:'http://localhost:8529/_db/testdb/_api/document/users',
charset:"utf-8",
httpparams : params
);
I did a test post http method into Arangodb and it come out correct.
Adobe Coldfusion 11 support page says,
cfprocessingdirective
In general, one should no longer need to use this tag to specify the page encoding as the ColdFusion server should be able to identify it automatically.
Also think this code is not correct,
<cfhttpparam type="header" name="Content-Type" value="application/json; charset=utf-8" />
Anyway good luck and it seems to work fine on the OpenBD end.
Sorry but I missed the <cfscript></cfscript> tags in server.cfm
I tried out your code but sadly it gave me the same result as before, mgrandb.
Another thing that might be worth pointing out is that if I pass in those characters directly (like so <cfset myStruct = { "values": { "value1": "æøå" }} />) in my JSON structure then it also executes correctly in OpenBD. But for it to work I have to drop the <cfprocessingdirective> - which I also did before trying out your code, mgrandb.
This has me stumped. Any other suggestions?
client.cfm
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Client</title>
</head>
<body>
<form action="server.cfm" method="post" accept-charset="UTF-8">
<input type="text" name="test" />
<button type="submit">Test</button>
</form>
</body>
</html>
server.cfm
mystruct = {};
mystruct.values = {};
mystruct.values.value1 = form.test;
WriteDump(Serializejson(mystruct));
Try this to see if OpenBD is serializing the json string properly,
it's working on my end.
Is the obd Administrator Default Character Set at UTF-8?
Output in Chrome from your code:
{"values":{"value1":"æøå"}}
Output in the developer tools network request preview tab in Chrome:
{"values":{"value1":"æøå"}}
I checked bluedragon.xml to be certain that <defaultcharset> is indeed set to UTF-8 (which it is) but I still get json: {"values":{"value1":"???"}} in the REST API. Running the exact same code on my Adobe ColdFusion installation give me json: {"VALUES":{"VALUE1":"æøå"}} in the REST API while the output in Chrome matches what I get from OpenBD.
This is what makes me think <cfhttp> / the http function is at the heart of the problem.
I just tried calling my REST API using Ajax and jQuery instead of relying on <cfhttp> like this:
server.cfm
<cfscript>
mystruct = {};
mystruct.values = {};
mystruct.values.value1 = form.test;
WriteDump(Serializejson(mystruct));
</cfscript><script>
$(document).ready(function() {
$.ajax({
type: 'POST',
url: '[myRestServer]/test',
contentType: 'application/json; charset=utf-8',
dataType: 'json',
data: JSON.stringify(<cfoutput>#serializeJSON(mystruct)#</cfoutput>)
}).done(function(data) {
console.log(data);
alert('great success!');
});
});
</script>The result was {"values":{"value1":"æøå"}} in the REST API - which is what I was looking for!
So it does appear to be something going on with how <cfhttp> handles encoding.
Just curious, does the call to the rest server via cfhttp url have to be enclosed in brackets?
<cfhttp url="[myRestServer]/test" method="post" result="result" charset="utf-8">?
I don't think OpenBD has any problems with cfhttp because I did a post to Arangodb's rest api,
and it worked like it should.
Those brackets in the cfhttp code has me wondering if that might be causing it
Sorry, I should've clarified what I meant by those square brackets. I didn't think the REST API URL itself was that interesting, so I just shortened it and added the actual endpoint ("/test") used in my example to indicate that this was a call to my REST API. My actual URL is "http://localhost:8181/rest/upload/test".
I have identified my mistake. When I downloaded OpenBD 3.2's "Ready2Run" package I actually downloaded OpenBD 3.1's "Ready2Run" package thinking I'd downloaded the correct one.
Everything works as expected with OpenBD 3.2 and the bug fixes mentioned in the release notes found here http://openbd.org/notes/ that addresses <cfhttp> fixes the issue I had.
Sorry to have wasted your time, mgrandb, but I'm grateful for the insight and help you provided along the way.