Encoding issues with raw data

[Tobias Neitzel]

AS @mschader already mentioned in pull request #4 the handling of raw data (not UTF-8 encoded) is flawed in the current cstc implementation. If raw data is send to cstc, it gets modified automatically by Java due to some superfluous string transformations.

Pull #4 should fix this for the initial part of the cstc workflow, but also all plugins should be checked for this behavior. Furthermore, it should be verified that changes on the encoding (like pull request #4) do not break other functionality.

[Markus]

The result of a transformed input (after baking the receipt) has in some cases the same issue. There is an if block which leads to this part:

cstc/src/de/usd/cstchef/view/RecipePanel.java

Lines 402 to 411 in 23eb279

 String headers = new String(result, 0, info.getBodyOffset()); 
 Pattern p = Pattern.compile("Content-Length:\\s*(\\d*)"); 
 Matcher m = p.matcher(headers); 
 if (m.find()) { 
     headers = m.replaceFirst("Content-Length: " + String.valueOf(contentLen)); 
 } 
 byte[] request = new byte[headers.length() + contentLen]; 
 System.arraycopy(headers.getBytes(), 0, request, 0, headers.length()); 
 System.arraycopy(result, info.getBodyOffset(), request, headers.length(), contentLen); 
 return request; 

So the result will be casted to a new string and upon copying it to the array, it calls .getBytes(), which again will lead to a malformed result if the input only consisted of raw data.

[Tobias Neitzel]

List of affected operations that need to be modified.

Data Format:

• Url encode
• Url decode

Extractors:

• Regex (Wont Fix)
• HTTP URI
• HTTP POST Parameter
• HTTP GET Parameter
• HTTP Method
• HTTP Header

String:

• Replace
• Static String
• Substring

[Tobias Neitzel]

As described in pull-request #11, this issue is basically implemented and fixed. We will close this for now and open smaller issues for places, where cstc is still not working correctly with raw bytes.