Exception on string with £ sign

[Adam Kirk]

    ** (FunctionClauseError) no function clause matching in Poison.Encoder.BitString.chunk_size/3
    
    The following arguments were given to Poison.Encoder.BitString.chunk_size/3:
    
        # 1
        <<163, 50, 46, 53, 48, 47, 109, 111, 110, 116, 104, 41>>
    
        # 2
        nil
    
        # 3
        8
    
    Attempted function clauses (showing 7 out of 7):
    
        defp chunk_size(<<(<<char::integer()>>::binary()), _::binary()>>, _mode, acc) when -char <= 31- or -char === 92 or char === 34-
        defp chunk_size(<<(<<char::integer()>>::binary()), rest::binary()>>, mode, acc) when -char < 128-
        defp chunk_size(-<<(<<_::utf8()>>::binary()), _::binary()>>-, -:unicode-, acc)
        defp chunk_size(-<<(<<char::utf8()>>::binary()), _::binary()>>-, -:javascript-, acc) when -char === 8233 or char === 8232-
        defp chunk_size(-<<(<<codepoint::utf8()>>::binary()), rest::binary()>>-, mode, acc)
        defp chunk_size(-<<char::integer()>>-, _, _)
        defp chunk_size(-""-, _, acc)
    
        (poison) lib/poison/encoder.ex:139: Poison.Encoder.BitString.chunk_size/3
        (poison) lib/poison/encoder.ex:134: Poison.Encoder.BitString.escape/2
        (poison) lib/poison/encoder.ex:88: Poison.Encoder.BitString.encode/2
        (poison) lib/poison/encoder.ex:227: anonymous fn/4 in Poison.Encoder.Map.encode/3
        (poison) lib/poison/encoder.ex:228: Poison.Encoder.Map."-encode/3-lists^foldl/2-0-"/3
        (poison) lib/poison/encoder.ex:228: Poison.Encoder.Map.encode/3
        (poison) lib/poison/encoder.ex:227: anonymous fn/4 in Poison.Encoder.Map.encode/3
        (poison) lib/poison/encoder.ex:228: Poison.Encoder.Map."-encode/3-lists^foldl/2-0-"/3
        (poison) lib/poison/encoder.ex:228: Poison.Encoder.Map.encode/3
        (poison) lib/poison/encoder.ex:259: anonymous fn/3 in Poison.Encoder.List.encode/3
        (poison) lib/poison/encoder.ex:260: Poison.Encoder.List."-encode/3-lists^foldr/2-1-"/3
        (poison) lib/poison/encoder.ex:260: Poison.Encoder.List.encode/3
        (poison) lib/poison/encoder.ex:227: anonymous fn/4 in Poison.Encoder.Map.encode/3
        (poison) lib/poison/encoder.ex:228: Poison.Encoder.Map."-encode/3-lists^foldl/2-0-"/3
        (poison) lib/poison/encoder.ex:228: Poison.Encoder.Map.encode/3
        (poison) lib/poison.ex:41: Poison.encode!/2
        (phoenix) lib/phoenix/controller.ex:740: Phoenix.Controller.do_render/4
        (ja_resource) lib/ja_resource/plug.ex:61: JaResource.Plug.call/2
        (zipbooks) lib/zipbooks_web/controllers/item_controller.ex:1: ZB.ItemController.phoenix_controller_pipeline/2
        (zipbooks) lib/zipbooks_web/endpoint.ex:1: ZB.Endpoint.instrument/4

I did this to figure out what <<163, 50, 46, 53, 48, 47, 109, 111, 110, 116, 104, 41>> was:

iex(3)> String.graphemes(<<163, 50, 46, 53, 48, 47, 109, 111, 110, 116, 104, 41>>)
[<<163>>, "2", ".", "5", "0", "/", "m", "o", "n", "t", "h", ")"]

In the database its this string:

Yearly (£2.50/month)

Any ideas why that would be happening?

[Adam Kirk]

Ok so it wasn't the pound sign. Seems like it might've been some invisible character that got in there. Would it be valuable to have this library ignore bytes that can't be converted? Or maybe an option? I know Ruby has something like that, where you can just have it leave out characters that can't be converted to unicode.

[Stevie Graham]

This might be an encoding issue with your data. 163 is the unicode codepoint for the £ sign, but <<163>> isn't valid UTF-8, it's encoded over two bytes, i.e. <<194, 163>>.

[Adam Kirk]

Right it definitely is, so thats my question, is the contract for this library that you can only pass it valid utf8 strings to encode?

[Stevie Graham]

Elixir strings are UTF-8 encoded binaries. Binaries with different encodings are not strings. Poison will encode character lists, but it encodes them to JSON arrays.

[Adam Kirk]

Ok thanks