Use of Emoji from Shapefile in TextSymbolizer rendered as rectangles

Question

Use of Emoji from Shapefile in TextSymbolizer rendered as rectangles

Sieboldianus opened this issue 2 years ago · comments

Alexander Dunkel commented 2 years ago

Context: I am trying to implement a specific Tag Maps rendering (https://github.com/Sieboldianus/TagMaps) in Mapnik

My first test works fine, as long as I do not try to plot Emoji characters on the map.

Sample Shapefile

Shapefile Fields:

[ImpTag] - the label for polygons to be placed on the map
[Weights] - used to calculate Label size in the XML style
[emoji] - indicates whether label is emoji (=1) or not (=0), used in filter expression

The shapefile was generated in Python using Fiona/Shapely.

Example output:

style.xml

This is the output when <Filter>[emoji] = 1</Filter> is included in a rule. As you can see, some emoji appear to be rendered fine, but (most of) the emoji are rendered as boxes. I believe these are grapheme cluster emoji, e.g. with Zero Width joiners.

<?xml version="1.0" encoding="utf-8"?>

<Map background-color="white" srs="+proj=utm +zone=33 +datum=WGS84 +units=m +no_defs">

  <Style name="TagMaps Style">
    <Rule>
      <Filter>[emoji] = 1</Filter> 
      <TextSymbolizer placement="point" size="20" face-name="Segoe UI Emoji Regular" fill="black" justify-alignment="center">
        <Format size="6+[Weights]*0.18">[ImpTag]</Format>
      </TextSymbolizer>
    </Rule>
    <Rule>
      <Filter>[emoji] = 0</Filter> 
      <TextSymbolizer placement="point" face-name="Arial Regular" fill="black" justify-alignment="center">
        <Format size="6+[Weights]*0.18">[ImpTag]</Format>
      </TextSymbolizer>
    </Rule>
  </Style>

  <Layer name="tagmaps" srs="+proj=utm +zone=33 +datum=WGS84 +units=m +no_defs">
    <StyleName>TagMaps Style</StyleName>
    <Datasource>
      <Parameter name="encoding">UTF-8</Parameter>
      <Parameter name="type">shape</Parameter>
      <Parameter name="file">../data-test/sample_shapefile.shp</Parameter>
    </Datasource>
  </Layer>

</Map>

For completeness, this is the corresponding python file to process the image:

#!/usr/bin/env python3

import mapnik

mapnik.register_fonts('~/.fonts/')

stylesheet = 'tagmap_style.xml'
image = 'tagmap_style.png'
m = mapnik.Map(2500,1400)
mapnik.load_map(m, stylesheet)
m.zoom_all() 
mapnik.render_to_file(m, image)
print(f"rendered image to {image}")

Versions:

mapnik-config -v: 3.0.23
lsb_release -d: Ubuntu 20.04.4 LTS

Observation:

the issue does not appear to be related to the shapefile encoding
- the shapefile is utf-8
- also corroborated by using ogr2ogr sample_shapefile_utf8.shp sample_shapefile.shp -lco ENCODING=UTF-8
the issue does not appear to be related to the font used
- I switched both, the emoji font (Segoe UI Emoji Regular in the example) and the regular font (Arial in the example)

What I have tried:

different emoji fonts: Segoe UI Emoji Regular, Noto Emoji Regular, Noto Sans Regular, Symbola Regular, Twitter Color Emoji - all the same behaviour. Sometimes, rectangles are rendered as diamond-questionmarks
use of fonts directory from windows (/c/windows/fonts) or from linux (~/.fonts/), same behaviour
different Mapnik verisons: 3.0.23 and 3.1.0, same beaviour

Any help would be much appreciated.

Here is the final map generated in ESRI ArcMap, the one I am trying to replicate in Mapnik - I am trying to get away from ESRI software:

I think this issue might be similar or connected to #3931

Alexander Dunkel · Answer 1 · Fri Jul 15 2022 18:43:36 GMT+0800 (China Standard Time)

Closing this issue. It was a problem with my data, not with Mapnik.

Note to myself: Always create a minimal reproducible example.

Minimal reproducible example

emoji_sample.xml

<?xml version="1.0" encoding="utf-8"?>

<!-- Origin: https://github.com/mapnik/mapnik/issues/3931 -->

<Map background-color="#ffffff">
    <Parameters>
        <Parameter name="sizes">256, 256</Parameter>
        <Parameter name="bbox">-1, -1, 1, 1</Parameter>
    </Parameters>
    <FontSet name="fontset">
        <Font face-name="Segoe UI Emoji Regular" />
		<Font face-name="Noto Sans Regular" />
		<Font face-name="Noto Sans CJK JP Regular" />
		<Font face-name="Noto Sans Adlam Regular" />
		<Font face-name="Noto Sans Adlam Unjoined Regular" />
		<Font face-name="Noto Sans Bamum Regular" />
		<Font face-name="Noto Sans Batak Regular" />
		<Font face-name="Noto Sans Bengali UI Regular" />
		<Font face-name="Noto Sans Buginese Regular" />
		<Font face-name="Noto Sans Buhid Regular" />
		<Font face-name="Noto Sans Canadian Aboriginal Regular" />
		<Font face-name="Noto Sans Chakma Regular" />
		<Font face-name="Noto Sans Cham Regular" />
		<Font face-name="Noto Sans Cherokee Regular" />
		<Font face-name="Noto Sans Coptic Regular" />
		<Font face-name="Noto Sans Devanagari UI Regular" />
		<Font face-name="Noto Sans Ethiopic Regular" />
		<Font face-name="Noto Sans Georgian Regular" />
		<Font face-name="Noto Sans Gujarati UI Regular" />
		<Font face-name="Noto Sans Gurmukhi UI Regular" />
		<Font face-name="Noto Sans Hanunoo Regular" />
		<Font face-name="Noto Sans Hebrew Regular" />
		<Font face-name="Noto Sans Javanese Regular" />
		<Font face-name="Noto Sans Kannada UI Regular" />
		<Font face-name="Noto Sans Kayah Li Regular" />
		<Font face-name="Noto Sans Khmer UI Regular" />
		<Font face-name="Noto Sans Lao UI Regular" />
		<Font face-name="Noto Sans Lepcha Regular" />
		<Font face-name="Noto Sans Limbu Regular" />
		<Font face-name="Noto Sans Lisu Regular" />
		<Font face-name="Noto Sans Malayalam UI Regular" />
		<Font face-name="Noto Sans Mandaic Regular" />
		<Font face-name="Noto Sans Mongolian Regular" />
		<Font face-name="Noto Sans Myanmar UI Regular" />
		<Font face-name="Noto Sans New Tai Lue Regular" />
		<Font face-name="Noto Sans NKo Regular" />
		<Font face-name="Noto Sans Ol Chiki Regular" />
		<Font face-name="Noto Sans Oriya UI Regular" />
		<Font face-name="Noto Sans Osage Regular" />
		<Font face-name="Noto Sans Osmanya Regular" />
		<Font face-name="Noto Sans Samaritan Regular" />
		<Font face-name="Noto Sans Saurashtra Regular" />
		<Font face-name="Noto Sans Shavian Regular" />
		<Font face-name="Noto Sans Sinhala UI Regular" />
		<Font face-name="Noto Sans Sinhala Regular" />
		<Font face-name="Noto Sans Sundanese Regular" />
		<Font face-name="Noto Sans Symbols Regular" />
		<Font face-name="Noto Sans Symbols2 Regular" />
		<Font face-name="Noto Sans Tagalog Regular" />
		<Font face-name="Noto Sans Tagbanwa Regular" />
		<Font face-name="Noto Sans Tai Le Regular" />
		<Font face-name="Noto Sans Tai Tham Regular" />
		<Font face-name="Noto Sans Tai Viet Regular" />
		<Font face-name="Noto Sans Tamil UI Regular" />
		<Font face-name="Noto Sans Telugu UI Regular" />
		<Font face-name="Noto Sans Thaana Regular" />
		<Font face-name="Noto Sans Thai UI Regular" />
		<Font face-name="Noto Sans Tibetan Regular" />
		<Font face-name="Noto Sans Tifinagh Regular" />
		<Font face-name="Noto Sans Vai Regular" />
		<Font face-name="Noto Sans Yi Regular" />
		<Font face-name="Noto Sans Arabic UI Regular" />
		<Font face-name="Noto Emoji Regular" />
		<Font face-name="Noto Naskh Arabic UI Regular" />
		<Font face-name="DejaVu Sans Book" />
		<Font face-name="HanaMinA Regular" />
		<Font face-name="HanaMinB Regular" />
		<Font face-name="Unifont Medium" />
		<Font face-name="Unifont Upper Medium" />
    </FontSet>
    <Style name="example-text">
        <Rule>
            <TextSymbolizer
                placement="point"
                size="30"
                fontset-name="fontset"
                fill="#000000"
                justify-alignment="center"
                >
                [text]
            </TextSymbolizer>
        </Rule>
    </Style>
    <Layer name="example-text">
        <StyleName>example-text</StyleName>
        <Datasource>
            <Parameter name="type">csv</Parameter>
            <Parameter name="inline">
id|wkt|text
1|Point(0 0)|🌸
2|Point(1 0)|🍺
3|Point(0 1)|📚
4|Point(1 1)|📸
5|Point(0 -1)|📸
6|Point(-1 0)|🍸
7|Point(-1 -1)|💥
8|Point(-1 1)|🎸
9|Point(1 -1)|🚙
            </Parameter>
        </Datasource>
    </Layer>
</Map>

#!/usr/bin/env python3

import mapnik

mapnik.register_fonts('../fonts')
# mapnik.register_fonts('/home/alex/.fonts/')
# for face in mapnik.FontEngine.face_names(): print(face)

stylesheet = 'emoji_issue.xml'
image = 'emoji_issue.png'
m = mapnik.Map(256, 256)
mapnik.load_map(m, stylesheet)
bbox = mapnik.Box2d(-3, -3, 3, 3)
m.zoom_to_box(bbox)
im = mapnik.Image(m.width, m.height)
mapnik.render(m, im)
im.save(image)

.. the example above returned emoji from Mapnik just fine:

.. this is possibly of no interest to anyone, but my data was malformed because it went through ESRI ArcMap, which apparently has a bug that exists for a long time that stores Emoji characters in a non-standard way in shapefiles. It was once reported in the Shapely and GDal Repos. Once I stored emoji directly from Shapely/Fiona (gdal) to the shapefile, everything worked.

Here's the output with Segoe UI Symbol Regular as font/face-name