2

I worked on this problem for a while and I have localized it pretty well now to ArcGIS 10.4 at its core (haven't tested 10.5): A column that contains special Unicode Characters (emojis) is not read correctly from the Shapefile.

It looks like this:

Unicode column with emojis and corresponding unicode_name of character in left column

The Left Column contains the unicode-special-characters (emojis), the right column the corresponding unicoce-names of codepoints (unicode_name(emoji_str)). I am writing this Shapefile with fiona (explicit encoding in UTF-8): with fiona.open('output.shp', mode='w', encoding='utf-8', driver='ESRI Shapefile', schema=schema,crs=from_epsg(epsg_code)) as c:

The output.cpg explicitly states UTF-8.

No other program seems to have this problem with the DBF or Shapefile. QGIS has problems printing the emoji (not suprisingly), but it is still correctly read: QGIS view of table

If I open the output.dbf of the shapefile in LibreOffice, everything looks perfect: LibreOffice View of table

A workaround for me was the following:

  • open DBF in LibreOffice
  • export to output.CSV
  • add output.CSV to ArcGIS:
    • CSV imported to ArcGIS
    • looks good!
  • join Table to original Shapefile:
    • CSV joined to original shapefile
    • joined column (selected) looks still good!
    • I can export the Shapefile now with the joined column, and this I find really strange, it still looks good:
    • JoinedCSVExportedShapefile
    • this tells me that ArcGIS Shapefile can handle such characters, but the original Shapefile seems to have some problems that I cannot locate at the moment
  • in case you wonder if I could export the Shapefile from QGIS:
    • this does not work, the exported Shapefile appears to have the same problems in ArcGIS, despite that it is correctly viewed in QGIS

The text that appears in the problematic columns seems to originate from surrounding columns, some sort of substitution going on. It is not just visual, I can copy the text and if I export the shapefile, it gets hard-coded/written.

I followed this tech article and set my default codepage for ArcGIS to UTF-8 - no changes. I have done all sorts of checks in Python and Fiona, but the Charactersets appear all to be ok and the correct view in LibreOffice and CSV corroborate my assumption that there is something completely going wrong in ArcGIS.

If you want to try yourself, here is the test shapefile from above: output_shapefile.zip (https://file.io/oyz87D) Or you can try writing a shapefile with Fiona and an example Emoji such as this one: 😍

How can I create an ESRI Shapefile from Python that can be loaded to ArcGIS with Unicode-Emojis showing correctly?

This is an example Python program to reproduce the issue:

#!/usr/bin/env python
# coding: utf-8
# emojiShapeUnicodeArcGISTest
import fiona
from fiona.crs import from_epsg
import emoji
from unicodedata import name as unicode_name

def extract_emojis(str):
    return list(c for c in str if c in emoji.UNICODE_EMOJI)

#Example Emojis
emojistring = '😍 test text ❤️👨‍⚕️'
emojilist = extract_emojis(emojistring)

# Define polygon feature geometry
schema = {
    'geometry': 'Polygon',
    'properties': {'emoji': 'str',
                'emoji_text': 'str'},
}
#write Shapefile
with fiona.open('output.shp', mode='w', encoding='UTF-8', driver='ESRI Shapefile', schema=schema,crs=from_epsg(4326)) as c:
    for emoji in emojilist:            
        c.write({'geometry': {'coordinates': [[(-4.663611, 51.158333),
                                (-4.669168, 51.159439),
                                (-4.673334, 51.161385),
                                (-4.674445, 51.165276),
                                (-4.67139, 51.185272),
                                (-4.669445, 51.193054),
                                (-4.665556, 51.195),
                                (-4.65889, 51.195),
                                (-4.656389, 51.192215),
                                (-4.646389, 51.164444),
                                (-4.646945, 51.160828),
                                (-4.651668, 51.159439),
                                (-4.663611, 51.158333)]],
                'type': 'Polygon'},
                'properties': {'emoji': emoji, 
                                'emoji_text': str(unicode_name(emoji))},
        })

The Shapefile-table it produces looks like this in ArcGIS 10.4 (with bug visible in row 1 and 3): Result of Test Code

I also put the code on Pastebin

  • 1
    This doesn't appear to be a question. Esri has a mechanism for bug reporting, and GIS SE isn't involved. – Vince Mar 2 '18 at 20:19
  • 3
    I'm voting to close this question as off-topic because, as stated in the help, "We are not a formal GIS Software Support Site and are likely to close questions and recommend you seek official routes for support if you:... seem to be trying to submit bug reports and enhancement requests here instead of direct to the vendor/development team of the software involved." – Vince Mar 2 '18 at 22:20
  • Thanks for your answers. I was aware of this but still posted it because it could also have something to do with the way Ogre/Fiona write Shapefiles. As is described in my answer, if I import the data through CSV, the Shapefile's table is just fine. I couldn't find much information on how to modify Fiona encoding output, except encoding='utf-8' – Alex Mar 2 '18 at 23:00
  • There doesn't appear to be an actual question here, and even in your comment above you refer to it being your "answer". – Midavalo Mar 3 '18 at 0:39
  • I thought the question is obvious: How can I create a shapefile with Fiona that does work with Unicode in ArcGIS. ArcGIS does support Unicode EMojis in Shapefiles, as shown above, just not in those Shapefiles created from Fiona, apparently. Whether this is a Fiona issue or ArcGIS issue I don't know. Maybe there's even a better workaround than the one I described. Maybe someone knows a better tool than Fiona to write ESRI Shapefiles from Python. All these things would be really helpful, and I think these are real questions.. – Alex Mar 3 '18 at 3:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.