Unicode problems while writing selected features to a new shapefile with Fiona

Question

I have a shapefile with a large number of geographic features (marine protected areas). I'm trying to read through all of the areas in the shapefile, select those that fit a certain criterion, and write a new shapefile with only the selected areas (areas with reefs).

Many of the areas have names with non-latin characters (i.e. in Swedish or Croatian). I believe I have properly encoded everything in utf-8.

import shapefile
import fiona
from shapely.geometry import Point, shape
import pandas as pd

#read in habitat code list
hab_data = pd.read_csv(habitat_data.csv)

#read in shapefile
with fiona.open("all_protected_areas.shp") as source:
    polygons = [pol for pol in source]
    with fiona.open('reef_habitats.shp', 'w', encoding='utf-8', **source.meta) as outfile:

        #loop over polygons
        for i, poly in enumerate(polygons):

            #match the site code with the habitat code from csv
            sitecode = poly['properties']['SITECODE']
            habitatcodes = hab_data.loc[hab_data.SITECODE==sitecode]['HABITATCODE']

            #choose only reefs (1170)
            if '1170' in habitatcodes.values:

                outfile.write(poly)

The code works fine until I run into a polygon with a record containing non-latin characters. The first one is: ('SITENAME', b'Obala izme\x1au rta \x8ailo i Vodoto\x1a'). The shapefile fails to write on the final line with:

ValueError: Invalid field type <class 'bytes'>

I'm using Python 3. From my own investigations I've seen that there are some tricky differences in the way that Python 3 and 2.7 handle unicode. There also seems to be some weirdness in writing unicode with fiona.

By using Python 2.7 and various combinations of .encode() and .decode(), I've managed to get errors like UnicodeEncodeError and UnicodeDecodeError but no solution.

I'm open to using a different package if there is an easier way to do this feature selection.

Answer 1

Fiona has a built-in parameter to specify the encoding of your data. Since you’re not explicitly setting the encoding, its probably defaulting to utf-8 (unless you are on windows). If your data is from a decent source it should specify the encoding somewhere in metadata. My best guess would be Latin-1.

with fiona.open(“all_protected_areas.shp”, encoding=”iso-8859-1”) as source: