I have a shapefile with a large number of geographic features (marine protected areas). I'm trying to read through all of the areas in the shapefile, select those that fit a certain criterion, and write a new shapefile with only the selected areas (areas with reefs).
Many of the areas have names with non-latin characters (i.e. in Swedish or Croatian). I believe I have properly encoded everything in utf-8.
import shapefile
import fiona
from shapely.geometry import Point, shape
import pandas as pd
#read in habitat code list
hab_data = pd.read_csv(habitat_data.csv)
#read in shapefile
with fiona.open("all_protected_areas.shp") as source:
polygons = [pol for pol in source]
with fiona.open('reef_habitats.shp', 'w', encoding='utf-8', **source.meta) as outfile:
#loop over polygons
for i, poly in enumerate(polygons):
#match the site code with the habitat code from csv
sitecode = poly['properties']['SITECODE']
habitatcodes = hab_data.loc[hab_data.SITECODE==sitecode]['HABITATCODE']
#choose only reefs (1170)
if '1170' in habitatcodes.values:
outfile.write(poly)
The code works fine until I run into a polygon with a record containing non-latin characters. The first one is: ('SITENAME', b'Obala izme\x1au rta \x8ailo i Vodoto\x1a')
. The shapefile fails to write on the final line with:
ValueError: Invalid field type <class 'bytes'>
I'm using Python 3. From my own investigations I've seen that there are some tricky differences in the way that Python 3 and 2.7 handle unicode. There also seems to be some weirdness in writing unicode with fiona.
By using Python 2.7 and various combinations of .encode()
and .decode()
, I've managed to get errors like UnicodeEncodeError
and UnicodeDecodeError
but no solution.
I'm open to using a different package if there is an easier way to do this feature selection.