I have a shapefile with a large number of geographic features (marine protected areas). I'm trying to read through all of the areas in the shapefile, select those that fit a certain criterion, and write a new shapefile with only the selected areas (areas with reefs).
Many of the areas have names with non-latin characters (i.e. in Swedish or Croatian). I believe I have properly encoded everything in utf-8.
import shapefile import fiona from shapely.geometry import Point, shape import pandas as pd #read in habitat code list hab_data = pd.read_csv(habitat_data.csv) #read in shapefile with fiona.open("all_protected_areas.shp") as source: polygons = [pol for pol in source] with fiona.open('reef_habitats.shp', 'w', encoding='utf-8', **source.meta) as outfile: #loop over polygons for i, poly in enumerate(polygons): #match the site code with the habitat code from csv sitecode = poly['properties']['SITECODE'] habitatcodes = hab_data.loc[hab_data.SITECODE==sitecode]['HABITATCODE'] #choose only reefs (1170) if '1170' in habitatcodes.values: outfile.write(poly)
The code works fine until I run into a polygon with a record containing non-latin characters. The first one is:
('SITENAME', b'Obala izme\x1au rta \x8ailo i Vodoto\x1a'). The shapefile fails to write on the final line with:
ValueError: Invalid field type <class 'bytes'>
I'm using Python 3. From my own investigations I've seen that there are some tricky differences in the way that Python 3 and 2.7 handle unicode. There also seems to be some weirdness in writing unicode with fiona.
By using Python 2.7 and various combinations of
.decode(), I've managed to get errors like
UnicodeDecodeError but no solution.
I'm open to using a different package if there is an easier way to do this feature selection.