Changing shapefile's field type using fiona?

Question

I need to have a layer time-enabled for Esri application-making. Coming from the source, it has an int type column called year. I need to find a way to automate its conversion to date.

I first tried with Geopandas, but it gives a ValueError if I try and save to Shapefile with a datetime[ms] dtype:

ValueError: Invalid field type <class 'pandas.tslib.Timestamp'>

I know Fiona can handle date, time, and datetime field types though:

>>> fiona.FIELD_TYPES_MAP

{'date': fiona.rfc3339.FionaDateType,
 'datetime': fiona.rfc3339.FionaDateTimeType,
 'float': float,
 'int': int,
 'str': str,
 'time': fiona.rfc3339.FionaTimeType}

When I load it with Fiona I get:

{'geometry': 'Point',
 'properties': OrderedDict([('npri_id', 'int:9'),
              ('facility', 'str:80'),
              ('year', 'int:9'),
              ...])}

How would I go about changing it to date?

Answer 1

You can automate it with geopandas, but there seems to be an issue in automatically converting the pandas datetime objects to the right properties schema. Fortunately, as geopandas is built directly on top of fiona for reading and writing you can specify a schema for writing output, e.g.:

schema = {
    'geometry': 'Point',
    'properties': {
        'npri_id': 'int',
        'facility': 'str',
        'year': 'datetime',
}}

geodataframe.to_file('output.shp', schema=schema)

Note that the number of fields in the schema must match the number of fields in the geodataframe to export (though you can of course subset the fields to export).

Answer 2

Thanks again to om_henners for the answer! I'll go through my own steps through the problem so that it can be used for others with this problem. Maybe eventually find its way in geopandas proper? 😀

import geopandas as gpd
import pandas as pd

gdf = gpd.read_postgis(query, con)  # These are defined elsewhere

# Make sure 'year' is type 'datetime'
gdf['year'] = pd.to_datetime(gdf['year'], format='%Y')

schema = gpd.io.file.infer_schema(gdf)
print(schema)

>>> {'geometry': 'Point',
     'properties': OrderedDict([('npri_id', 'int'),
                   ('facility', 'str'),
                   ('year', 'int'),
                  ...])}

In my example, I want to change year's datatype to datetime.

schema['properties']['year'] = 'datetime'
gdf.to_file('test.shp', schema=schema)

And this indeed becomes a Shapefile that is "time-enabled" in Esri applications.

Answer 3

The existing answers did not work for me (fiona 1.8.13, geopandas 0.8.0). The ESRI shapefile format supports python's date class, but not the datetime class https://github.com/Toblerity/Fiona/issues/572.

Instead of this:

gdf['datetime'] = pd.to_datetime(x)  # datetime

Use this:

gdf['date'] = pd.to_datetime(x).date()  # date
gdf['time'] = pd.to_datetime(x).time()  # time