Only read specific attribute columns of a shapefile with Geopandas / Fiona

Question

Having geopandas installed in my Python environment, I can read a shapefile into a geodataframe with

In:
import geopandas as gpd
myShapefile = gpd.read_file(path_to_my_shapefile)
print(myShapefile)

Out:
myShapefile as a geodataframe

Unfortunately, I have some shapefiles which contain lots of attribute columns which I don't need in the end, slowing down the reading process a lot. Is there any possibility to limit the reading of the shapefile to specific attribute columns?

In regular pandas, I could use the usecols argument to the read_csv and read_table functions to limit the reading to the specified columns, e.g.

import pandas as pd
pd.read_csv(path_to_my_csv_file, usecols=['onlyThisColumn', 'andThatColumnAsWell', 'butNoOther'])

However, using usecols in geopandas' read_file method gives an error, probably because geopandas uses Fiona to read shapefiles which does not accept the argument.

  File "C:\Python34-64bit\lib\site-packages\geopandas\io\file.py", line 13, in read_file
    with fiona.open(filename, **kwargs) as f:
TypeError: open() got an unexpected keyword argument 'usecols'

Is there any other argument or way to achieve this with geopandas/Fiona?

Answer 1

Update for this topic:

geopandas & fiona now support reading specifiec columns from files using the ignore_fields or include_fields parameters in gpd.read_file().

https://geopandas.org/en/stable/docs/user_guide/io.html

Answer 2

Building on gene's answer, you can use GeoDataFrame.from_features

The following should do the trick:

import fiona

def records(filename, usecols, **kwargs):
    with fiona.open(filename, **kwargs) as source:
        for feature in source:
            f = {k: feature[k] for k in ['id', 'geometry']}
            f['properties'] = {k: feature['properties'][k] for k in usecols}
            yield f

And then

gpd.GeoDataFrame.from_features(records(filename), ['prop1', 'prop2'])

I'd be curious to know if this speeds up your code, or if the properties need to be ignored in Fiona rather than the GeoDataFrame construction.

Answer 3

You can "simulate" usecols with a generator:

1) Classical method

import fiona
layer = fiona.open('test.shp')
layer.schema
{'geometry': 'Point', 'properties': OrderedDict([(u'dip', 'int:2'), (u'dipdir', 'int:3'), (u'Type', 'str:10')])}
for feature in layer:
   print feature
{'geometry': {'type': 'Point', 'coordinates': (272070.600041, 155389.38792000001)}, 'type': 'Feature', 'id': '0', 'properties': OrderedDict([(u'dip', 30), (u'dipdir, 130), (u'Type', u'incl')])}
{'geometry': {'type': 'Point', 'coordinates': (271066.03214800003, 154475.63137700001)}, 'type': 'Feature', 'id': '1', 'properties': OrderedDict([(u'dip', 55), (u'dipdir', 145), (u'Type', u'incl')])}
{'geometry': {'type': 'Point', 'coordinates': (273481.498868, 153923.49298800001)}, 'type': 'Feature', 'id': '2', 'properties': OrderedDict([(u'dip', 40), (u'dipdir', 155), (u'Type', u'incl')])}

2) With a generator (with only geometry and one column here, but it is easy to adapt)

def records(filename, col):
   # the generator
   reader = fiona.open(filename)
   for feat in reader:
          new  = {}
          new['geometry'] = feat['geometry']
          new['properties'] = {}
          new['properties'][col] = feat['properties'][col]
          yield new

 for feat in records("test.shp","dip"):
      print feat
  {'geometry': {'type': 'Point', 'coordinates': (272070.600041, 155389.38792000001)}, 'properties': {'dip': 30}}
  {'geometry': {'type': 'Point', 'coordinates': (271066.03214800003, 154475.63137700001)}, 'properties': {'dip': 55}}
  {'geometry': {'type': 'Point', 'coordinates': (273481.498868, 153923.49298800001)}, 'properties': {'dip': 40}}

Answer 4

Well, not a direct solution to the problem, but in some cases the following might be helpful.

It is possible to use the ogr2ogr tool to do the attribute filtering in advance and then read the filtered shapefiles with geopandas / Fiona. My subjective impression is that this is faster than reading the unfiltered shapefiles directly (however, I thought both ogr2ogr and Fiona use the same shapefile reading engine under the hood?!).

To do so (with a properly installed GDAL/OGR on your system):

def filterShapefilesWithOgr2Ogr(inputFolder, outputFolder, attributesToSave):

    # inputFolder: the folder where the original shapefiles are located
    # outputFolder: the folder where the filtered shapefiles are to be stored
    # attributesToSave: string of attributes that shall be copied into the output, e.g. 'ID, location', length'

    import subprocess

    # traverse through the input folder
    os.chdir(inputFolder)
    for filename in glob.glob('*.shp'):

        # filter each shapefile from the input folder and save it at output folder
        print('filtering ' + filename)
        subprocess.call(["ogr2ogr", "-f", "ESRI Shapefile", "-select", attributesToSave, outputFolder + '/' + filename, filename])

    return

If one is only interested into the attributes and not necessarily into the geometry features of the shapefiles, reading only the dbf table of the shapefile into a pandas dataFrame can also speed things up a lot (in my case, 10s vs. 10min).

To do so, I use dbfread (simply install with pip install dbfread).

def readDBFToDataFrame(inputFolder):

    import glob
    import dbfread import DBF

    # traverse through the input folder
    os.chdir(inputFolder)
    for filename in glob.glob('*.dbf'):

        print('reading dbf file ' + filename)

        # load the dbf file
        table = DBF(filename, load=False)

        # iterate through dbf file and store each record in a list
        listOfRecords = []
        for record in table:
            listOfRecords.append(record)

        # create a pandas dataframe from the list of records
        df = pd.DataFrame(listOfRecords)

    return df