Having geopandas installed in my Python environment, I can read a shapefile into a geodataframe with

In:
import geopandas as gpd
myShapefile = gpd.read_file(path_to_my_shapefile)
print(myShapefile)

Out:
myShapefile as a geodataframe

Unfortunately, I have some shapefiles which contain lots of attribute columns which I don't need in the end, slowing down the reading process a lot. Is there any possibility to limit the reading of the shapefile to specific attribute columns?

In regular pandas, I could use the usecols argument to the read_csv and read_table functions to limit the reading to the specified columns, e.g.

import pandas as pd
pd.read_csv(path_to_my_csv_file, usecols=['onlyThisColumn', 'andThatColumnAsWell', 'butNoOther'])

However, using usecols in geopandas' read_file method gives an error, probably because geopandas uses Fiona to read shapefiles which does not accept the argument.

  File "C:\Python34-64bit\lib\site-packages\geopandas\io\file.py", line 13, in read_file
    with fiona.open(filename, **kwargs) as f:
TypeError: open() got an unexpected keyword argument 'usecols'

Is there any other argument or way to achieve this with geopandas/Fiona?

Building on gene's answer, you can use GeoDataFrame.from_features

The following should do the trick:

import fiona

def records(filename, usecols, **kwargs):
    with fiona.open(filename, **kwargs) as source:
        for feature in source:
            f = {k: feature[k] for k in ['id', 'geometry']}
            f['properties'] = {k: feature['properties'][k] for k in usecols}
            yield f

And then

gpd.GeoDataFrame.from_features(records(filename), ['prop1', 'prop2'])

I'd be curious to know if this speeds up your code, or if the properties need to be ignored in Fiona rather than the GeoDataFrame construction.

  • Uhm, thanks for your support but I got the impression that this did not speed up anything, reading the files takes more or less as long as before :o I just cannot investigate this "scientifically" (using time measurement) at the moment, maybe later today or tomorrow. Did you have a different impression? – Dirk Jan 8 '15 at 13:14
  • 1
    Fiona doesn't support field picking, so there is no speedup: geopandas must read every field and filter them. Could be a good feature, maybe sketch it out at github.com/Toblerity/Fiona/issues? – sgillies Jan 8 '15 at 16:34

You can "simulate" usecols with a generator:

1) Classical method

import fiona
layer = fiona.open('test.shp')
layer.schema
{'geometry': 'Point', 'properties': OrderedDict([(u'dip', 'int:2'), (u'dipdir', 'int:3'), (u'Type', 'str:10')])}
for feature in layer:
   print feature
{'geometry': {'type': 'Point', 'coordinates': (272070.600041, 155389.38792000001)}, 'type': 'Feature', 'id': '0', 'properties': OrderedDict([(u'dip', 30), (u'dipdir, 130), (u'Type', u'incl')])}
{'geometry': {'type': 'Point', 'coordinates': (271066.03214800003, 154475.63137700001)}, 'type': 'Feature', 'id': '1', 'properties': OrderedDict([(u'dip', 55), (u'dipdir', 145), (u'Type', u'incl')])}
{'geometry': {'type': 'Point', 'coordinates': (273481.498868, 153923.49298800001)}, 'type': 'Feature', 'id': '2', 'properties': OrderedDict([(u'dip', 40), (u'dipdir', 155), (u'Type', u'incl')])}

2) With a generator (with only geometry and one column here, but it is easy to adapt)

def records(filename, col):
   # the generator
   reader = fiona.open(filename)
   for feat in reader:
          new  = {}
          new['geometry'] = feat['geometry']
          new['properties'] = {}
          new['properties'][col] = feat['properties'][col]
          yield new

 for feat in records("test.shp","dip"):
      print feat
  {'geometry': {'type': 'Point', 'coordinates': (272070.600041, 155389.38792000001)}, 'properties': {'dip': 30}}
  {'geometry': {'type': 'Point', 'coordinates': (271066.03214800003, 154475.63137700001)}, 'properties': {'dip': 55}}
  {'geometry': {'type': 'Point', 'coordinates': (273481.498868, 153923.49298800001)}, 'properties': {'dip': 40}}
  • As already stated in my comment to jakew's answer, thanks for your approach, but I did not recognize an improvement in reading speed from this. Do you have a different impression? – Dirk Jan 8 '15 at 13:17
  • 1
    No, it is only an alternative approach to give a solution to the question of jakew. – gene Jan 8 '15 at 18:07

Well, not a direct solution to the problem, but in some cases the following might be helpful.

It is possible to use the ogr2ogr tool to do the attribute filtering in advance and then read the filtered shapefiles with geopandas / Fiona. My subjective impression is that this is faster than reading the unfiltered shapefiles directly (however, I thought both ogr2ogr and Fiona use the same shapefile reading engine under the hood?!).

To do so (with a properly installed GDAL/OGR on your system):

def filterShapefilesWithOgr2Ogr(inputFolder, outputFolder, attributesToSave):

    # inputFolder: the folder where the original shapefiles are located
    # outputFolder: the folder where the filtered shapefiles are to be stored
    # attributesToSave: string of attributes that shall be copied into the output, e.g. 'ID, location', length'

    import subprocess

    # traverse through the input folder
    os.chdir(inputFolder)
    for filename in glob.glob('*.shp'):

        # filter each shapefile from the input folder and save it at output folder
        print('filtering ' + filename)
        subprocess.call(["ogr2ogr", "-f", "ESRI Shapefile", "-select", attributesToSave, outputFolder + '/' + filename, filename])

    return

If one is only interested into the attributes and not necessarily into the geometry features of the shapefiles, reading only the dbf table of the shapefile into a pandas dataFrame can also speed things up a lot (in my case, 10s vs. 10min).

To do so, I use dbfread (simply install with pip install dbfread).

def readDBFToDataFrame(inputFolder):

    import glob
    import dbfread import DBF

    # traverse through the input folder
    os.chdir(inputFolder)
    for filename in glob.glob('*.dbf'):

        print('reading dbf file ' + filename)

        # load the dbf file
        table = DBF(filename, load=False)

        # iterate through dbf file and store each record in a list
        listOfRecords = []
        for record in table:
            listOfRecords.append(record)

        # create a pandas dataframe from the list of records
        df = pd.DataFrame(listOfRecords)

    return df

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.