3

I'm very new to GIS but have been asked to parse through any given ESRI GDB file-folder without using any ESRI product. I've been trying to focus on using Fiona and Shapely as these seem to be mentioned the most as alternatives.

I am trying to determine if one particular feature in one layer of the GDB is WITHIN another feature of another layer of the SAME GDB.

As it stands right now the below sample code takes over 5.5 hours to parse through an 80MB GDB file-folder. I was told by the GDB owner it takes ~ 5 minutes via arcpy.

Is there anything that looks obvious to a GIS veteran that I should / should not be doing that is causing execution to be unnecessarily longer?

I tried using the next() command to try and get the next layer of the GDB as I traversed the GDB, but this would crash my script. Using Fiona's listlayers worked around this problem.

with fiona.drivers():
    #The GDB full path is:  sourceDatabaseFile
    numBLDGinPerennialWaters = 0

    for i, featureName in enumerate(fiona.listlayers(sourceDatabaseFile)):
        for j, bldgName in enumerate(fiona.listlayers(sourceDatabaseFile)):
            with fiona.open(sourceDatabaseFile, 'r', layer=i) as gdb:
                with fiona.open(sourceDatabaseFile, 'r', layer=j) as gdbStructures:

                    for feature in gdb:
                        if ((featureName == "HydrographySrf") and ('ZI024_HYP' in feature['properties']) and (feature['properties']['ZI024_HYP'] > 0)):

                            for buildings in gdbStructures:
                                if (bldgName == "StructureSrf"):

                                    if (('WID' in buildings['properties'])  and (buildings['properties']['WID'] > 0)):
                                        if shape(buildings['geometry']).within(shape(featureName['geometry'])):
                                            numBLDGinPerennialWaters = numBLDGinPerennialWaters + 1

    print "Finished parsing GDB file."
5

I got an orders of magnitude speed-up simply by moving two if statements just underneath their corresponding fiona.listlayers calls.

if (featureName == "HydrographySrf")

if (bldgName == "StructureSrf")

I just went from taking over 5.5 hours of parsing down to less than 2.5 minutes.

Sorry for getting things spun up on what is otherwise an easy fix.

1
  • Something else I realized is that I can provide the layer names directly in the fiona.open() statements thereby rendering the fiona.listlayers() calls pointless.
    – George
    May 23 '16 at 18:32
1

It could be that your GDB files (which are self-contained databases) have indexes designed to make these kinds of operations very fast for ArcGIS.

If OGR (which is what Fiona uses to read GDB files) supports these indexes, you should be able to get a significant speed up if not the same performance as ArcGIS.

The section at http://toblerity.org/fiona/manual.html#slicing-and-masking-iterators shows how to use a bounding box when querying a dataset. I suggest you use that within this kind of approach:

  1. Read the features in your gdb file into a list like list(gdb)

  2. For each feat in this list do list(gdbStructures.items(bbox=fiona.bounds(feat))) to get a list of the features in the second dataset that have bounding boxes intersecting the bounding box of feat.

  3. For each element of that result list, make a a.within(b) test as you were doing.

Shapely's within() can be expensive if your shapes have a lot of detail. The approach above will use the spatial indexes in your GDB files to make those calculations only when needed.

1
  • Many thanks for your response. I figured out what I was doing wrong. I will post what I did. So easy I can kick myself. Having said that, I will take what you're saying into consideration because eventually I will probably notice how I find one thing within another will incur a heavy cost and will head your steps. With respect to the GDB from what I can see, my GDB files have a lot of detail (from the perspective of a noob). Thanks again for trying to unmuddy the GIS waters. Your response is greatly appreciated.
    – George
    May 23 '16 at 16:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.