0

I have python code that currently works fine with a GDB whose size is about ~ 82.7MB. However when I try to use the same code against a GDB whose size ~ 612.8MB the code seems to hang eventually leading to the script terminating.

Specifically the code where the hang takes place is (first line):

buildingFeatures = [pt for pt in fiona.open(sourceDatabaseFile, 'r', layer=layer1)]

waterFeatures = [pol for pol in fiona.open(sourceDatabaseFile, 'r', layer=layer2)]

    for building in buildingFeatures:
        buildingShape = shape(building['geometry'])

        #if the geometry intersects the spatial index
        for id in idx.intersection(buildingShape.bounds):
            water = waterFeatures[id]

            #Check to see if current hydrographySrf feature intersection is a perennial water
            if (('ZI024_HYP' in water['properties']) and (water['properties']['ZI024_HYP'] == 1)):
                waterShapeGeometry = shape(water['geometry'])

                #Check to see if perennial water actually contains building point
                if waterShapeGeometry.contains(buildingShape):
                    discoveredBuildingsContainedInWaters.append(building['id'])

Is there a better way to open a large GDB using Fiona or is there some sort of size limit for GDBs used with Fiona?

  • 1
    It's not so much "opening" a file geodatabase so much as reading the entire contents into memory that's causing your difficulty. That makes this more of a pure Python coding issue than a GIS one. – Vince Nov 11 '16 at 1:02
  • @Vince Is there a better algorithm you'd recommend to read a layer chunks at a time to approach this problem? – George Nov 14 '16 at 16:33
  • It's hard to believe the documentation wouldn't address this. – Vince Nov 14 '16 at 23:58
1

So I rewrote some of my code to exclude properties I was not interested in storing to minimize my memory footprint. In addition I closed file streams just in case.

for bldg in buildingLayerList:
    with fiona.open(sourceDatabaseFile, 'r', layer=bldg) as sourceFile1:
        for building in sourceFile1:
            buildingFeatures.append((building['id'], building['geometry']))
        sourceFile1.close()

    #Use current list of buildings and compare against all hydrography layers
    for hydro in hydrographyLayerList:
        with fiona.open(sourceDatabaseFile, 'r', layer=hydro) as sourceFile2:
            for water in sourceFile2:
                waterFeatures.append((water['id'], water['geometry'], water['properties']))
            sourceFile2.close()

        #from rtree import index
        idx = index.Index() #Create an R-Tree index and store the features in it (bounding box)
        for pos, poly in enumerate(waterFeatures):
            idx.insert(pos, shape(poly[1]).bounds)

        for building in buildingFeatures:
            buildingShape = shape(building[1])

            #if the geometry intersects the spatial index
            for id in idx.intersection(buildingShape.bounds):
                water = waterFeatures[id]

                #Check to see if current hydrography feature intersection is a perennial water
                if (('ZI024_HYP' in water[2]) and (water[2]['ZI024_HYP'] == 1)):
                    waterShapeGeometry = shape(water[1])
                    numBoundedBldgHydroFeaturesEvaluated += 1

                    #Check to see if perennial water actually contains building feature
                    if waterShapeGeometry.contains(buildingShape):
                        discoveredBuildingsContainedInWaters.append(building[0])

            if len(discoveredBuildingsContainedInWaters) > 0:
                totalBldgIDs.update({bldg + " vs " + hydro: sorted(discoveredBuildingsContainedInWaters)})

        discoveredBuildingsContainedInWaters[:] = [] # Just in case
        waterFeatures[:] = [] # Just in case
    buildingFeatures[:] = [] # Just in case

The end result was I got much further parsing through the GDB but the code eventually terminated for the same reasons Vince stated previously before.

I got a hold of the administrator of the server I was running my script on and found out that the environment I was running in had 16GB of RAM shared between all concurrently logged in users (typically at least 6 users). It also turns out that the environment was not set up with a swap file therefore it was purely RAM. The administrator added an 8GB swap file and this fixed my problems.

For the sake of characterizing my problem, the GDB I was using had the following feature count of building vs. hydrography layers:

StructurePnt - 1,533,842 features

StructureCrv - 476 features

HydrographySrf - 7,729 features

HydrographyCrv - 60,090 features

Worst-Case scenario was 92,168,565,780 permutations (StructurePnt vs HydrographyCrv)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.