1

I'm using a nested for-loop to do Shapely analysis between two Python lists created via two pre-reads of the same GDB file. One list contains all the features of a particular layer while the other list contains all the features of another layer.

I'm using an ~ 80MB test GDB to try and figure things out in an attempt to do an all-Python approach.

My code takes about 44 minutes to fully analyze all permutations on a thin-client running Windows 7 connected to a Linux VM via Remote Desktop where my code runs. It's what I'm limited to.

By comparison, a co-worker of mine is using Arcpy's MakeFeatureLayer_management and SelectLayerByLocation_management calls to go through the same GDB in 24 seconds directly on a Windows box and 8 minutes on a thin-client. Basically the thin-client is 20x slower.

Roughly speaking, I'm speculating that my code SHOULD have a run time of about 2.2 minutes running directly on a Linux box based on my co-worker's observation of his own code. 2.2 minutes is still way too long.

What approach should I take to reduce my processing time? I'm very new to GIS, so I'm not sure if there is a more obvious faster way of achieving what I'm trying to do.

NEW:

with fiona.drivers():
    try:
        if os.path.isdir(sourceDatabaseFile):
            numBLDGwithinPerennialWaters = 0
            shapelyEvaulationsAvoided = 0
            discoveredBuildings = [] # List of discovered buildings to avoid re-evaluating in Shapely
            listOfPerennialWaters = []
            listOfAllBuildings = []

           with fiona.open(sourceDatabaseFile, 'r', layer="HydrographySrf") as gdb: # Pointer to HydrographySrf layer of source file

                for hydroFeature in gdb:
                    if (len(gdb) > 0):
                        if (('ZI024_HYP' in hydroFeature['properties']) and (hydroFeature['properties']['ZI024_HYP'] > 0)):
                            listOfPerennialWaters.append(hydroFeature)
                    else:
                        print "No bodies of water to parse in hydrographySrf!!!\n"
                        break

                print "Finished pre-parsing HydrographySrf layer.  ", str(len(listOfPerennialWaters)) + " perennial waters discovered!!!\n"

            with fiona.open(sourceDatabaseFile, 'r', layer="StructurePnt") as gdbStructures: # Pointer to StructurePnt layer of source file
                for building in gdbStructures:
                    if (len(gdbStructures) > 0):
                        listOfAllBuildings.append(building)
                    else:
                        print "No buildings to parse in structurePnt!!!\n"
                        break
                print "Finished pre-parsing StructurePnt layer.  ", str(len(listOfAllBuildings)) + " buildings discovered!!!\n"

            outerLoop = 0

            for perennialWater in listOfPerennialWaters:
                outerLoop = outerLoop + 1
                for building in listOfAllBuildings:
                    if (building['id'] in discoveredBuildings):
                        print "Building already discovered so skipping to next building for evaluation! \n"
                        shapelyEvaulationsAvoided = shapelyEvaulationsAvoided + 1
                        continue
                    elif shape(perennialWater['geometry']).contains(shape(building['geometry'])):
                        discoveredBuildings.append(building['id'])
                        print "Found building ID: ", str(building['id']), "contained in perennial waters!!!\n"
                        numBLDGwithinPerennialWaters = numBLDGwithinPerennialWaters + 1
  • 1
    Can you post your code? It will be hard to determine if there are performance hiccups without seeing the code. – crmackey Jun 2 '16 at 2:57
  • @crmackey I edited my original post with my code snippet. Please see above and thanks in advance! – George Jun 2 '16 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.