1

I am attempting to convert 15 feature classes containing roughly 70,000 contours each into polygons record by record. Using my current code, each feature class takes roughly 16 hours on my current computer using a Python script (i.e. not executed from within ArcGIS, only using the ArcPy library). My computer has a 3.2 GHz W3670 processer, 12 GB RAM, and an SSD. So, I think it should be able to process the data faster.

I have had no exposure to multiprocessing until I started looking around yesterday trying to find a method to speed up this process. I found several articles (see ESRI blog post and user post; earlier posts on GIS stack exchange 1 and 2)

Using the above examples, I attempted to mash-up what I saw there with my existing code and came up with:

import multiprocessing
import arcpy
arcpy.env.overwriteOutput = True

def contour2polygon(oid):
    contour = arcpy.management.MakeFeatureLayer('[some_path]\\test1.gdb\\contours','layer{0}'.format(oid[0]),'"OBJECTID" = {}'.format(oid[0]))
    out_name = "[some_path]\\test2.gdb\\feat{}".format(str(oid[0]))
    polygons = arcpy.FeatureToPolygon_management(contour, memory_name)
    return polygons

def main():
    contours = '[some_path]\\test1.gdb\\contours'
    oids = [row[0] for row in arcpy.da.SearchCursor(contours, 'OBJECTID')]

    cpuNum = multiprocessing.cpu_count() - 1

    pool = multiprocessing.Pool(processes = cpuNum)
    polygons = pool.map(contour2polygon, oids)
    pool.close()
    pool.join()

if __name__ == '__main__':
    main()

However, when I execute the script, I just see a blank box and my Task Manager does not indicate any change in activity. Is there something my code is missing? I understand from the above, linked examples that I should have a worker function, but I get lost in the rest of the details especially with how I could convert my old scripts to using multiprocessing.

Improve this question
6
  • 1
    I found multiprocessing in ArcGis only works well with very heavy tasks, e.g. clip raster- project - clip. With fast things involving read/write it is unstable to say the least.
    – FelixIP
    Mar 24, 2018 at 9:03
  • 1
    One of the key concepts of parallel processing is partitioning the problem to eliminate (or at least strictly limit) concurrent access. Failure to do this will result in longer execution or even deadlock. At a minimum, you'd need one file geodatabase per task, but the high I/O load represented by these tasks doesn't lend them toward a parallel solution. You might be better off clipping the feature classes into more skinnier layers (width wise), then Union and Dissolve them later.
    – Vince
    Mar 24, 2018 at 12:34
  • @FelixIP Okay, thanks for your note. Is there any other way to speed up this processing or am I stuck? N.B. I am replying to Vince as well in a post after this. Sorry for the flaming.
    – jwx
    Mar 24, 2018 at 16:47
  • @Vince In my scenario I have a primary file geodatabase with the initial contours feature class which I then use to create a secondary set of file geodatabases and fill them with the polygons created from contours. I'm guessing this is not even advisable with parallel processing?
    – jwx
    Mar 24, 2018 at 16:47
  • 1
    Contention is bad is parallel operation. You have FGDB access contention and disk contention in this scenario. If there were multiple independent SSDs, you might have a better chance, but you need to smart-size the tasks first, since 16 hours is far too long for a single atomic task.
    – Vince
    Mar 24, 2018 at 17:35

1 Answer 1

Reset to default
1

I think the problem is how you are accessing the OBJECTID value in your worker function.

In the main() function you create a list of OBJECTIDS called oids. Each value in that list should be passed into your contour2polygon() function as an integer.

But within your code you are treating it as a list oid[0] which would cause an error and all the cores lock up. Reference the OBJECTID simply as oid in your code.

So your code should be:

def contour2polygon(oid):
    contour = arcpy.management.MakeFeatureLayer('[some_path]\\test1.gdb\\contours','layer{0}'.format(str(oid)),'"OBJECTID" = {}'.format(str(oid)))
    out_name = "[some_path]\\test2.gdb\\feat{}".format(str(str(oid)))
    polygons = arcpy.FeatureToPolygon_management(contour, memory_name)
    return polygons
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.