2

I'm trying to cut down on arcpy processing time by using the python multiprocessing toolbox. Using the example from http://wltrimbl.github.io/2014-06-10-spelman/intermediate/python/04-multiprocessing.html as a guideline.

Basically, I'm trying to select all tracks (point features) with a specific trackID (in field 'track'), give it a buffer and extract all road features within that buffer as preprocessing for another script. But the data set is so large it would take me about a week to run it.

I now have the following code, but I'm having a problem with the program executing normally, but not outputting the files in Tracks.gdb. Does anyone have advice?

import arcpy
from arcpy import env
import multiprocessing
def MyFunction(i):
    TrackFeatureOut = "C:\tryingOut\Tracks.gdb\Track_" + str(i)
    RoadFeatureOut = "C:\tryingOut\Tracks.gdb\Road_" + str(i)
    if not (RoadFeatureOut in featureclasses):
        BufferFeatureOut = "C:\tryingOut\Temporary.gdb\Buffer_" + str(i)
        Query = "track =" + str(i)

        arcpy.Select_analysis("C:\tryingOut\Input.gdb\Tracks", TrackFeatureOut, Query)

        arcpy.Buffer_analysis(TrackFeatureOut, BufferFeatureOut, "100 Meters")

        arcpy.Clip_analysis("C:\tryingOut\Input.gdb\Roads", BufferFeatureOut, RoadFeatureOut)

if __name__ == '__main__':
    global featureclasses
    env.workspace = "C:\tryingOut\Tracks.gdb"
    featureclasses = arcpy.ListFeatureClasses()

    # Getting all unique track numbers
    Track = set([])
    cursor = arcpy.da.SearchCursor("C:\tryingOut\Input.gdb\Tracks",["track"],sql_clause=(None, 'ORDER BY track' ))
    for row in cursor:
        Track.add(row[0])

    # Trying to apply multiprocessing
    pool = multiprocessing.Pool(multiprocessing.cpu_count())
    for i in Track:
        if Counter % 1000 == 0:
            print "Counter = ", Counter, "Current Track = ",i
        Counter += 1    
        pool.apply_async(MyFunction, i)
5
  • Your featureclasses does not exist inside my_function. Remove global and fill list after import line
    – FelixIP
    Feb 28, 2017 at 19:55
  • You might be interested in this document I wrote on geonet?
    – Hornbydd
    Feb 28, 2017 at 23:45
  • Also put an r in front of your paths as \t is a tab character, for example you want to be writing TrackFeatureOut = r"C:\tryingOut\Tracks.gdb\Track_" + str(i).
    – Hornbydd
    Feb 28, 2017 at 23:53
  • I'm currently looking at the suggestions, and all code-advice is appreciated. But the script runs well without the implementation of multiprocessing, and it does not return any errors even with multiprocessing. However, it does not seem to enter the MyFunction for some reason Mar 1, 2017 at 9:48

1 Answer 1

0

There's two improvements I should've made here:

Firstly, there's the arcpy function arcpy.SplitByAttributes_analysis(), which (in my experience) works more efficiently than the select function.

Secondly, my form of multiprocessing was off, or at least I found a way that did work. I managed to write a short function which should take any function and iterable list as input arguments and multiprocess it to the number of cores available:

def multiprocessingfunc(Func, List):
    Count = 0
    while Count < len(List):
        if len(multiprocessing.active_children()) < multiprocessing.cpu_count():
            p = multiprocessing.Process(target=Func, args=(List[Count],)
            p.start()
            Count += 1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.