Combining python (parallel) multiprocessing toolbox with arcpy

Question

I'm trying to cut down on arcpy processing time by using the python multiprocessing toolbox. Using the example from http://wltrimbl.github.io/2014-06-10-spelman/intermediate/python/04-multiprocessing.html as a guideline.

Basically, I'm trying to select all tracks (point features) with a specific trackID (in field 'track'), give it a buffer and extract all road features within that buffer as preprocessing for another script. But the data set is so large it would take me about a week to run it.

I now have the following code, but I'm having a problem with the program executing normally, but not outputting the files in Tracks.gdb. Does anyone have advice?

import arcpy
from arcpy import env
import multiprocessing
def MyFunction(i):
    TrackFeatureOut = "C:\tryingOut\Tracks.gdb\Track_" + str(i)
    RoadFeatureOut = "C:\tryingOut\Tracks.gdb\Road_" + str(i)
    if not (RoadFeatureOut in featureclasses):
        BufferFeatureOut = "C:\tryingOut\Temporary.gdb\Buffer_" + str(i)
        Query = "track =" + str(i)

        arcpy.Select_analysis("C:\tryingOut\Input.gdb\Tracks", TrackFeatureOut, Query)

        arcpy.Buffer_analysis(TrackFeatureOut, BufferFeatureOut, "100 Meters")

        arcpy.Clip_analysis("C:\tryingOut\Input.gdb\Roads", BufferFeatureOut, RoadFeatureOut)

if __name__ == '__main__':
    global featureclasses
    env.workspace = "C:\tryingOut\Tracks.gdb"
    featureclasses = arcpy.ListFeatureClasses()

    # Getting all unique track numbers
    Track = set([])
    cursor = arcpy.da.SearchCursor("C:\tryingOut\Input.gdb\Tracks",["track"],sql_clause=(None, 'ORDER BY track' ))
    for row in cursor:
        Track.add(row[0])

    # Trying to apply multiprocessing
    pool = multiprocessing.Pool(multiprocessing.cpu_count())
    for i in Track:
        if Counter % 1000 == 0:
            print "Counter = ", Counter, "Current Track = ",i
        Counter += 1    
        pool.apply_async(MyFunction, i)

Answer 1

There's two improvements I should've made here:

Firstly, there's the arcpy function arcpy.SplitByAttributes_analysis(), which (in my experience) works more efficiently than the select function.

Secondly, my form of multiprocessing was off, or at least I found a way that did work. I managed to write a short function which should take any function and iterable list as input arguments and multiprocess it to the number of cores available:

def multiprocessingfunc(Func, List):
    Count = 0
    while Count < len(List):
        if len(multiprocessing.active_children()) < multiprocessing.cpu_count():
            p = multiprocessing.Process(target=Func, args=(List[Count],)
            p.start()
            Count += 1