I need to processe a large dataset through a bunch of interwoven scripts using arcpy. On a small test dataset my workflow functions fine so far but when it comes to the large dataset i see no other way as to create subsets to iterate on like below:

jobs = []
for i in list_of_subsetFCs:
    args = (i[0],i[1],10,5,1,"string") #inputFC, parameters and outputfile syntax
    p = multiprocessing.Process(target=my_function, args=args) #my function is importet

my_function makes use of:

  • FieldMappings
  • MakeFeatureLayer_management
  • GetCount_management
  • Select_analysis
  • Intersect_analysis
  • SplitLineAtPoint_management
  • SpatialJoin_analyses
  • AddField_management
  • Append_management

Is it possible and/or desirable to change the whole workflow in my_function to shapefiles in order to avoid GDB schema-locks or is there perhaps another possibility?

(Arcgis 10.2.2, 64bit background processing)

  • Didn't work for me. I even tried to work with subsets saved in different folders. It hits the wall at some stage "..python stopped working". Using 2 processors of of 8 was a max I could get. Benefits don't worth time spend on debugging etc. Leave it to run overnight ;) Works fine though with heavy raster processing tasks. – FelixIP Nov 9 '17 at 22:18
  • @FelixIP i already had some parts of it running successfully on 4 cores. Run over night: SplitLineAtPoint had been already running for 10 nights on 9 million line segments to no avail. That leads me to the conclusion that I NEED a multicore solution but thanks for your input from which i assume you could change the whole process? – maxwhere Nov 9 '17 at 22:31
  • 2
    It depends on what you're doing, read locks can be ignored most of the time but write locks will cause the other processes to fail. In C I would use a mutex docs.python.org/2/library/mutex.html and pass it around for writing. I use the subprocess lib docs.python.org/2/library/subprocess.html to good effect but one thing you must be aware of is sharing the %TMP% directory, to get around this I create a TEMP_{RANDOM} and set environ TMP stackoverflow.com/questions/5971312/… to avoid clashing sessions. – Michael Stimson Nov 9 '17 at 22:56
  • The script I posted gis.stackexchange.com/questions/112946/… maybe naive, but the idea of splitting the area and processing it cell by cell might work, because intermediate Results will fit into memory. How long does it take 100k of features? – FelixIP Nov 10 '17 at 2:33
  • You for sure get lock errors on things like addField and you will need to have copies of those files and merge them at the end. Same with append, you will lock if you try to do multiple appends to one file. If you can create separate files for each process and merge at the end you will see the best gains. I have had limited luck using the Process call from multiprocessing with arcpy. Please verify you are launching multiple python processes and that they all appear to be running. I have seen it launch processes and then only work on one at a time. multiprocessing.map_async works best for me. – Brian W. Nov 15 '17 at 22:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.