4

I have 11000 point shapefiles to run Near tool on against another point shapefile. I am looking for a way to process 10 files at the same time.

Is there a way to process so many files using Python and arcpy in a resonable time considering that each file takes 5 min on average?

  • Have a look at the processing module. – Fezter May 17 '17 at 22:40
  • Beware that disk thrashing will slow access, so you might find that 1000 concurrent files takes ~400 hours. Even 10 concurrent files might take 12-15 minutes to complete. – Vince May 18 '17 at 0:13
  • At 5 min/shapefile a single process would take 38 days to process all of the files. – Bjorn May 18 '17 at 2:45
4

If you are restricted to windows you can write a DOS sript and run it in parallel as decribed parallel execution of shell processes

If can run in linux there is a procedure pretty well worked out using gnu-parallel. I did something similar for QGIS described in more detail at How to run processing commands in parallel in QGIS

Although the examples I give are for QGIS they would work in principle for arcpy in DOS too. ie If you can generate the the proces you neeed to carry out on one file as a BASH script which incorporates the file names as inputs, then you can use gnu parallel in linux or a parallel DOS script in DOS to handle the parallel work. Based on Oles answer to: parallelising-gis-operations-in-pyqgis eg:

Assuming you had a bash script which could run your "near" process as follows

my_standalone_script.py /path/to/a/point/shpfile.shp /path/to/another/point/file.shp /path/to/output/file.shp

You could install gnu parallel and run the script in parallel with:

find /path/to/point/shpfiles -type f -name *.shp | parallel my_standalone_script.py {} /path/to/another_point_file.shp /path/to-output/folder/neared_{/}.shp

As far as reasonable time is concerned gnu parallel will carry out the operation about as well as could be be achieved given any hardware limitations. You can even cluster more computers together and have them all carry out some of the solution each. (provided you can install the software required on each unit).

  • As this question is tagged arcpy, it's likely that the OP is using ArcGIS Desktop which doesn't run on Linux. – Fezter May 18 '17 at 1:51
  • I didnt know it DIDNT run. I'll update to reflect the alternative in DOS – Mr Purple May 18 '17 at 2:03
  • find and GNU Parallel work in Cygwin. It is not DOS, but maybe it is close enough? – Ole Tange May 22 '17 at 5:56
2

Esri has a useful blog post on utilizing Python's multiprocessing module. In many cases the multiprocessing module works fine with Esri's arcpy site package, mostly for embarrassingly parallel geoprocessing tasks that do not require any shared data (e.g. feature classes in a file geodatabase) among the workers.

I have run into so many problems developing arcpy based multiprocessing applications that I rarely bother any more and usually either focus on streamlining and optimizing existing code or opting for open-source multiprocessing solutions.

In any case, when using the multiprocessing module, you will need to optimize the multiprocessing on your machine as there will be a threshold where the workers max out your system resources resulting in diminishing returns.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.