Tag Info

Hot answers tagged

19

I've set this up as a ArcToolbox type of script, rather than field calculator as whuber did. This is pretty much a straight port of whubers Avenue code. EDIT: the script assumes the height is stored in a field in the featureclass attribute table, not a featureclass with 3D geometries (PolygonZ) import arcpy,os,math arcpy.env.overwriteOutput=True ...


16

I made a few improvements to @Luke's version of the script, mainly for better performance. The GP methods really don't like being called in a tight loop; by removing an unnecessary GP call and moving the necessary one outside of the innermost loop I sped up performance by 10x at least. This also fixes shadows not being created for inner rings, degrees are ...


10

As alluded to in the comments of my previous answer (kept intact instead of edited, for comparison purposes), performance could be a lot better with additional optimizations (using the in_memory workspace instead of using geometry objects, move GP calls outside of loops where possible) as well as by utilizing multiprocessing. Here is another version of the ...


8

In my experience the biggest problem is managing stability. If you do six weeks of processing in a single night you will also have six weeks of inexplicable errors and bugs. An alternative approach is to develop standalone scripts that can run independently and fail without causing problems: Split the data into chunks that a single core can process in ...


8

Here is an example of a multicore arcpy script. The process is very CPU-intensive so it scales very well: Port "Producing Building Shadows" Avenue code to ArcGIS 10 Some more general info in this answer: Can concurrent processes be run in a single model?


7

Parallel processing is easier 'shown than done.' In the case of stuffing this all into a button, I'm guessing two issues: Multiple threads block the ArcMap UI thread, or ArcMap puts its own schema lock on the data source and doesn't permit the python process access to the data. Hmm looking further issue has been documented here in an ArcGIS Resources ...


6

OK. That was a day of my life that I'll never get back again. Turns out the problem was not in the code I posted above. That's totally fine. Turns out that this was a case of threading.Thread vs. multiprocessing.Process. As pointed out in the python documentation: The multiprocessing package offers both local and remote concurrency, effectively ...


5

Here are some things to check: Are you using cursors? Are you releasing them? Are you trying to re-use any objects in different processes? Are you sharing the same temp location? Are you doing in memory processing? In general, the arcpy is just a wrapper around the com objects and any type of multiprocessing will be tricky.


4

Each IWorkspace connection (i.e each database connection) has thread affinity. Two threads cannot share the same workspace. You can have one thread own the resource and then sync the access, but if you are going to be using straight gp functions, then that is even not an option. The easiest (lame) way is to create separate processes and then do multi ...


4

If there is no QGIS official documentation available for multiprocessing QGIS using Python then I recommend reviewing earlier Q&As here that have tags multiprocessing, qgis and python. There are two with answers that would seem relevant to what you are trying to do: How can I parallelise embarrassingly parallel GIS operations in QGIS python? python ...


4

There are several overlapping issues here. ArcGIS Desktop is single-threaded, but can make use of a multi-core machine because it can then get the exclusive use of one core. Unless there's a Direct Connect connection to an enterprise geodatabase, in which case, each connection will be run as an additional thread. ArcGIS Server supports multiple cores ...


4

You're probably getting some sort of exception being raised. Perhaps use a Queue to pass messages back to the parent process. Tested working code: import os, arcpy, arcgisscripting, time, sys from multiprocessing import Process from multiprocessing.queues import SimpleQueue def ConvertCADtoGDB(msgs,in_dgn,out_gdb): try: gp = ...


3

Yes, you can run multiprocessing child processes from a toolbox script. Below is some code to demonstrate in a Python Toolbox (*.pyt). There are a number of "gotchas". Some (but not all) will be applicable to Python script tools in a binary toolbox (*.tbx), but I only use Python Toolboxes these days so have not tested. Some "gotchas"/tips: Make sure ...


3

Environments aren't propagated from process to process, so changing extent in one won't affect the other at all.


3

I've encountered this as well and have yet to find a sound fix. My work around is 1) to make sure that the multiprocessing task is robust enough to check if tasks are complete or not then create a new job list. 2) schedule two scripts to launch every 10-15 minutes. One script contains a command to kill select running python processes and the second ...


3

I must admit I am at this point, just a multithreading wannabee, but a blog at http://pythongisandstuff.wordpress.com/2011/07/11/using-arcpy-with-multiprocessing-%e2%80%93-part-3/ suggests that integrating the arcpy.Exists() function is key to making it happen.


3

I found this issue arises when the arcpy.env.workspace and arcpy.env.scratchWorkspace are the same for two different processes. Arc writes almost all intermediate rasters to the workspace (or scratch workspace) in the ESRI GRID format. You can't write two ESRI GRID rasters into the same directory at the same time due to the pseudo-database structure of the ...


3

You have several threads competing for the same resource. Try moving your 'import arcpy' statement into the target of the multiprocessing. You'll ensure arcpy is working with it's own set of environment variables and memory. It sounds absurd, but even though you are setting environment variables in the Multiprocess target method, python is still using ...


3

Just use the following function def run_MultiPros(function, variables): """<function, variables> Execute a process on multiple processors. INPUTS: function(required) Name of the function to be executed. variables(required) Variable to be passed to function. Description: This function will run the given fuction on to multiprocesser. ...


3

See this blog post, it should cover it http://blogs.esri.com/esri/arcgis/2012/09/26/distributed-processing-with-arcgis-part-1/


2

I was wrong in my comment to original post. Suggested workaround does work. In case someone else has that issue here is what needs to be done in QGIS 2.0 before instantiating Manager(). # OSGeo4W does not bundle python in exec_prefix for python path = os.path.abspath(os.path.join(sys.exec_prefix, '../../bin/pythonw.exe')) mp.set_executable(path) sys.argv = ...


2

I don't know if it will work with ArcGis 10 but the Districting tool might meet your requirements. It can be downloaded free from the Esri website Districting for ArcGIS is an add-on that allows you to create defined groupings of geographic data, such as census tracts, ZIP Codes, and precincts, by creating a districting plan. With Districting for ArcGIS, ...


2

I found that I was getting the INFADI error when trying have multiple threads/cores save and modify rasters in one folder. Assigning a subfolder to each task for outputs seems to solve the issue. I believe that the problem had to do with multiple read/writes to peripheral files associated with the raster (e.g. the "info" folder). I now also employ the ...


2

Just to add to @PolyGeo's answer that I also could not find any official documentation regarding multiprocessing (if it even exists!). But there is also another method, described in this blog, which uses multithreading in QGIS which might be useful. Main difference between the two methods are (more of which is discussed here): Multiprocessing allows ...


2

Never tried multiprocessing, decided to give it a go. This script: import os, sys, arcpy, multiprocessing from arcpy import env env.overwriteoutput=1 scratchGDB=r'd:\rubbish\TEST.gdb' def function(inputs): print ("got arg %s" % inputs) outTblName = inputs[0] city = inputs[1] pop = inputs[2] with ...


2

Some general steps for debugging this kind of problem... Set the current workspace, and the scratch workspace to something random and specific to the process. Use actual output files instead of variables. You will be able to see what has been created, and avoid the use of automatically generated file names. Some spatial analyst functions ...


2

I finally found the time to look into this. I don't fully understand the "unpickleable" error message, but a workaround is to pass only strings into the multiprocessor. Something like this: import multiprocessing, arcpy, os def doProcess(fClass): #This function doesn't do anything, it's just to show that accessing arcpy methods is possible ...


1

I think the blocking point you are currently encountering can be illustrated in this short bit of code: import arcpy def doCity(lyr): print(lyr.name) if __name__ == '__main__': arcpy.MakeFeatureLayer_management("C:\Program Files (x86)\ArcGIS\Desktop10.2\TemplateData\TemplateData.gdb\city", "citiesLyr") citiesLyrObject = ...


1

You can use dblink in a native Postgres query to split the query up into separate database connections and execute them simultaneously. This is effectively parallelism in Postgres on a single server. It could be mimicked in Python, but I haven't tried it. There are some limitations: 1) the operation needs to be an insert, not an update. Inserts are ...


1

I think this may be an example of what you are looking for...check out this Esri article on multiprocess geocoding. It says, "Download the tool even if you’re not a geocoder, you may be able to leverage the logic to parallelize other geoprocessing jobs – and if you do, don’t forget to share your own tools!" "... lets you leverage all available CPU cores on ...



Only top voted, non community-wiki answers of a minimum length are eligible