Questions tagged [parallel-processing]

Use this tag when asking questions about multi or parallel processing - a programming paradigm used to speed up computationally intensive geoprocessing tasks by executing them over multiple CPU cores or threads.

Filter by
Sorted by
Tagged with
16
votes
3answers
4k views

Porting Avenue code for Producing Building Shadows to ArcPy/Python for ArcGIS Desktop? [duplicate]

Whuber provided an answer at Producing building shadows using ArcGIS Desktop? which required using Avenue code. Any idea how to make it work in ArcGIS Desktop 10?
15
votes
3answers
3k views

Parallelising GIS operations in PyQGIS?

A common requirement in GIS is to apply a processing tool to a number of files or apply a process for a number of features in one file to another file. Much of these operations are embarrassingly ...
7
votes
1answer
4k views

Can multiprocessing with arcpy be run in a script tool?

I have been tinkering around with the multiprocessing module to resolve some processing of data in ArcMap 10.3. I have created a script which works fine if I run it in IDLE. I see all my cores max ...
8
votes
5answers
1k views

python.multiprocessing and “FATAL ERROR (INFADI) MISSING DIRECTORY”

While trying to do multiprocessing with arcpy, I am occasionally running into this error: FATAL ERROR (INFADI) MISSING DIRECTORY I have no clue what is triggering this error, and it crashes the ...
13
votes
2answers
3k views

Multiprocessing Errors - ArcGIS implementation

I was wondering if anyone else in the community here has attempted to use multi-processing for spatial analyses. Namely I am trying to iterate through a series of rasters, create a multiprocessing job ...
20
votes
1answer
2k views

ArcGIS 10.1 Python AddIn Using Multiprocessing crashes ArcMap?

I would like to run a multiprocessing task from a python add-in tool. My issue is that the process keeps failing. Basically crashes ArcMap. Here is my basic code: def function(startOID, endOID, fc)...
3
votes
1answer
800 views

arcpy with multiprocessing throws “UnpickleableError: Cannot pickle <type 'geoprocessing Layer object'> objects”

I'm attempting to speed up a process which is currently running synchronously, by using the python multiprocessing module. I'm having trouble sending a feature layer to a function which is called by ...
2
votes
1answer
998 views

Multiprocessing error in QGIS with Python on Windows?

I have this error when I try to use Manager() from python multiprocessing library. (Just start QGIS open Python console and execute the two following lines) >>> import multiprocessing >&...
22
votes
2answers
3k views

Arcpy Parallel Processing Environment

I recently discovered the Parallel Processing environment setting in ArcGIS (this must be new to 10.1), however, there is no mention as to what tools are designed to utilize this environment setting. ...
6
votes
1answer
417 views

parallelization of raster mosaic

I would like to create a raster mosaic of the national USGS NED 30m DEM (~50 GB). I'd like to leverage parallel processing using python multiprocessing module where possible. I'm using ArcGIS 10.0. My ...
12
votes
1answer
2k views

OGR/GDAL threading results in low core utilization

I'm trying to process some raster data using ogr/gdal and I can't seem to get full utilization of all the cores on my machine. When I only run the process on a single core, I get 100% utilization of ...
4
votes
1answer
2k views

How to write to a single table via Python multiprocessing

A comment in the post Can multiprocessing with arcpy be run in a script tool? got me thinking, as I often need to do exactly this: Just beware of deadlocking situations (two Insert cursors in the ...
3
votes
2answers
4k views

Execute multiple process in parallel of Arcpy script

I have seen some difficult ways to do parallel processing, but I wonder if it is possible to simply execute multiple process of the same ArcPy script at the same time. My script makes some changes to ...
5
votes
0answers
884 views

Is it possible to do parallel processing in GDAL and QGIS?

It seems like processing a large raster dataset in QGIS can cause a crash, for example in QGIS failing to complete clip or intersect and crash dump in qgis. Is it possible to use GDAL parallel ...
5
votes
1answer
926 views

How to implement multiprocessing with multiple geoprocesses?

I am relatively new to Python and thought I would give a stab at multiprocessing. I have a script that runs well in IDLE or as an ArcMap toolbox script. After perusing these forums and docs.python, I ...
4
votes
2answers
2k views

How to achieve parallel Kriging in R to speed up the process?

I have tried the gstat package in R to perform bathymetry predictions based on Kriging interpolation methods. However, the process is very time consuming due the large amount of sampling data I'm ...
4
votes
1answer
1k views

R - multicore approach to extract raster values using spatial points

It might look like this question is duplicated, but I'm asking about data extraction by points, rather than by polygons, and I couldn't find hints about point extraction. Therefore, please bare with ...
3
votes
0answers
522 views

Using Multiprocessing inside a plugin in Qgis [closed]

I spent quite a bit of time looking on how to use the multiprocessing package, but couldn't find anything on how to use it inside a plugin in QGIS. I am developing a plugin that does some ...