Questions tagged [parallel-processing]

Use this tag when asking questions about multi or parallel processing - a programming paradigm used to speed up computationally intensive geoprocessing tasks by executing them over multiple CPU cores or threads.

Filter by
Sorted by
Tagged with
22
votes
2answers
3k views

Arcpy Parallel Processing Environment

I recently discovered the Parallel Processing environment setting in ArcGIS (this must be new to 10.1), however, there is no mention as to what tools are designed to utilize this environment setting. ...
20
votes
1answer
2k views

ArcGIS 10.1 Python AddIn Using Multiprocessing crashes ArcMap?

I would like to run a multiprocessing task from a python add-in tool. My issue is that the process keeps failing. Basically crashes ArcMap. Here is my basic code: def function(startOID, endOID, fc)...
15
votes
3answers
4k views

Porting Avenue code for Producing Building Shadows to ArcPy/Python for ArcGIS Desktop? [duplicate]

Whuber provided an answer at Producing building shadows using ArcGIS Desktop? which required using Avenue code. Any idea how to make it work in ArcGIS Desktop 10?
15
votes
3answers
3k views

Parallelising GIS operations in PyQGIS?

A common requirement in GIS is to apply a processing tool to a number of files or apply a process for a number of features in one file to another file. Much of these operations are embarrassingly ...
13
votes
2answers
3k views

Multiprocessing Errors - ArcGIS implementation

I was wondering if anyone else in the community here has attempted to use multi-processing for spatial analyses. Namely I am trying to iterate through a series of rasters, create a multiprocessing job ...
12
votes
2answers
3k views

Methods for optimizing multcore processing in ArcGIS

I am interested in learning methods to utilize the full extent of multicore processing power available on a desktop computer. Arc states that background geoprocessing allows the user to utilize ...
12
votes
1answer
2k views

OGR/GDAL threading results in low core utilization

I'm trying to process some raster data using ogr/gdal and I can't seem to get full utilization of all the cores on my machine. When I only run the process on a single core, I get 100% utilization of ...
11
votes
1answer
2k views

Does GDAL support parallel processing?

I would like to speed up the process corresponding to this command: gdalwarp -t_srs EPSG:4326 -overwrite input.ntf output.tif Is there a way to use parallel processing in GDAL? Semi-offtopic: If not,...
8
votes
5answers
1k views

python.multiprocessing and “FATAL ERROR (INFADI) MISSING DIRECTORY”

While trying to do multiprocessing with arcpy, I am occasionally running into this error: FATAL ERROR (INFADI) MISSING DIRECTORY I have no clue what is triggering this error, and it crashes the ...
7
votes
2answers
2k views

How parallelize the extract function for raster files in R?

I´m trying to parallelize a simple "extract" operation like this: UseCores<-detectCores() -1 cl<- makeCluster(UseCores) registerDoParallel(cl) all.test.poly_period.test<-extract(stack.ts, ...
7
votes
1answer
4k views

Can multiprocessing with arcpy be run in a script tool?

I have been tinkering around with the multiprocessing module to resolve some processing of data in ArcMap 10.3. I have created a script which works fine if I run it in IDLE. I see all my cores max ...
7
votes
1answer
246 views

Arcpy and multiprocessing not working

I want to utilize multiprocessing within my arcpy scripts but my first test has run into a problem which I don't understand. I'm running the script in the python window: import arcpy from ...
6
votes
1answer
1k views

How to do parallel queries in postgis [closed]

Is there any way to do queries in postgis using multiple cores in a transparent way, other than splitting my queries among several client connections ?
6
votes
3answers
403 views

How to run qgis scripts in parallel across remote server(s)

I have a stand alone python script using qgis that I wrote with the help of the SE question here The script takes three arguments being; an input file path, an output file path, and another input ...
6
votes
1answer
415 views

parallelization of raster mosaic

I would like to create a raster mosaic of the national USGS NED 30m DEM (~50 GB). I'd like to leverage parallel processing using python multiprocessing module where possible. I'm using ArcGIS 10.0. My ...
5
votes
1answer
910 views

How to implement multiprocessing with multiple geoprocesses?

I am relatively new to Python and thought I would give a stab at multiprocessing. I have a script that runs well in IDLE or as an ArcMap toolbox script. After perusing these forums and docs.python, I ...
5
votes
1answer
864 views

Multiprocessing in a QGIS Plugin

I have developed a QGIS plugin that processes an image using a bunch of other information, but it runs slowly. I would like to parallelize, as I can do with Multiprocessing.Process outside a QGIS ...
5
votes
0answers
869 views

Is it possible to do parallel processing in GDAL and QGIS?

It seems like processing a large raster dataset in QGIS can cause a crash, for example in QGIS failing to complete clip or intersect and crash dump in qgis. Is it possible to use GDAL parallel ...
4
votes
2answers
2k views

Processing multiple files simultanously using python

I have 11000 point shapefiles to run Near tool on against another point shapefile. I am looking for a way to process 10 files at the same time. Is there a way to process so many files using Python ...
4
votes
1answer
1k views

R - multicore approach to extract raster values using spatial points

It might look like this question is duplicated, but I'm asking about data extraction by points, rather than by polygons, and I couldn't find hints about point extraction. Therefore, please bare with ...
4
votes
2answers
1k views

How to achieve parallel Kriging in R to speed up the process?

I have tried the gstat package in R to perform bathymetry predictions based on Kriging interpolation methods. However, the process is very time consuming due the large amount of sampling data I'm ...
4
votes
1answer
1k views

Can I increase the number of processes in ArcGis Server 10.1?

I use ArcMap 10.1. I have a map with many Mosaic and Raster layers. They are stored in many gdb-s. I publish this map with option "Create tiles on demand". But this tiles are created very slowly (may ...
4
votes
1answer
2k views

How to write to a single table via Python multiprocessing

A comment in the post Can multiprocessing with arcpy be run in a script tool? got me thinking, as I often need to do exactly this: Just beware of deadlocking situations (two Insert cursors in the ...
4
votes
1answer
311 views

Splitting data into geographic zones for multiprocessing

I have been looking at multiprocessing and tools like Find Duplicates in Big Data and they seem to simply split the data into sections based on rows/number of records. I am trying to do an intersect ...
4
votes
1answer
80 views

Parallelize Extracting Large Raster Values (one band) to Points

I have sf data with single point geometries (points) and a 23,000 x 18,000 elevation RasterLayer with a single band, loaded from a geotiff (elev). They have the same CRS. I want to extract the ...
4
votes
1answer
5k views

Python Postgres Multiprocessing speed increase

I'm trying to speed up my PostGIS queries using multiprocessing. My current setup is using python and psycopg2 such as below. Although this has given a speed increase, it still seems like there are ...
4
votes
0answers
297 views

How to use the multiprocessing module on python to execute a spatial join? [closed]

I have several shapefiles of the entire Amazon rainforest each with millions of polygons (they are fairly detailed and take some time to draw on ArcMap). I need each of these shapefiles to be ...
3
votes
2answers
686 views

Python and QGIS multiprocessing documentation?

Where can we find better documention for python QGIS multiprocessing. Especially regarding the multiprocessing of python and QGIS.
3
votes
2answers
4k views

Execute multiple process in parallel of Arcpy script

I have seen some difficult ways to do parallel processing, but I wonder if it is possible to simply execute multiple process of the same ArcPy script at the same time. My script makes some changes to ...
3
votes
1answer
2k views

Unable to do a parallel INSERT using Postgres 9.6.0 & PostGIS 2.3.0

The following PostGIS query executes in parallel using Postgres 9.6's new parallel processing feature: SELECT pnts.mb_code11, bdys.mb_code11 FROM testing.mb_random_points AS pnts INNER JOIN testing....
3
votes
1answer
169 views

Python rtree and parallelized code

I'm trying to parallelize with multiprocessing.Pool() some processes that involve queries of R-trees. In the non-parallel procedure, it works as expected. But when I try to do it in parallel (even ...
3
votes
1answer
787 views

arcpy with multiprocessing throws “UnpickleableError: Cannot pickle <type 'geoprocessing Layer object'> objects”

I'm attempting to speed up a process which is currently running synchronously, by using the python multiprocessing module. I'm having trouble sending a feature layer to a function which is called by ...
3
votes
1answer
710 views

How to use multiprocessing or pool to boost up the performance of ArcPy script

I created some arcpy script which actually runs slow on huge dataset. such as 100000 feature in it. I read from below path that multiprocessing can boost up arcpy script script performance. http://...
3
votes
1answer
1k views

Multiprocessing issues with ArcPy

I am using ArcGIS 'watershed' routine in a script which makes use of multiprocessing. This works fine, but during execution I get the message: Unable to remove directory. Possible causes: 1- Not ...
3
votes
0answers
208 views

Multiprocessing with geopandas

I am learning how to implement the multiprocessing with spatial data using the module multiprocessing. I am trying to implement a spatial intersection between a polygon file and a grid file. I am ...
3
votes
0answers
345 views

Multiprocessing in python qgis?

When i try to do mutliprocessing in python Qgis it doesn't work from multiprocessing import Process,Lock import multiprocessing def name1(l,i): print i if __name__=='__main__': lock=Lock() ...
3
votes
0answers
510 views

Using Multiprocessing inside a plugin in Qgis [closed]

I spent quite a bit of time looking on how to use the multiprocessing package, but couldn't find anything on how to use it inside a plugin in QGIS. I am developing a plugin that does some ...
3
votes
0answers
381 views

python.multiprocessing and “FATAL ERROR (INFADI) MISSING DIRECTORY” [closed]

I have a question which is closely related to this old question here: python.multiprocessing and "FATAL ERROR (INFADI) MISSING DIRECTORY" I have also looked at this, which seems to relate ...
2
votes
2answers
1k views

Full utilization of CPU-cores with gdalwarp

I need to process many large GeoTiff files with gdalwarp. For this I have used the following command: gdalwarp -r cubic -s_srs EPSG:25832 -t_srs EPSG:3857 -cutline shapefile.shp -crop_to_cutline -...
2
votes
1answer
351 views

Setting ArcPy environment variables with Python multiprocessing?

I have a script which uses ArcPy + Python multiprocessing. Are ArcPy environment variables shared between separate Python processes? If I write to arcpy.env.extent from within each process am I ...
2
votes
1answer
994 views

Multiprocessing error in QGIS with Python on Windows?

I have this error when I try to use Manager() from python multiprocessing library. (Just start QGIS open Python console and execute the two following lines) >>> import multiprocessing >&...
2
votes
1answer
333 views

problem deleting shapefiles because still in use by arcpy cursor [duplicate]

I have a function outputTblContents which contains def outputTblContents ( path ): rows = arcpy.SearchCursor(path, fields="alpha; beta") for row in rows: print("{0} => {1}").format(...
2
votes
1answer
77 views

How to use multiprocessing with Make Route Analysis Layer tool

I am using ArcGIS 10.4.1 and Network Analyst extension. As an IDE i use Spyder and Python version 2.7. I created a script where there is a method with the following arguments. def ra(EndPoint, ...
2
votes
1answer
137 views

Dividing a large raster into blocks for iteration in R

I have very big rasters for processing (#pixels >1,5*10^10) for this project I am working on. Simple functions such as finding the area of a pixels with a particular value, across the entire raster 1) ...
2
votes
1answer
212 views

PyWPS with python multiprocessing

I have a python script that runs a set of queries in parallel using multiprocessing.pool and map_async. The script from @Aaron works perfectly. Now I want to integrate the same functions into a ...
2
votes
1answer
429 views

Batch intersect and multiprocessing with arcpy

I admit what I am working on is way over my head, especially as I've only been dabbling in python for a few months. I am attempting to write a script that allows me to use Intersect analysis on about ...
2
votes
0answers
65 views

Parallel optimization for query over LiDAR dataset

I'm using PostGIS + pointcloud to store a huge dataset of LiDAR point (2M patches of 600points) I query this database to obtain the altimetic profile between two points. At the moment I have a huge ...
2
votes
1answer
505 views

Combining python (parallel) multiprocessing toolbox with arcpy

I'm trying to cut down on arcpy processing time by using the python multiprocessing toolbox. Using the example from http://wltrimbl.github.io/2014-06-10-spelman/intermediate/python/04-multiprocessing....
2
votes
0answers
1k views

How to make a Python Multiprocessing Script work on Windows OS [closed]

I use the Python multiprocessing module to speed up a function (get_craters_for_buffering_BCC) in my script. In this function, I want to identify points (passed as a list: crater_features_list) within ...
2
votes
0answers
68 views

Avoiding edges using GRASS generalization?

I'm trying to process the national wide land cover data into vector polygons. In order to be quick I separate the data into several tiles and process them in parallel. process flow is : 1. v.in.ogr ...