Use this tag when asking questions about multi or parallel processing - a programming paradigm used to speed up computationally intensive geoprocessing tasks by executing them over multiple CPU cores or threads.
0
votes
0answers
77 views
Spatial analysis and parallel processing in R
I am trying to do a point-in-poly analysis using R. I have 4.5 million points and 5k polygons. This is computationally intensive. I was pointed to parallel processing through another post and tried to ...
0
votes
0answers
29 views
Parallel processing issue biomod2 - snowfall
Is there a known issue between the recent biomod2 version (3.3-15) and snowfall (version 81.84-6.1)? I adapted the multi-species-modeling approach from Guisan, Thuiller & Zimmermann 2017 and ...
1
vote
0answers
11 views
Multiprocessing GALayerToRasters
I'm new to ArcPy and am creating a script to automate some interpolation work.
I have multiple GA Layers outputted by an Empirical Bayesian Kriging interpolation (using arcpy....
0
votes
0answers
48 views
Speeding up Euclidian distance estimation in raster using R?
I am estimating the euclidian distance in a huge raster with the package raster of R in Windows system.
However, the processing takes a lot of time (in this case, more than 12 hours!).
Is there any ...
1
vote
1answer
52 views
Converting polylines to polygons in ArcGIS using multiprocessing
I am attempting to convert 15 feature classes containing roughly 70,000 contours each into polygons record by record. Using my current code, each feature class takes roughly 16 hours on my current ...
0
votes
0answers
57 views
arcpy and multiprocessing: switch workflow to shapefiles to avoid schemalocks
I need to processe a large dataset through a bunch of interwoven scripts using arcpy. On a small test dataset my workflow functions fine so far but when it comes to the large dataset i see no other ...
1
vote
2answers
149 views
Using MultiProcessing With Update Cursor?
I am trying to use multiprocessing to help me run a tool that will clip a feature and sum an attribute in it. The tool needs to iterate through about 24,000 rows of a featureclass while doing this. ...
4
votes
1answer
626 views
R - multicore approach to extract raster values using spatial points
It might look like this question is duplicated, but I'm asking about data extraction by points, rather than by polygons, and I couldn't find hints about point extraction. Therefore, please bare with ...
1
vote
1answer
193 views
Does anyone already parallelized mosaic function for rasters in R?
Does anyone already paralleled mosaic function for rasters in R?
It seems that snow still doesn´t deal with this on an automatic way.
The idea is something like this:
r1<-stack("fit__test23KLR....
0
votes
0answers
88 views
Multiprocessing Script Tool doesn't crash but will not function
I've made created a script that uses multiprocessing to break up a very large point dataset into pieces and extract values from 2 different rasters to them. When I tested it using the command line ...
4
votes
2answers
1k views
How parallelize the extract function for raster files in R?
I´m trying to parallelize a simple "extract" operation like this:
UseCores<-detectCores() -1
cl<- makeCluster(UseCores)
registerDoParallel(cl)
all.test.poly_period.test<-extract(stack.ts, ...
3
votes
1answer
157 views
PyWPS with python multiprocessing
I have a python script that runs a set of queries in parallel using multiprocessing.pool and map_async.
The script from @Aaron works perfectly.
Now I want to integrate the same functions into a ...
3
votes
2answers
1k views
Processing multiple files simultanously using python
I have 11000 point shapefiles to run Near tool on against another point shapefile.
I am looking for a way to process 10 files at the same time.
Is there a way to process so many files using Python ...
10
votes
1answer
1k views
Does GDAL support parallel processing?
I would like to speed up the process corresponding to this command:
gdalwarp -t_srs EPSG:4326 -overwrite input.ntf output.tif
Is there a way to use parallel processing in GDAL?
Semi-offtopic: If not,...
5
votes
1answer
545 views
Multiprocessing in a QGIS Plugin
I have developed a QGIS plugin that processes an image using a bunch of other information, but it runs slowly. I would like to parallelize, as I can do with Multiprocessing.Process outside a QGIS ...
2
votes
2answers
812 views
How to achieve parallel Kriging in R to speed up the process?
I have tried the gstat package in R to perform bathymetry predictions based on Kriging interpolation methods. However, the process is very time consuming due the large amount of sampling data I'm ...
0
votes
2answers
542 views
Multiplication of two raster stack with R using multicore parallel processes
I am working with two global raster stacks, each containing 60 layers using the R Software. The layers are global raster files (in GTiff format) at a resolution of 5 min of arc. The first stack ...
0
votes
0answers
48 views
Parallel batch processing with ArcPy? [duplicate]
How can I perform parallel batch processing with ArcPy / ArcGIS Desktop?
My use case requires batch processing of many similar files and I would like to run these jobs in parallel. Unfortunately ...
2
votes
1answer
394 views
Combining python (parallel) multiprocessing toolbox with arcpy
I'm trying to cut down on arcpy processing time by using the python multiprocessing toolbox. Using the example from http://wltrimbl.github.io/2014-06-10-spelman/intermediate/python/04-multiprocessing....
2
votes
2answers
996 views
Full utilization of CPU-cores with gdalwarp
I need to process many large GeoTiff files with gdalwarp.
For this I have used the following command:
gdalwarp -r cubic -s_srs EPSG:25832 -t_srs EPSG:3857 -cutline shapefile.shp -crop_to_cutline -...
3
votes
1answer
2k views
Unable to do a parallel INSERT using Postgres 9.6.0 & PostGIS 2.3.0
The following PostGIS query executes in parallel using Postgres 9.6's new parallel processing feature:
SELECT pnts.mb_code11, bdys.mb_code11
FROM testing.mb_random_points AS pnts
INNER JOIN testing....
2
votes
0answers
738 views
How to make a Python Multiprocessing Script work on Windows OS [closed]
I use the Python multiprocessing module to speed up a function (get_craters_for_buffering_BCC) in my script. In this function, I want to identify points (passed as a list: crater_features_list) within ...
2
votes
1answer
361 views
Batch intersect and multiprocessing with arcpy
I admit what I am working on is way over my head, especially as I've only been dabbling in python for a few months. I am attempting to write a script that allows me to use Intersect analysis on about ...
1
vote
1answer
3k views
Using 'sp' method to subset SpatialPointsDataFrame with SpatialPolygon returns error in parLapply()
I have numerous very large data sets of ship tracking data (Automated Information System; AIS). These are originally in a file geodatabase and can be downloaded here (this is a smaller file for ...
7
votes
1answer
191 views
Arcpy and multiprocessing not working
I want to utilize multiprocessing within my arcpy scripts but my first test has run into a problem which I don't understand. I'm running the script in the python window:
import arcpy
from ...
5
votes
0answers
725 views
Is it possible to do parallel processing in GDAL and QGIS?
It seems like processing a large raster dataset in QGIS can cause a crash, for example in QGIS failing to complete clip or intersect and crash dump in qgis. Is it possible to use GDAL parallel ...
2
votes
0answers
66 views
Avoiding edges using GRASS generalization?
I'm trying to process the national wide land cover data into vector polygons.
In order to be quick I separate the data into several tiles and process them in parallel.
process flow is :
1. v.in.ogr
...
1
vote
1answer
576 views
R rasterEngine Parallel Processing, serialized and unserialized memory errors
I am trying to perform some raster math and wanted to try parallel processing in R using rasterEngine.
I am able to run the script for a few minutes and watch my CPU usage increase substantially, ...
3
votes
1answer
598 views
How to use multiprocessing or pool to boost up the performance of ArcPy script
I created some arcpy script which actually runs slow on huge dataset. such as 100000 feature in it.
I read from below path that multiprocessing can boost up arcpy script script performance.
http://...
3
votes
0answers
299 views
Multiprocessing in python qgis?
When i try to do mutliprocessing in python Qgis it doesn't work
from multiprocessing import Process,Lock
import multiprocessing
def name1(l,i):
print i
if __name__=='__main__':
lock=Lock()
...
3
votes
2answers
553 views
Python and QGIS multiprocessing documentation?
Where can we find better documention for python QGIS multiprocessing.
Especially regarding the multiprocessing of python and QGIS.
0
votes
2answers
2k views
Really slow extraction from raster even after using crop?
I have a large raster file (245295396) cells and stacks of rasters having 4 layers each which lie in the extent of this large raster. To start with I am trying to get value from one stack (3 channels) ...
3
votes
0answers
444 views
Using Multiprocessing inside a plugin in Qgis [closed]
I spent quite a bit of time looking on how to use the multiprocessing package, but couldn't find anything on how to use it inside a plugin in QGIS.
I am developing a plugin that does some ...
6
votes
3answers
367 views
How to run qgis scripts in parallel across remote server(s)
I have a stand alone python script using qgis that I wrote with the help of the SE question here
The script takes three arguments being; an input file path, an output file path, and another input ...
2
votes
1answer
232 views
problem deleting shapefiles because still in use by arcpy cursor [duplicate]
I have a function outputTblContents which contains
def outputTblContents ( path ):
rows = arcpy.SearchCursor(path, fields="alpha; beta")
for row in rows:
print("{0} => {1}").format(...
1
vote
2answers
481 views
Multiprocessing with arcpy and Solar Analyst (spatial analyst)
I need help getting multiprocessing to work with the spatial analyst tool - Solar Analyst.
I have tested the multiprocessing bit and it works fine. This sample is passing it a file containing 2 lines ...
4
votes
1answer
1k views
How to write to a single table via Python multiprocessing
A comment in the post Can multiprocessing with arcpy be run in a script tool? got me thinking, as I often need to do exactly this:
Just beware of deadlocking situations (two Insert cursors in the ...
7
votes
1answer
3k views
Can multiprocessing with arcpy be run in a script tool?
I have been tinkering around with the multiprocessing module to resolve some processing of data in ArcMap 10.3.
I have created a script which works fine if I run it in IDLE. I see all my cores max ...
4
votes
0answers
248 views
How to use the multiprocessing module on python to execute a spatial join? [closed]
I have several shapefiles of the entire Amazon rainforest each with millions of polygons (they are fairly detailed and take some time to draw on ArcMap). I need each of these shapefiles to be ...
0
votes
0answers
147 views
A Distributed Cost-Distance Algorithm
I'm currently trying to implement a distributed cost-distance algorithm, however I can't seem to invent my own and still make it fast. I've been googling around a bit but haven't found anything that ...
2
votes
0answers
308 views
Multiprocessing and ArcGIS in_memory workspace? [closed]
I am trying a script that makes use of multiprocessing and a number of tools within arcpy. The script works only if I write all input, intermediate, and output from child multiprocessing executions to ...
0
votes
1answer
102 views
Parallel Computing error using GP tool in C#
I have been using ArcGIS Geoprocessor(GP) tool whenever possible even though it is like a black box in my ArcObject development.
Most of my operations are involved with data processing, such as ...
15
votes
3answers
3k views
Parallelising GIS operations in PyQGIS?
A common requirement in GIS is to apply a processing tool to a number of files or apply a process for a number of features in one file to another file.
Much of these operations are embarrassingly ...
2
votes
1answer
289 views
Setting ArcPy environment variables with Python multiprocessing?
I have a script which uses ArcPy + Python multiprocessing.
Are ArcPy environment variables shared between separate Python processes?
If I write to arcpy.env.extent from within each process am I ...
3
votes
1answer
650 views
arcpy with multiprocessing throws “UnpickleableError: Cannot pickle <type 'geoprocessing Layer object'> objects”
I'm attempting to speed up a process which is currently running synchronously, by using the python multiprocessing module.
I'm having trouble sending a feature layer to a function which is called by ...
1
vote
1answer
174 views
Maintaining reference to feature layer (rather than its name string) throughout Python script? [duplicate]
When creating a Layer in arcpy, the syntax uses a string to denote the layer name, as shown in the Make Feature Layer sample script:
import arcpy
arcpy.env.workspace = "C:\Program Files (x86)\ArcGIS\...
1
vote
0answers
38 views
Sending commands into FGDB
I have a large .gdb of all blocks in the United States, available from Census. I would like to create a point shape file of all census tract centroids in the US.
I have QGIS with FileGDB enabled, as ...
1
vote
1answer
2k views
How to write output from subprocess to disk (using multiprocessing)
This code was developed from another question I posted about how to kill an arcpy tool if it takes longer than a specified time. See How to cancel tool execution in python. Using the multiprocssing ...
3
votes
1answer
1k views
Multiprocessing issues with ArcPy
I am using ArcGIS 'watershed' routine in a script which makes use of multiprocessing.
This works fine, but during execution I get the message:
Unable to remove directory. Possible causes:
1- Not ...
4
votes
1answer
4k views
Python Postgres Multiprocessing speed increase
I'm trying to speed up my PostGIS queries using multiprocessing. My current setup is using python and psycopg2 such as below. Although this has given a speed increase, it still seems like there are ...
