Questions tagged [parallel-processing]
Use this tag when asking questions about multi or parallel processing - a programming paradigm used to speed up computationally intensive geoprocessing tasks by executing them over multiple CPU cores or threads.
129 questions
1
vote
1
answer
3k
views
Parallel processing in Pandas
Using this implementation of parallelization raises an ImportError: cannot import name 'Parallel' from 'multiprocessing' The following code tries parallelization with the "denominator" function and ...
- 399
6
votes
1
answer
1k
views
Parallel processing in QGIS3 on Windows system
I am working on a big vector data set with more than 2.4 million features containing 10x10m squares. I wrote a small script in Python to delete uninteresting fields and perform calculations on the ...
- 399
3
votes
0
answers
456
views
Running Standalone PyQGIS Script with Multiprocessing?
Is it possible to run a standalone PyQGIS script using multiprocessing for better performance?
I'm creating 1,800 rendered map images and would like to run it across 4 processes to speed it up.
If I ...
- 767
1
vote
1
answer
474
views
Multiprocess Spatial Join with spatial index
I'm a newbie in implementing GIS on Python, but the lack of any GIS software at the office has forced me to learn about it.
I have a huge grid (a multipolygon) that I'm currently filling out with ...
- 43
0
votes
1
answer
199
views
Using all processor cores when exporting map to TIFF using ArcGIS Desktop?
By default, ArcGIS Desktop uses only one processor core when you export a map. Esri help talks about the possibility of setting environment parameters for geoprocessing tools, but found nothing about ...
1
vote
0
answers
321
views
Arcpy multiprocessing arcpy.Select_analysis
I have the following script:
import arcpy
from arcpy import env
import datetime
import os
env.workspace = r"Q:\\SDE\Direct Connection to Contours.sde"
dataFC = arcpy.ListFeatureClasses("","","...
- 39
0
votes
1
answer
185
views
Clipping Raster Mosiac Simultaneously Within Separate ArcPy Scripts?
I have a large raster mosaic dataset that I would like to clip within several scripts simultaneously, but whenever I run two instances of a script simultaneously one always fails with the error:
...
5
votes
1
answer
766
views
Parallelize Extracting Large Raster Values (one band) to Points
I have sf data with single point geometries (points) and a 23,000 x 18,000 elevation RasterLayer with a single band, loaded from a geotiff (elev). They have the same CRS. I want to extract the ...
- 331
1
vote
0
answers
858
views
asyncio with multiprocessing
I am trying to run a multiprocessing on a zonal statistic in pure Python. I am new on it. I have to apply a zonal statistic with a shapefile with more than 6 million features on a taster of 6 GBs (...
- 573
1
vote
1
answer
447
views
Enable "Render layers in parallel using many CPU cores" from python console QGIS [closed]
How can I enable "Render layers in parallel using many CPU cores" option from QGIS python console?
- 123
9
votes
1
answer
5k
views
Multiprocessing with geopandas
I am learning how to implement the multiprocessing with spatial data using the module multiprocessing.
I am trying to implement a spatial intersection between a polygon file and a grid file.
I am ...
- 573
2
votes
1
answer
432
views
How to use multiprocessing with Make Route Analysis Layer tool
I am using ArcGIS 10.4.1 and Network Analyst extension. As an IDE i use Spyder and Python version 2.7. I created a script where there is a method with the following arguments.
def ra(EndPoint, ...
- 73
2
votes
1
answer
1k
views
Dividing a large raster into blocks for iteration in R
I have very big rasters for processing (#pixels >1,5*10^10) for this project I am working on. Simple functions such as finding the area of a pixels with a particular value, across the entire raster 1) ...
- 840
5
votes
1
answer
2k
views
Python rtree and parallelized code
I'm trying to parallelize with multiprocessing.Pool() some processes that involve queries of R-trees. In the non-parallel procedure, it works as expected.
But when I try to do it in parallel (even ...
- 1,100
2
votes
0
answers
156
views
Parallel optimization for query over LiDAR dataset
I'm using PostGIS + pointcloud to store a huge dataset of LiDAR point (2M patches of 600points)
I query this database to obtain the altimetic profile between two points.
At the moment I have a huge ...
3
votes
0
answers
2k
views
Spatial analysis and parallel processing in R
I am trying to do a point-in-poly analysis using R. I have 4.5 million points and 5k polygons. This is computationally intensive. I was pointed to parallel processing through another post and tried to ...
- 83
2
votes
0
answers
62
views
Multiprocessing GALayerToRasters
I'm new to ArcPy and am creating a script to automate some interpolation work.
I have multiple GA Layers outputted by an Empirical Bayesian Kriging interpolation (using arcpy....
- 141
1
vote
1
answer
153
views
Converting polylines to polygons in ArcGIS using multiprocessing
I am attempting to convert 15 feature classes containing roughly 70,000 contours each into polygons record by record. Using my current code, each feature class takes roughly 16 hours on my current ...
- 713
0
votes
0
answers
163
views
arcpy and multiprocessing: switch workflow to shapefiles to avoid schemalocks
I need to processe a large dataset through a bunch of interwoven scripts using arcpy. On a small test dataset my workflow functions fine so far but when it comes to the large dataset i see no other ...
- 1,158
1
vote
2
answers
791
views
Using MultiProcessing With Update Cursor?
I am trying to use multiprocessing to help me run a tool that will clip a feature and sum an attribute in it. The tool needs to iterate through about 24,000 rows of a featureclass while doing this. ...
- 58
5
votes
1
answer
3k
views
R - multicore approach to extract raster values using spatial points
It might look like this question is duplicated, but I'm asking about data extraction by points, rather than by polygons, and I couldn't find hints about point extraction. Therefore, please bare with ...
- 616
3
votes
1
answer
784
views
Does anyone already parallelized mosaic function for rasters in R?
Does anyone already paralleled mosaic function for rasters in R?
It seems that snow still doesn´t deal with this on an automatic way.
The idea is something like this:
r1<-stack("fit__test23KLR....
- 419
0
votes
0
answers
466
views
Multiprocessing Script Tool doesn't crash but will not function
I've made created a script that uses multiprocessing to break up a very large point dataset into pieces and extract values from 2 different rasters to them. When I tested it using the command line ...
- 89
9
votes
3
answers
6k
views
How parallelize the extract function for raster files in R?
I´m trying to parallelize a simple "extract" operation like this:
UseCores<-detectCores() -1
cl<- makeCluster(UseCores)
registerDoParallel(cl)
all.test.poly_period.test<-extract(stack.ts, ...
- 419
2
votes
1
answer
314
views
PyWPS with python multiprocessing
I have a python script that runs a set of queries in parallel using multiprocessing.pool and map_async.
The script from @Aaron works perfectly.
Now I want to integrate the same functions into a ...
- 73
4
votes
2
answers
5k
views
Processing multiple files simultanously using python
I have 11000 point shapefiles to run Near tool on against another point shapefile.
I am looking for a way to process 10 files at the same time.
Is there a way to process so many files using Python ...
- 221
18
votes
1
answer
10k
views
Does GDAL support parallel processing?
I would like to speed up the process corresponding to this command:
gdalwarp -t_srs EPSG:4326 -overwrite input.ntf output.tif
Is there a way to use parallel processing in GDAL?
Semi-offtopic: If not,...
- 441
7
votes
1
answer
2k
views
Multiprocessing in a QGIS Plugin
I have developed a QGIS plugin that processes an image using a bunch of other information, but it runs slowly. I would like to parallelize, as I can do with Multiprocessing.Process outside a QGIS ...
- 101
4
votes
2
answers
4k
views
How to achieve parallel Kriging in R to speed up the process?
I have tried the gstat package in R to perform bathymetry predictions based on Kriging interpolation methods. However, the process is very time consuming due the large amount of sampling data I'm ...
- 3,176
0
votes
2
answers
2k
views
Multiplication of two raster stack with R using multicore parallel processes
I am working with two global raster stacks, each containing 60 layers using the R Software. The layers are global raster files (in GTiff format) at a resolution of 5 min of arc. The first stack ...
- 51
2
votes
1
answer
937
views
Combining python (parallel) multiprocessing toolbox with arcpy
I'm trying to cut down on arcpy processing time by using the python multiprocessing toolbox. Using the example from http://wltrimbl.github.io/2014-06-10-spelman/intermediate/python/04-multiprocessing....
- 61
4
votes
2
answers
3k
views
Full utilization of CPU-cores with gdalwarp
I need to process many large GeoTiff files with gdalwarp.
For this I have used the following command:
gdalwarp -r cubic -s_srs EPSG:25832 -t_srs EPSG:3857 -cutline shapefile.shp -crop_to_cutline -...
user86086
4
votes
2
answers
5k
views
Unable to do a parallel INSERT using Postgres 9.6.0 & PostGIS 2.3.0
The following PostGIS query executes in parallel using Postgres 9.6's new parallel processing feature:
SELECT pnts.mb_code11, bdys.mb_code11
FROM testing.mb_random_points AS pnts
INNER JOIN testing....
- 767
2
votes
0
answers
2k
views
How to make a Python Multiprocessing Script work on Windows OS [closed]
I use the Python multiprocessing module to speed up a function (get_craters_for_buffering_BCC) in my script. In this function, I want to identify points (passed as a list: crater_features_list) within ...
- 285
2
votes
1
answer
638
views
Batch intersect and multiprocessing with arcpy
I admit what I am working on is way over my head, especially as I've only been dabbling in python for a few months. I am attempting to write a script that allows me to use Intersect analysis on about ...
- 23
1
vote
1
answer
5k
views
Using 'sp' method to subset SpatialPointsDataFrame with SpatialPolygon returns error in parLapply()
I have numerous very large data sets of ship tracking data (Automated Information System; AIS). These are originally in a file geodatabase and can be downloaded here (this is a smaller file for ...
- 597
7
votes
1
answer
882
views
Arcpy and multiprocessing not working
I want to utilize multiprocessing within my arcpy scripts but my first test has run into a problem which I don't understand. I'm running the script in the python window:
import arcpy
from ...
- 439
9
votes
1
answer
2k
views
Is it possible to do parallel processing in GDAL and QGIS?
It seems like processing a large raster dataset in QGIS can cause a crash, for example in QGIS failing to complete clip or intersect and crash dump in qgis. Is it possible to use GDAL parallel ...
- 1,046
2
votes
0
answers
85
views
Avoiding edges using GRASS generalization?
I'm trying to process the national wide land cover data into vector polygons.
In order to be quick I separate the data into several tiles and process them in parallel.
process flow is :
1. v.in.ogr
...
- 21
1
vote
1
answer
1k
views
R rasterEngine Parallel Processing, serialized and unserialized memory errors
I am trying to perform some raster math and wanted to try parallel processing in R using rasterEngine.
I am able to run the script for a few minutes and watch my CPU usage increase substantially, but ...
- 748
3
votes
1
answer
1k
views
How to use multiprocessing or pool to boost up the performance of ArcPy script
I created some arcpy script which actually runs slow on huge dataset. such as 100000 feature in it.
I read from below path that multiprocessing can boost up arcpy script script performance.
http://...
- 613
3
votes
0
answers
722
views
Multiprocessing in python qgis?
When i try to do mutliprocessing in python Qgis it doesn't work
from multiprocessing import Process,Lock
import multiprocessing
def name1(l,i):
print i
if __name__=='__main__':
lock=Lock()
...
- 313
6
votes
2
answers
2k
views
Python and QGIS multiprocessing documentation?
Where can we find better documention for python QGIS multiprocessing.
Especially regarding the multiprocessing of python and QGIS.
- 313
1
vote
2
answers
5k
views
Really slow extraction from raster even after using crop?
I have a large raster file (245295396) cells and stacks of rasters having 4 layers each which lie in the extent of this large raster. To start with I am trying to get value from one stack (3 channels) ...
- 113
3
votes
0
answers
907
views
Using Multiprocessing inside a plugin in Qgis [closed]
I spent quite a bit of time looking on how to use the multiprocessing package, but couldn't find anything on how to use it inside a plugin in QGIS.
I am developing a plugin that does some ...
- 419
6
votes
3
answers
668
views
How to run qgis scripts in parallel across remote server(s)
I have a stand alone python script using qgis that I wrote with the help of the SE question here
The script takes three arguments being; an input file path, an output file path, and another input ...
- 1,481
2
votes
1
answer
508
views
problem deleting shapefiles because still in use by arcpy cursor [duplicate]
I have a function outputTblContents which contains
def outputTblContents ( path ):
rows = arcpy.SearchCursor(path, fields="alpha; beta")
for row in rows:
print("{0} => {1}").format(...
- 145
1
vote
2
answers
774
views
Multiprocessing with arcpy and Solar Analyst (spatial analyst)
I need help getting multiprocessing to work with the spatial analyst tool - Solar Analyst.
I have tested the multiprocessing bit and it works fine. This sample is passing it a file containing 2 lines ...
- 157
4
votes
1
answer
4k
views
How to write to a single table via Python multiprocessing
A comment in the post Can multiprocessing with arcpy be run in a script tool? got me thinking, as I often need to do exactly this:
Just beware of deadlocking situations (two Insert cursors in the ...
- 21.2k
8
votes
1
answer
8k
views
Can multiprocessing with arcpy be run in a script tool?
I have been tinkering around with the multiprocessing module to resolve some processing of data in ArcMap 10.3.
I have created a script which works fine if I run it in IDLE. I see all my cores max ...
- 44.6k