Questions tagged [parallel-processing]

Use this tag when asking questions about multi or parallel processing - a programming paradigm used to speed up computationally intensive geoprocessing tasks by executing them over multiple CPU cores or threads.

20 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6
votes
0answers
1k views

Is it possible to do parallel processing in GDAL and QGIS?

It seems like processing a large raster dataset in QGIS can cause a crash, for example in QGIS failing to complete clip or intersect and crash dump in qgis. Is it possible to use GDAL parallel ...
3
votes
0answers
140 views

Running Standalone PyQGIS Script with Multiprocessing?

Is it possible to run a standalone PyQGIS script using multiprocessing for better performance? I'm creating 1,800 rendered map images and would like to run it across 4 processes to speed it up. If I ...
3
votes
0answers
776 views

Multiprocessing with geopandas

I am learning how to implement the multiprocessing with spatial data using the module multiprocessing. I am trying to implement a spatial intersection between a polygon file and a grid file. I am ...
3
votes
0answers
426 views

Multiprocessing in python qgis?

When i try to do mutliprocessing in python Qgis it doesn't work from multiprocessing import Process,Lock import multiprocessing def name1(l,i): print i if __name__=='__main__': lock=Lock() ...
2
votes
0answers
48 views

Optimising use of st_within for parallelisation

I'm trying to use st_within() to calculate points within polygons on a fairly complex shapefile. This operation tends to take 4-5 hours which is a bummer for debugging the code. I have access to a ...
2
votes
0answers
96 views

Parallel optimization for query over LiDAR dataset

I'm using PostGIS + pointcloud to store a huge dataset of LiDAR point (2M patches of 600points) I query this database to obtain the altimetic profile between two points. At the moment I have a huge ...
2
votes
0answers
513 views

Spatial analysis and parallel processing in R

I am trying to do a point-in-poly analysis using R. I have 4.5 million points and 5k polygons. This is computationally intensive. I was pointed to parallel processing through another post and tried to ...
2
votes
0answers
68 views

Avoiding edges using GRASS generalization?

I'm trying to process the national wide land cover data into vector polygons. In order to be quick I separate the data into several tiles and process them in parallel. process flow is : 1. v.in.ogr ...
1
vote
1answer
87 views

How can I optimize the processing time of the zonal_stats module using multiprocessing in Python?

I am carrying out a spectral feature extraction in Python 3 starting from an 8-band image and a shapefile (consisting of about a million polygons) superimposed on it. I calculate the mean and standard ...
1
vote
1answer
117 views

Multiprocess Spatial Join with Spatial Index

I'm a newbie in implementing GIS on Python, but the lack of any GIS software at the office has forced me to learn about it. I have a huge grid (a multipolygon) that I'm currently filling out with ...
1
vote
0answers
78 views

Arcpy multiprocessing arcpy.Select_analysis

I have the following script: import arcpy from arcpy import env import datetime import os env.workspace = r"Q:\\SDE\Direct Connection to Contours.sde" dataFC = arcpy.ListFeatureClasses("","","...
1
vote
0answers
402 views

asyncio with multiprocessing

I am trying to run a multiprocessing on a zonal statistic in pure Python. I am new on it. I have to apply a zonal statistic with a shapefile with more than 6 million features on a taster of 6 GBs (...
1
vote
0answers
25 views

Multiprocessing GALayerToRasters

I'm new to ArcPy and am creating a script to automate some interpolation work. I have multiple GA Layers outputted by an Empirical Bayesian Kriging interpolation (using arcpy....
1
vote
0answers
40 views

Sending commands into FGDB

I have a large .gdb of all blocks in the United States, available from Census. I would like to create a point shape file of all census tract centroids in the US. I have QGIS with FileGDB enabled, as ...
0
votes
0answers
30 views

Two task at the same time in Model Builder QGIS 3

I am new to using the graphic modeler and I do not know how to do certain actions. I have several treatments to perform on the basis of a buffer zone. Indeed, I have to polygonize 2 raster (which we ...
0
votes
0answers
59 views

QGIS parallel processes

I am processing a bunch of GPS data using a Qgis python script. The processing takes a LONG time to execute and when it is executing I notice that only a single CPU is pinned at 100%, the remaining ...
0
votes
0answers
66 views

Using doparallel package to use multiple core for changing resolution in R

I want to use an aggregate function for my raster file to change the resolution from 30 30 to 300 300. The top code(For loop) is working very well. I want to use multiple cores to use the aggregate ...
0
votes
0answers
96 views

arcpy and multiprocessing: switch workflow to shapefiles to avoid schemalocks

I need to processe a large dataset through a bunch of interwoven scripts using arcpy. On a small test dataset my workflow functions fine so far but when it comes to the large dataset i see no other ...
0
votes
0answers
310 views

Multiprocessing Script Tool doesn't crash but will not function

I've made created a script that uses multiprocessing to break up a very large point dataset into pieces and extract values from 2 different rasters to them. When I tested it using the command line ...
0
votes
0answers
177 views

A Distributed Cost-Distance Algorithm

I'm currently trying to implement a distributed cost-distance algorithm, however I can't seem to invent my own and still make it fast. I've been googling around a bit but haven't found anything that ...