Questions tagged [parallel-processing]

Use this tag when asking questions about multi or parallel processing - a programming paradigm used to speed up computationally intensive geoprocessing tasks by executing them over multiple CPU cores or threads.

32 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
8 votes
1 answer
3k views

Multiprocessing with geopandas

I am learning how to implement the multiprocessing with spatial data using the module multiprocessing. I am trying to implement a spatial intersection between a polygon file and a grid file. I am ...
  • 563
3 votes
0 answers
372 views

Parallelizing calculations on large SF object in Windows environment

I have a large SF object, containing 1 km x 1 km grid cells, where I need to aggregate the area of polygons within the grid cells, i.e. grouping by grid cell, but also grouping by a second variable, ...
  • 41
3 votes
0 answers
256 views

Optimising use of st_within for parallelisation

I'm trying to use st_within() to calculate points within polygons on a fairly complex shapefile. This operation tends to take 4-5 hours which is a bummer for debugging the code. I have access to a ...
3 votes
0 answers
353 views

Running Standalone PyQGIS Script with Multiprocessing?

Is it possible to run a standalone PyQGIS script using multiprocessing for better performance? I'm creating 1,800 rendered map images and would like to run it across 4 processes to speed it up. If I ...
  • 739
3 votes
0 answers
1k views

Spatial analysis and parallel processing in R

I am trying to do a point-in-poly analysis using R. I have 4.5 million points and 5k polygons. This is computationally intensive. I was pointed to parallel processing through another post and tried to ...
3 votes
0 answers
618 views

Multiprocessing in python qgis?

When i try to do mutliprocessing in python Qgis it doesn't work from multiprocessing import Process,Lock import multiprocessing def name1(l,i): print i if __name__=='__main__': lock=Lock() ...
2 votes
0 answers
273 views

Why does ArcGIS Pro use only 5% of my CPU even though I activated parallel processing?

Whenever I'm trying to project a big raster it takes ages for ArcGIS Pro to finish the task. Therefore I did some research and found out that there is a Parallel Processing Factor which I can set. In ...
2 votes
0 answers
103 views

Two tasks at same time in QGIS Modeler

I am new to using the graphic modeler and I do not know how to do certain actions. I have several treatments to perform on the basis of a buffer zone. Indeed, I have to polygonize 2 raster (which we ...
  • 39
2 votes
0 answers
139 views

Parallel optimization for query over LiDAR dataset

I'm using PostGIS + pointcloud to store a huge dataset of LiDAR point (2M patches of 600points) I query this database to obtain the altimetic profile between two points. At the moment I have a huge ...
2 votes
0 answers
54 views

Multiprocessing GALayerToRasters

I'm new to ArcPy and am creating a script to automate some interpolation work. I have multiple GA Layers outputted by an Empirical Bayesian Kriging interpolation (using arcpy....
2 votes
0 answers
79 views

Avoiding edges using GRASS generalization?

I'm trying to process the national wide land cover data into vector polygons. In order to be quick I separate the data into several tiles and process them in parallel. process flow is : 1. v.in.ogr ...
  • 21
1 vote
1 answer
485 views

Multiprocessing clipping Python rasterio/rioxarray

Clipping a raster often takes a larger percentage of my computation time, therefore I want to try it in parallel. I tried multiprocessing with Python's multiprocessing or pathos. But I often either ...
1 vote
0 answers
82 views

Multithreading single function using ArcPy with ArcMap

I am using ArcMap 10.5. The problem I have a huge DEM raster (more than 6 GB in size). I need to process this raster through Hydrology toolset in ArcGIS (fill, flow direction, flow accumulation...etc)....
1 vote
1 answer
59 views

Multi-processing with network database and ArcMap

I am trying to take advantage of multiprocessing for what would otherwise be a very long run time for a loop execution. The basic idea is to solve for the optimal route between a given Origin and ...
1 vote
0 answers
81 views

How to do non-blocking OWS requests in Python?

Is it possible to use OWSLib with aiohttp? Is there a Python library for formatting/parsing WPS (and other OGC web services, such as WMS - web map service) requests, that also supports IO coroutines (...
  • 466
1 vote
0 answers
134 views

Speed up zonal statistics with additional calculation

I have two rasters, one is a classified image and the other the corresponding probabilities for a correct classification. Now I need to calculate zonal statistics for polygon segments in the following ...
  • 1,196
1 vote
0 answers
59 views

Can I make QGIS export one thing at the time?

While I am exporting a large amounts of TIFF files for my work, I am having a small bottleneck in my workflow. Up until now I have devided the TIFF images in smaller pieces so that my computer can ...
  • 11
1 vote
0 answers
382 views

QGIS parallel processes

I am processing a bunch of GPS data using a Qgis python script. The processing takes a LONG time to execute and when it is executing I notice that only a single CPU is pinned at 100%, the remaining ...
  • 51
1 vote
1 answer
344 views

Multiprocess Spatial Join with spatial index

I'm a newbie in implementing GIS on Python, but the lack of any GIS software at the office has forced me to learn about it. I have a huge grid (a multipolygon) that I'm currently filling out with ...
  • 33
1 vote
0 answers
208 views

Arcpy multiprocessing arcpy.Select_analysis

I have the following script: import arcpy from arcpy import env import datetime import os env.workspace = r"Q:\\SDE\Direct Connection to Contours.sde" dataFC = arcpy.ListFeatureClasses("","","...
  • 39
1 vote
0 answers
781 views

asyncio with multiprocessing

I am trying to run a multiprocessing on a zonal statistic in pure Python. I am new on it. I have to apply a zonal statistic with a shapefile with more than 6 million features on a taster of 6 GBs (...
  • 563
1 vote
0 answers
42 views

Sending commands into FGDB

I have a large .gdb of all blocks in the United States, available from Census. I would like to create a point shape file of all census tract centroids in the US. I have QGIS with FileGDB enabled, as ...
0 votes
0 answers
46 views

ArcGIS Pro - Route Solver Multiprocessing in Standalone Script

I am using ArcGIS Pro 2.9 with an advanced single use license and a local network dataset. I am attempting to write a standalone script that takes as its input an arbitrary set of origin-destination ...
  • 1
0 votes
1 answer
57 views

Remove ("erase") polygons by shared attribute

I need to do a pair-wise removal of a set of polygons within a larger set of polygons. pt<-data.frame("x"= c(0,0),"y"= c(0,5),pair= c("a","b")) pt<- ...
  • 9
0 votes
0 answers
151 views

Processing time applying dask_geopandas. Dissolve a polygon layer with parallelization

I'm testing the dask_geopandas library to dissolve a polygon layer. However, at the moment the processing time applying the dissolution without parallelization (df) is lower than applying the supposed ...
0 votes
0 answers
65 views

Set a workspace inside a multiprocessing function inside a toolbox in ArcGIS Pro

I am trying to set a workspace inside a multiprocessing function of an ArcGIS Pro toolbox. It works perfectly in standalone script, but when I use it in a toolbox I get this error message: ...
0 votes
0 answers
29 views

parallel changin region - WIND_OVERRIDE

INTRODUCTION: I have 5000 mammal species and I'm studying Extreme Climate Event impacts on them. I'm now analysing warm spell duration index (WSDI) which is defined as number of days in a year where ...
0 votes
0 answers
124 views

Big data and spatial regression: how to make spml more efficient in R?

I am running a spatial panel regression in R using spml from package splm. When I conduct this on about 1/200th of my data, the program executes. But when I try to do so on a third of my data, R uses ...
  • 375
0 votes
0 answers
298 views

QGIS: How to speed up raster processing?

Is there any way to speed up raster processing in QGIS? I have seen discussion about speeding up rendering and querying, but not processing. I am using SAGA tools to calculate morphometrics and the ...
  • 243
0 votes
0 answers
130 views

arcpy and multiprocessing: switch workflow to shapefiles to avoid schemalocks

I need to processe a large dataset through a bunch of interwoven scripts using arcpy. On a small test dataset my workflow functions fine so far but when it comes to the large dataset i see no other ...
  • 982
0 votes
0 answers
417 views

Multiprocessing Script Tool doesn't crash but will not function

I've made created a script that uses multiprocessing to break up a very large point dataset into pieces and extract values from 2 different rasters to them. When I tested it using the command line ...
0 votes
0 answers
199 views

A Distributed Cost-Distance Algorithm

I'm currently trying to implement a distributed cost-distance algorithm, however I can't seem to invent my own and still make it fast. I've been googling around a bit but haven't found anything that ...
  • 383