Questions tagged [parallel-processing]

Use this tag when asking questions about multi or parallel processing - a programming paradigm used to speed up computationally intensive geoprocessing tasks by executing them over multiple CPU cores or threads.

Filter by
Sorted by
Tagged with
0 votes
1 answer
45 views

Remove ("erase") polygons by shared attribute

I need to do a pair-wise removal of a set of polygons within a larger set of polygons. pt<-data.frame("x"= c(0,0),"y"= c(0,5),pair= c("a","b")) pt<- ...
  • 9
0 votes
1 answer
57 views

Using a dynamic formula with parrallel kringing in R

I am trying to achieve parallel Kriging in R on several variables using a loop. Here is a reproducible example using data meuse and a code for parallel kriging that I found here. In the code below, ...
  • 561
1 vote
1 answer
147 views

Parallelizing mask process using geopandas to cookie cut a rasterio object

I have a very large raster data and a shape file with 79,867 polygons. I want to efficiently extract statistics from the raster within each of the polygons. To do so, I have combined rasterio and ...
3 votes
1 answer
244 views

Parallelize raster area calculation per class in R

I am trying to get crop area for admin units of whole African continent using the data available here at 30-m spatial resolution globally (raster value 1 = crops). I've tried both terra and raster ...
0 votes
0 answers
124 views

Processing time applying dask_geopandas. Dissolve a polygon layer with parallelization

I'm testing the dask_geopandas library to dissolve a polygon layer. However, at the moment the processing time applying the dissolution without parallelization (df) is lower than applying the supposed ...
3 votes
1 answer
175 views

Concurrent ogr2ogr processing of GeoJSON using offset/limit

I have a line delimited GeoJSON file with around 600k features. Standard processing of it takes ~53 seconds: ogr2ogr -f "Esri Shapefile" ./out ./in.geojson Another approach I tried out was ...
  • 275
0 votes
1 answer
416 views

Multiprocessing clipping Python rasterio/rioxarray

Clipping a raster often takes a larger percentage of my computation time, therefore I want to try it in parallel. I tried multiprocessing with Python's multiprocessing or pathos. But I often either ...
1 vote
0 answers
69 views

Multithreading single function using ArcPy with ArcMap

I am using ArcMap 10.5. The problem I have a huge DEM raster (more than 6 GB in size). I need to process this raster through Hydrology toolset in ArcGIS (fill, flow direction, flow accumulation...etc)....
3 votes
0 answers
311 views

Parallelizing calculations on large SF object in Windows environment

I have a large SF object, containing 1 km x 1 km grid cells, where I need to aggregate the area of polygons within the grid cells, i.e. grouping by grid cell, but also grouping by a second variable, ...
  • 41
2 votes
0 answers
195 views

Why does ArcGIS Pro use only 5% of my CPU even though I activated parallel processing?

Whenever I'm trying to project a big raster it takes ages for ArcGIS Pro to finish the task. Therefore I did some research and found out that there is a Parallel Processing Factor which I can set. In ...
1 vote
1 answer
59 views

Multi-processing with network database and ArcMap

I am trying to take advantage of multiprocessing for what would otherwise be a very long run time for a loop execution. The basic idea is to solve for the optimal route between a given Origin and ...
1 vote
0 answers
76 views

How to do non-blocking OWS requests in Python?

Is it possible to use OWSLib with aiohttp? Is there a Python library for formatting/parsing WPS (and other OGC web services, such as WMS - web map service) requests, that also supports IO coroutines (...
  • 466
0 votes
0 answers
53 views

Set a workspace inside a multiprocessing function inside a toolbox in ArcGIS Pro

I am trying to set a workspace inside a multiprocessing function of an ArcGIS Pro toolbox. It works perfectly in standalone script, but when I use it in a toolbox I get this error message: ...
0 votes
0 answers
29 views

parallel changin region - WIND_OVERRIDE

INTRODUCTION: I have 5000 mammal species and I'm studying Extreme Climate Event impacts on them. I'm now analysing warm spell duration index (WSDI) which is defined as number of days in a year where ...
2 votes
1 answer
354 views

Creating indexes with PostGIS using all CPU cores

I have to run a whole bunch of GIST index creation, on some tables containing lots of records (scale of many countries, OSM data): CREATE INDEX CONCURRENTLY IF NOT EXISTS osm_footways_gist ON ...
  • 7,582
1 vote
2 answers
1k views

Fastest way to perform focal operations using R

I know there are already some questions out there discussing this issue, for example: How parallelize the extract function for raster files in R? Increasing speed of crop, mask, & extract raster ...
  • 767
0 votes
0 answers
117 views

Big data and spatial regression: how to make spml more efficient in R?

I am running a spatial panel regression in R using spml from package splm. When I conduct this on about 1/200th of my data, the program executes. But when I try to do so on a third of my data, R uses ...
  • 353
2 votes
1 answer
97 views

Parallel process for spatial weights matrix production in R

I have an sf object pData that is rather large (300,000 polygons) and I am trying to produce spatial weights for each polygon: tilesNb=knn2nb(knearneigh(st_centroid(pData), k = 2)) tilesWeights=...
  • 353
1 vote
0 answers
123 views

Speed up zonal statistics with additional calculation

I have two rasters, one is a classified image and the other the corresponding probabilities for a correct classification. Now I need to calculate zonal statistics for polygon segments in the following ...
  • 1,196
3 votes
1 answer
67 views

Use multiprocessing to speed up rebuild index in ArcPy

We are in ArcGIS 10.6.1 (on Windows Server 2012) and Enterprise GeoDatabase Oracle 19c (on Linux). I was wondering if it is feasible to use the Python's ProcessPoolExecutor to open two separate Oracle ...
0 votes
2 answers
115 views

Running parallel r.mapcalc in GRASS

I have thousands raster maps (3652, named raster_name.1, raster_name.2,... raster_name.x,... raster_name.3652) and for each map I want to do the r.mapcalc-threshold operation below here: ...
1 vote
0 answers
55 views

Can I make QGIS export one thing at the time?

While I am exporting a large amounts of TIFF files for my work, I am having a small bottleneck in my workflow. Up until now I have devided the TIFF images in smaller pieces so that my computer can ...
  • 11
0 votes
0 answers
269 views

QGIS: How to speed up raster processing?

Is there any way to speed up raster processing in QGIS? I have seen discussion about speeding up rendering and querying, but not processing. I am using SAGA tools to calculate morphometrics and the ...
  • 233
1 vote
1 answer
155 views

Multiprocessing ERROR 000210 Cannot create output 'name' Failed to execute Intersect

I am trying to use multiprocessing to process intersects between polygons and points for a list of states and territories (the data sets for the points are large, so my hope was that the ...
  • 1,781
3 votes
1 answer
695 views

How can I optimize the processing time of the zonal_stats module using multiprocessing in Python?

I am carrying out a spectral feature extraction in Python 3 starting from an 8-band image and a shapefile (consisting of about a million polygons) superimposed on it. I calculate the mean and standard ...
  • 71
2 votes
0 answers
100 views

Two tasks at same time in QGIS Modeler

I am new to using the graphic modeler and I do not know how to do certain actions. I have several treatments to perform on the basis of a buffer zone. Indeed, I have to polygonize 2 raster (which we ...
  • 39
3 votes
0 answers
237 views

Optimising use of st_within for parallelisation

I'm trying to use st_within() to calculate points within polygons on a fairly complex shapefile. This operation tends to take 4-5 hours which is a bummer for debugging the code. I have access to a ...
7 votes
1 answer
1k views

Multi-threaded compression in rasterio?

For writing large raster files (e.g. satellite image GeoTIFFs) to disk, the throughput rate tends to be limited by the compression algorithm rather than by the raw disk IO. (Benchmarks: 1, 2.) This is ...
  • 466
1 vote
0 answers
364 views

QGIS parallel processes

I am processing a bunch of GPS data using a Qgis python script. The processing takes a LONG time to execute and when it is executing I notice that only a single CPU is pinned at 100%, the remaining ...
  • 51
5 votes
2 answers
1k views

Definitive list of QGIS multiprocessing tools?

This question is not as straightforward as it seems, but can be summed up with the following: Is there a definitive list of geoprocessing tools for QGIS that currently support multithreaded ...
  • 240
2 votes
1 answer
759 views

Parallel processing on rasterstack using Raster and Parallel Packages

I want to run BFAST on a raster stack (768 Landsat NDVI) in parallel processing. In order to illustrate the problem, I used the following reproducible code: Load necessary packages; library(raster)...
  • 197
1 vote
1 answer
2k views

Parallel processing in Pandas

Using this implementation of parallelization raises an ImportError: cannot import name 'Parallel' from 'multiprocessing' The following code tries parallelization with the "denominator" function and ...
5 votes
1 answer
504 views

Parallel processing in QGIS3 on Windows system

I am working on a big vector data set with more than 2.4 million features containing 10x10m squares. I wrote a small script in Python to delete uninteresting fields and perform calculations on the ...
3 votes
0 answers
336 views

Running Standalone PyQGIS Script with Multiprocessing?

Is it possible to run a standalone PyQGIS script using multiprocessing for better performance? I'm creating 1,800 rendered map images and would like to run it across 4 processes to speed it up. If I ...
  • 739
1 vote
1 answer
333 views

Multiprocess Spatial Join with spatial index

I'm a newbie in implementing GIS on Python, but the lack of any GIS software at the office has forced me to learn about it. I have a huge grid (a multipolygon) that I'm currently filling out with ...
  • 33
0 votes
1 answer
129 views

Using all processor cores when exporting map to TIFF using ArcGIS Desktop?

By default, ArcGIS Desktop uses only one processor core when you export a map. Esri help talks about the possibility of setting environment parameters for geoprocessing tools, but found nothing about ...
1 vote
0 answers
197 views

Arcpy multiprocessing arcpy.Select_analysis

I have the following script: import arcpy from arcpy import env import datetime import os env.workspace = r"Q:\\SDE\Direct Connection to Contours.sde" dataFC = arcpy.ListFeatureClasses("","","...
  • 39
0 votes
1 answer
144 views

Clipping Raster Mosiac Simultaneously Within Separate ArcPy Scripts?

I have a large raster mosaic dataset that I would like to clip within several scripts simultaneously, but whenever I run two instances of a script simultaneously one always fails with the error: ...
5 votes
1 answer
676 views

Parallelize Extracting Large Raster Values (one band) to Points

I have sf data with single point geometries (points) and a 23,000 x 18,000 elevation RasterLayer with a single band, loaded from a geotiff (elev). They have the same CRS. I want to extract the ...
  • 331
1 vote
0 answers
751 views

asyncio with multiprocessing

I am trying to run a multiprocessing on a zonal statistic in pure Python. I am new on it. I have to apply a zonal statistic with a shapefile with more than 6 million features on a taster of 6 GBs (...
  • 553
1 vote
1 answer
300 views

Enable "Render layers in parallel using many CPU cores" from python console QGIS [closed]

How can I enable "Render layers in parallel using many CPU cores" option from QGIS python console?
7 votes
1 answer
3k views

Multiprocessing with geopandas

I am learning how to implement the multiprocessing with spatial data using the module multiprocessing. I am trying to implement a spatial intersection between a polygon file and a grid file. I am ...
  • 553
2 votes
1 answer
352 views

How to use multiprocessing with Make Route Analysis Layer tool

I am using ArcGIS 10.4.1 and Network Analyst extension. As an IDE i use Spyder and Python version 2.7. I created a script where there is a method with the following arguments. def ra(EndPoint, ...
2 votes
1 answer
858 views

Dividing a large raster into blocks for iteration in R

I have very big rasters for processing (#pixels >1,5*10^10) for this project I am working on. Simple functions such as finding the area of a pixels with a particular value, across the entire raster 1) ...
  • 790
5 votes
1 answer
1k views

Python rtree and parallelized code

I'm trying to parallelize with multiprocessing.Pool() some processes that involve queries of R-trees. In the non-parallel procedure, it works as expected. But when I try to do it in parallel (even ...
  • 1,053
2 votes
0 answers
139 views

Parallel optimization for query over LiDAR dataset

I'm using PostGIS + pointcloud to store a huge dataset of LiDAR point (2M patches of 600points) I query this database to obtain the altimetic profile between two points. At the moment I have a huge ...
3 votes
0 answers
1k views

Spatial analysis and parallel processing in R

I am trying to do a point-in-poly analysis using R. I have 4.5 million points and 5k polygons. This is computationally intensive. I was pointed to parallel processing through another post and tried to ...
2 votes
0 answers
52 views

Multiprocessing GALayerToRasters

I'm new to ArcPy and am creating a script to automate some interpolation work. I have multiple GA Layers outputted by an Empirical Bayesian Kriging interpolation (using arcpy....
1 vote
1 answer
105 views

Converting polylines to polygons in ArcGIS using multiprocessing

I am attempting to convert 15 feature classes containing roughly 70,000 contours each into polygons record by record. Using my current code, each feature class takes roughly 16 hours on my current ...
  • 713
0 votes
0 answers
127 views

arcpy and multiprocessing: switch workflow to shapefiles to avoid schemalocks

I need to processe a large dataset through a bunch of interwoven scripts using arcpy. On a small test dataset my workflow functions fine so far but when it comes to the large dataset i see no other ...
  • 984