Skip to main content

Questions tagged [parallel-processing]

Use this tag when asking questions about multi or parallel processing - a programming paradigm used to speed up computationally intensive geoprocessing tasks by executing them over multiple CPU cores or threads.

Filter by
Sorted by
Tagged with
0 votes
0 answers
69 views

ArcPy function produced None values in multiprocessing

I'm trying to get my script in multiprocessing, but the arcpy.management.CalculateGeometryAttributes produced None values. I tried it in for loop and it worked perfectly fine. import os import ...
Hieu Tran's user avatar
0 votes
0 answers
19 views

QGIS Server GetPrint multiprocessing documentation?

Where can we find better documention for QGIS Server GetPrint multiprocessing. Especially regarding the multiprocessing for generate multiple pdf at the same time. I am currently running a QGIS Server ...
Louis's user avatar
  • 15
0 votes
0 answers
27 views

ArcGIS 106385 "Workspace or data source is read only" when using multiprocess to intersect a featureclass on a network drive

I use arcpy.analysis.Intersect to bring data (specific to my study area) housed on a network drive to my local machine for analysis. Sometimes the study areas are quite large, and so I break them up ...
Matt's user avatar
  • 760
2 votes
1 answer
32 views

How to deal with concurrency when inserting polygons in a SQL table in a way that they don't intersect with each other

Problem: I have a table X, in which I am inserting polygons (ex: geometry data for a city). Before inserting them, I check if the input polygon intersects with any existing polygon in the table and ...
xorrax's user avatar
  • 21
0 votes
1 answer
84 views

GeoDataFrame.overlay parallelization with multiprocessing

I'm trying to speed up the calculation of the intersection between two geodataframes with multiprocessing. I'm using GeoPandas with JupyterLab from Anaconda on Linux. My multiprocessing code currently ...
banana store's user avatar
0 votes
0 answers
34 views

Optimizing batch image exports in Google Earth Engine

I'm creating a pipeline to train a machine learning model to create flooding probability maps at national scales using data in GEE via the Python API. For a given known flood event, I assemble data ...
Nissim Lebovits's user avatar
0 votes
1 answer
361 views

AttributeError: 'Dataset' object has no attribute 'rio' when using a LocalCluster from dask.distributed

I am trying to parallelize a function that uses a zarr datacube read from an s3 bucket to perform some calculations. However, when using a LocalCluster from dask.distributed, I lose the .rio accessor ...
LaserGlaciers's user avatar
0 votes
1 answer
29 views

Missing Outputs with ArcPy RasterToASCII and ThreadPoolExecutor

When I use ThreadPoolExecutor, I am missing a random number of expected outputs. I tried making an MRE to show what I am trying to do. The first MRE works but when I run a second MRE (which is closer ...
ENIAC-6's user avatar
  • 906
1 vote
1 answer
30 views

How to iterate through raster conversion concurrently while using map function in Python

I am trying to troubleshoot some issues I'm having with the Python map function that is part of a larger problem I'm having with trying to run my code concurrently. Why does map() here create an empty ...
ENIAC-6's user avatar
  • 906
0 votes
0 answers
240 views

Optimize the resampling and projection of multiple NetCDF files in R using terra

I have a set of 35 NetCDF files (continental scale), each containing 365 daily layers of climate data with a resolution of 5 km, s. I need to do two main tasks: Subset, s_sub, these files based on a ...
Sarash's user avatar
  • 45
0 votes
0 answers
104 views

Error while creating Worker Pools in GeoTIFF

I have to extract rasters from a large TIFF file (about 8MB). For this purpose I use GeoTIFF v.2.0.7 readRasters, but performance is unsatisfying: reading rasters took: 395143.48280000035 ms As a ...
Michał Dubrowski's user avatar
3 votes
2 answers
202 views

Accessing layers from QgsContext transferred by completed QgsTask using PyQGIS

We have a headless QGIS standalone script that needs to run some I/O intensive layer retrieval tasks, each as parallel background threads before doing further GIS processing. We are invoking our ...
eliangius's user avatar
  • 275
0 votes
0 answers
71 views

Multiprocessing passing a PySwig Object

I am trying to do some multiprocessing in Python using the PySAGA module. In order to avoid having to load the input raster for each worker I am trying to pickle the loaded raster to the ...
Hans Baumann's user avatar
1 vote
0 answers
152 views

Using ThreadPoolExectuor doesn't work with ArcPy

I have a Python function that downloads a geopackage from somewhere then does some stuff with it and at some point converts it to an ESRI file geodatabase. Now inside a python script I want to use ...
Mirco's user avatar
  • 11
3 votes
2 answers
591 views

Easy examples for multiprocessing in PyQGIS

I've gone through a large number of forum threads and cannot make sense of how possible it is to easily run two tools at the same time in PyQGIS. Basically what I'm wanting to do is: def function1(): ...
Buff Fox's user avatar
  • 109
1 vote
1 answer
134 views

Clipping gpkg's using parallel processing and R

I have two large gpkg datasets. I would like to clip one from the other and use parallel processing to speed up the process. However, I receive an error message. Code and error message shown below: #...
GISLad's user avatar
  • 11
0 votes
0 answers
160 views

How use multiprocessing with function arcpy.da.InsertCursor

I'm working with a table in the database that has more than 1 million records and a total load of this table takes about 8 hours because it has geographic data, the table is a featureclass. Basically, ...
Felipe L's user avatar
2 votes
1 answer
390 views

Concurrent requests while querying a map service returns duplicate sets

I have a Python script that hits a query endpoint for a map service on an unfederated server. I am avoiding the arcgis library because it is not performant enough and its compatibility with geopandas ...
jesnes's user avatar
  • 453
0 votes
0 answers
178 views

ArcGIS Pro - Route Solver Multiprocessing in Standalone Script

I am using ArcGIS Pro 2.9 with an advanced single use license and a local network dataset. I am attempting to write a standalone script that takes as its input an arbitrary set of origin-destination ...
DSG4's user avatar
  • 1
0 votes
1 answer
172 views

Remove ("erase") polygons by shared attribute

I need to do a pair-wise removal of a set of polygons within a larger set of polygons. pt<-data.frame("x"= c(0,0),"y"= c(0,5),pair= c("a","b")) pt<- ...
Wanner's user avatar
  • 9
0 votes
1 answer
84 views

Using a dynamic formula with parrallel kringing in R

I am trying to achieve parallel Kriging in R on several variables using a loop. Here is a reproducible example using data meuse and a code for parallel kriging that I found here. In the code below, ...
ePoQ's user avatar
  • 611
1 vote
1 answer
595 views

Parallelizing mask process using geopandas to cookie cut a rasterio object

I have a very large raster data and a shape file with 79,867 polygons. I want to efficiently extract statistics from the raster within each of the polygons. To do so, I have combined rasterio and ...
Julianno Sambatti's user avatar
3 votes
1 answer
832 views

Parallelize raster area calculation per class in R

I am trying to get crop area for admin units of whole African continent using the data available here at 30-m spatial resolution globally (raster value 1 = crops). I've tried both terra and raster ...
Badar Munir's user avatar
1 vote
1 answer
378 views

Processing time applying dask_geopandas. Dissolve a polygon layer with parallelization

I'm testing the dask_geopandas library to dissolve a polygon layer. However, at the moment the processing time applying the dissolution without parallelization (df) is lower than applying the supposed ...
José Manuel Izquierdo Segovia's user avatar
3 votes
1 answer
448 views

Concurrent ogr2ogr processing of GeoJSON using offset/limit

I have a line delimited GeoJSON file with around 600k features. Standard processing of it takes ~53 seconds: ogr2ogr -f "Esri Shapefile" ./out ./in.geojson Another approach I tried out was ...
Avocado's user avatar
  • 295
1 vote
1 answer
1k views

Multiprocessing clipping Python rasterio/rioxarray

Clipping a raster often takes a larger percentage of my computation time, therefore I want to try it in parallel. I tried multiprocessing with Python's multiprocessing or pathos. But I often either ...
Sebastian H's user avatar
1 vote
0 answers
141 views

Multithreading single function using ArcPy with ArcMap

I am using ArcMap 10.5. The problem I have a huge DEM raster (more than 6 GB in size). I need to process this raster through Hydrology toolset in ArcGIS (fill, flow direction, flow accumulation...etc)....
Yousif Almamalachy's user avatar
3 votes
0 answers
853 views

Parallelizing calculations on large SF object in Windows environment

I have a large SF object, containing 1 km x 1 km grid cells, where I need to aggregate the area of polygons within the grid cells, i.e. grouping by grid cell, but also grouping by a second variable, ...
Kuns's user avatar
  • 41
2 votes
0 answers
949 views

Why does ArcGIS Pro use only 5% of my CPU even though I activated parallel processing?

Whenever I'm trying to project a big raster it takes ages for ArcGIS Pro to finish the task. Therefore I did some research and found out that there is a Parallel Processing Factor which I can set. In ...
ulrichz94's user avatar
1 vote
1 answer
79 views

Multi-processing with network database and ArcMap

I am trying to take advantage of multiprocessing for what would otherwise be a very long run time for a loop execution. The basic idea is to solve for the optimal route between a given Origin and ...
ckitchens's user avatar
1 vote
1 answer
216 views

How to do non-blocking OWS requests in Python?

Is it possible to use OWSLib with aiohttp? Is there a Python library for formatting/parsing WPS (and other OGC web services, such as WMS - web map service) requests, that also supports IO coroutines (...
benjimin's user avatar
  • 476
0 votes
0 answers
108 views

Set a workspace inside a multiprocessing function inside a toolbox in ArcGIS Pro

I am trying to set a workspace inside a multiprocessing function of an ArcGIS Pro toolbox. It works perfectly in standalone script, but when I use it in a toolbox I get this error message: ...
Philippe Leblanc's user avatar
0 votes
0 answers
36 views

parallel changin region - WIND_OVERRIDE

INTRODUCTION: I have 5000 mammal species and I'm studying Extreme Climate Event impacts on them. I'm now analysing warm spell duration index (WSDI) which is defined as number of days in a year where ...
matteo s.'s user avatar
2 votes
1 answer
700 views

Creating indexes with PostGIS using all CPU cores

I have to run a whole bunch of GIST index creation, on some tables containing lots of records (scale of many countries, OSM data): CREATE INDEX CONCURRENTLY IF NOT EXISTS osm_footways_gist ON ...
swiss_knight's user avatar
  • 10.5k
2 votes
2 answers
2k views

Fastest way to perform focal operations using R

I know there are already some questions out there discussing this issue, for example: How parallelize the extract function for raster files in R? Increasing speed of crop, mask, & extract raster ...
Lenn's user avatar
  • 1,297
0 votes
1 answer
337 views

Big data and spatial regression: how to make spml more efficient in R?

I am running a spatial panel regression in R using spml from package splm. When I conduct this on about 1/200th of my data, the program executes. But when I try to do so on a third of my data, R uses ...
C. Ashley's user avatar
  • 405
2 votes
1 answer
162 views

Parallel process for spatial weights matrix production in R

I have an sf object pData that is rather large (300,000 polygons) and I am trying to produce spatial weights for each polygon: tilesNb=knn2nb(knearneigh(st_centroid(pData), k = 2)) tilesWeights=...
C. Ashley's user avatar
  • 405
1 vote
0 answers
205 views

Speed up zonal statistics with additional calculation

I have two rasters, one is a classified image and the other the corresponding probabilities for a correct classification. Now I need to calculate zonal statistics for polygon segments in the following ...
s6hebern's user avatar
  • 1,236
3 votes
1 answer
113 views

Use multiprocessing to speed up rebuild index in ArcPy

We are in ArcGIS 10.6.1 (on Windows Server 2012) and Enterprise GeoDatabase Oracle 19c (on Linux). I was wondering if it is feasible to use the Python's ProcessPoolExecutor to open two separate Oracle ...
user3481432's user avatar
0 votes
2 answers
163 views

Running parallel r.mapcalc in GRASS

I have thousands raster maps (3652, named raster_name.1, raster_name.2,... raster_name.x,... raster_name.3652) and for each map I want to do the r.mapcalc-threshold operation below here: ...
matteo s.'s user avatar
1 vote
0 answers
74 views

Can I make QGIS export one thing at the time?

While I am exporting a large amounts of TIFF files for my work, I am having a small bottleneck in my workflow. Up until now I have devided the TIFF images in smaller pieces so that my computer can ...
Nahuel's user avatar
  • 11
1 vote
0 answers
569 views

QGIS: How to speed up raster processing?

Is there any way to speed up raster processing in QGIS? I have seen discussion about speeding up rendering and querying, but not processing. I am using SAGA tools to calculate morphometrics and the ...
nateroe's user avatar
  • 253
1 vote
1 answer
239 views

Multiprocessing ERROR 000210 Cannot create output 'name' Failed to execute Intersect

I am trying to use multiprocessing to process intersects between polygons and points for a list of states and territories (the data sets for the points are large, so my hope was that the ...
gwydion93's user avatar
  • 1,913
3 votes
1 answer
1k views

How can I optimize the processing time of the zonal_stats module using multiprocessing in Python?

I am carrying out a spectral feature extraction in Python 3 starting from an 8-band image and a shapefile (consisting of about a million polygons) superimposed on it. I calculate the mean and standard ...
Jando's user avatar
  • 89
2 votes
0 answers
175 views

Two tasks at same time in QGIS Modeler

I am new to using the graphic modeler and I do not know how to do certain actions. I have several treatments to perform on the basis of a buffer zone. Indeed, I have to polygonize 2 raster (which we ...
ANG64's user avatar
  • 31
5 votes
0 answers
505 views

Optimising use of st_within for parallelisation

I'm trying to use st_within() to calculate points within polygons on a fairly complex shapefile. This operation tends to take 4-5 hours which is a bummer for debugging the code. I have access to a ...
Jeremy Kidwell's user avatar
7 votes
1 answer
2k views

Multi-threaded compression in rasterio?

For writing large raster files (e.g. satellite image GeoTIFFs) to disk, the throughput rate tends to be limited by the compression algorithm rather than by the raw disk IO. (Benchmarks: 1, 2.) This is ...
benjimin's user avatar
  • 476
1 vote
0 answers
539 views

QGIS parallel processes

I am processing a bunch of GPS data using a Qgis python script. The processing takes a LONG time to execute and when it is executing I notice that only a single CPU is pinned at 100%, the remaining ...
Eanema's user avatar
  • 51
7 votes
2 answers
2k views

Definitive list of QGIS multiprocessing tools?

This question is not as straightforward as it seems, but can be summed up with the following: Is there a definitive list of geoprocessing tools for QGIS that currently support multithreaded ...
anakaine's user avatar
  • 290
2 votes
1 answer
1k views

Parallel processing on rasterstack using Raster and Parallel Packages

I want to run BFAST on a raster stack (768 Landsat NDVI) in parallel processing. In order to illustrate the problem, I used the following reproducible code: Load necessary packages; library(raster)...
SA Khan's user avatar
  • 197