16
votes
Accepted
How parallelize the extract function for raster files in R?
You need to use the beginCluster and endCluster functions of the raster package. See the example below.
library(raster)
library(snow)
# Make test data
# RasterStack
r <- raster(ncol=36, nrow=18)
...
- 2,302
16
votes
Does GDAL support parallel processing?
Yes, GDAL supports parallel processing, and this support applies to gdalwarp by default.
Use the -multi option with gdalwarp to enable multithreading, as opposed to only multiple cores.
Details:
...
- 441
8
votes
Multi-threaded compression in rasterio?
Yes, although this seems not to be mentioned in the documentation.
If you include num_threads=8 or num_threads='all_cpus' as an argument to rasterio.open then multithreading will be enabled (for ...
- 486
6
votes
Definitive list of QGIS multiprocessing tools?
Thread safety
QGIS Processing algorithms can be thread-safe or not. According to the docs, the latter are :
[...] algorithms which manipulate the current project, layer selections, or with ...
- 37k
6
votes
Accepted
Unable to do a parallel INSERT using Postgres 9.6.0 & PostGIS 2.3.0
Writing data and reading data is a different story. Modifying data from different parallel resources always needs to be serialized and prepared for changes in source data while writing. It is ...
- 28.2k
6
votes
Accepted
R - multicore approach to extract raster values using spatial points
I ended up using an approach based on the snowfall package. It is quite simple, works really good and the point extraction function is as fast as the number of cores that you can use. The approach I ...
- 616
5
votes
Python rtree and parallelized code
The problem is that rtree is a ctypes wrapper around libspatialindex, and pickle has no way of storing the pointer to the rtree in memory.
You could instead store the index to disk:
from ...
- 4,127
5
votes
Accepted
Full utilization of CPU-cores with gdalwarp
If you are using a Linux OS, you could consider using GNU parallel to process multiple files in parallel.
For instance, a simple example using gdalinfo in parallel is:
cat list_of_images.txt | ...
- 4,912
5
votes
How to achieve parallel Kriging in R to speed up the process?
This is a simple and reproducible example with meuse dataset of gstat package based on @Spacedman answer and using the parallel package:
More info and help here: Parallelizing and clustering in R
# ...
- 3,176
5
votes
Accepted
Concurrent ogr2ogr processing of GeoJSON using offset/limit
You can't skip ahead in a GeoJSON file so each thread has to read up to the point you've asked it to start processing, discarding the features as it goes. This is one of the things that makes GeoJSON ...
- 83.2k
5
votes
Parallelize raster area calculation per class in R
You could write a function like this
fun <- function(rastfile, vectfile) {
library(terra)
r <- rast(rastfile)
s <- vect(vectfile)
rs <- crop(r, s, mask = TRUE)
if (...
- 11.1k
4
votes
Accepted
How to achieve parallel Kriging in R to speed up the process?
Once you've got a variogram fitted then kriging is trivially parallelizable. Kriging predictions are independent of each other.
So, divide your prediction points (grid) into N sets, where N is your ...
- 66.5k
4
votes
Processing multiple files simultanously using python
If you are restricted to windows you can write a DOS sript and run it in parallel as decribed parallel execution of shell processes
If can run in linux there is a procedure pretty well worked out ...
- 1,481
4
votes
Accepted
Enable "Render layers in parallel using many CPU cores" from python console QGIS
Use this:
from PyQt5.QtCore import QSettings, QThread
# parallel rendering
QSettings().setValue("/qgis/parallel_rendering", True)
for use max cores
from qgis.core import QgsApplication
threadcount =...
- 7,908
4
votes
Multiprocessing with geopandas
Two things that need to be addressed first:
the title of your question is far too broad and misleading with respect to your actual problem.
if you have spatial operations such as spatial joins, or in ...
4
votes
Fastest way to perform focal operations using R
Following my comment I will try to explain the steps of using tabularaster to speed up the extraction process.
First let's get ourselves a vector data object and a raster:
library(raster)
library(...
- 550
4
votes
Parallelizing raster file generation in ArcGIS Pro
If you have set it to say 100% then ONLY the tools that honour it will parallelize their operation. It does not parallelize python. You don't really say HOW your rasters are created, so for example if ...
- 44.6k
3
votes
Accepted
Using MultiProcessing With Update Cursor?
My answer is not about resolving your multiprocessor issue but the general approach to count overlapping parts. I advised a method using spatial join in this answer to How to remove duplicate features ...
- 5,013
3
votes
How parallelize the extract function for raster files in R?
Given that it's .tif we now need to know if it's tiled. It probably is, and raster extract is very slow in this situation (and is effectively in an un-maintained state with no known prospect of ...
- 8,256
3
votes
Multiprocessing in a QGIS Plugin
Thanks for the suggestion Gustry, I look forward to trying it out when QGIS 3 comes alive. In the meantime I tried different things, and the only way I could get my plugin to parallelize and speed up ...
- 101
3
votes
Use multiprocessing to speed up rebuild index in ArcPy
Index construction is an I/O-intensive task, not generally CPU-bound. It's also likely to involve locks on table extents, and is therefore not amenable to multiprocessing (INDEX requests to a single ...
- 20.3k
3
votes
Fastest way to perform focal operations using R
Let's benchmark some different approaches.
Here is some example data. Please note that we explode the MULTIPOLYGON geometry to simplify matters a bit.
library(sf)
library(raster)
library(terra)
...
- 32k
3
votes
Creating indexes with PostGIS using all CPU cores
GIST index building cannot utilize multiple cores yet.
PostgreSQL can build indexes while leveraging multiple CPUs in order to process the table rows faster. This feature is known as parallel index ...
- 131
3
votes
Clipping gpkg's using parallel processing and R
future_sapply works along columns of the data, not rows. Here's using a sample data set generated from example(st_read); gpkg_source=nc. If I print the function arg in the loop I can see its the ...
- 66.5k
3
votes
Accepted
Easy examples for multiprocessing in PyQGIS
I strongly recommend you follow PyQGIS Developer Cookbook | 15. Tasks - doing heavy work in the background to create your tasks but below is the explanation of your error.
In Python functions are ...
- 11.3k
3
votes
Accepted
Missing Outputs with ArcPy RasterToASCII and ThreadPoolExecutor
Multithreading and ArcPy is a topic that comes up from time to time, usually on Esri Community. I believe part of my answer from Does arcpy support multithreading? - Esri Community a few years ago is ...
- 2,449
2
votes
Accepted
Is it possible to do parallel processing in GDAL and QGIS?
GDAL can use multiple threads for some processing. For example gdal2tiles https://gdal.org/programs/gdal2tiles.html can create base tiles in multiple processes. Gdalwarp https://gdal.org/programs/...
- 69k
2
votes
Processing multiple files simultanously using python
Esri has a useful blog post on utilizing Python's multiprocessing module. In many cases the multiprocessing module works fine with Esri's arcpy site package, mostly for embarrassingly parallel ...
- 51.8k
2
votes
Accepted
Does anyone already parallelized mosaic function for rasters in R?
I found a reasonable solution using the gdalUtils package:
library(gdalUtils)
gdalbuildvrt("fit__test23KLR.img", "r1.vrt", sd=1)
gdalbuildvrt("fit__test23KKR.img", "r2.vrt", sd=1)
gdalbuildvrt("...
- 419
2
votes
Accepted
How to use multiprocessing with Make Route Analysis Layer tool
Now I can see your worker function, one immediate issue that leaps out is that your layer names are not unique and this will upset things, so this line needs to be adjusted
arcpy....
- 44.6k
Only top scored, non community-wiki answers of a minimum length are eligible