Questions tagged [parallel-processing]
Use this tag when asking questions about multi or parallel processing - a programming paradigm used to speed up computationally intensive geoprocessing tasks by executing them over multiple CPU cores or threads.
97
questions
1
vote
1answer
178 views
Multiprocess Spatial Join with Spatial Index
I'm a newbie in implementing GIS on Python, but the lack of any GIS software at the office has forced me to learn about it.
I have a huge grid (a multipolygon) that I'm currently filling out with ...
1
vote
0answers
44 views
Creating indexes with PostGIS using all CPU cores
I have to run a whole bunch of GIST index creation, on some tables containing lots of records (scale of many countries, OSM data):
CREATE INDEX CONCURRENTLY IF NOT EXISTS osm_footways_gist ON ...
0
votes
2answers
65 views
Fastest way to perform focal operations using R
I know there are already some questions out there discussing this issue, for example:
How parallelize the extract function for raster files in R?
Increasing speed of crop, mask, & extract raster ...
0
votes
2answers
43 views
Running parallel r.mapcalc in GRASS
I have thousands raster maps (3652, named raster_name.1, raster_name.2,... raster_name.x,... raster_name.3652) and for each map I want to do the r.mapcalc-threshold operation below here:
...
0
votes
0answers
30 views
Big data and spatial regression: how to make spml more efficient in R?
I am running a spatial panel regression in R using spml from package splm. When I conduct this on about 1/200th of my data, the program executes. But when I try to do so on a third of my data, R uses ...
1
vote
0answers
37 views
Speed up zonal statistics with additional calculation
I have two rasters, one is a classified image and the other the corresponding probabilities for a correct classification. Now I need to calculate zonal statistics for polygon segments in the following ...
2
votes
1answer
27 views
Parallel process for spatial weights matrix production in R
I have an sf object pData that is rather large (300,000 polygons) and I am trying to produce spatial weights for each polygon:
tilesNb=knn2nb(knearneigh(st_centroid(pData), k = 2))
tilesWeights=...
1
vote
0answers
47 views
Two tasks at same time in QGIS Modeler
I am new to using the graphic modeler and I do not know how to do certain actions.
I have several treatments to perform on the basis of a buffer zone. Indeed, I have to polygonize 2 raster (which we ...
3
votes
1answer
38 views
Use multiprocessing to speed up rebuild index in ArcPy
We are in ArcGIS 10.6.1 (on Windows Server 2012) and Enterprise GeoDatabase Oracle 19c (on Linux).
I was wondering if it is feasible to use the Python's ProcessPoolExecutor to open two separate Oracle ...
0
votes
0answers
25 views
Error Scipy in QGIS SCP plugin
After clipping I am receiving these errors. I was clipping some satellite images from Sentinel 2 via SCP plugin but QGIS is not processing it. I already posted my problem with Scipy where in QGIS 3.8 ...
1
vote
1answer
197 views
How can I optimize the processing time of the zonal_stats module using multiprocessing in Python?
I am carrying out a spectral feature extraction in Python 3 starting from an 8-band image and a shapefile (consisting of about a million polygons) superimposed on it. I calculate the mean and standard ...
1
vote
0answers
38 views
Can I make QGIS export one thing at the time?
While I am exporting a large amounts of TIFF files for my work, I am having a small bottleneck in my workflow. Up until now I have devided the TIFF images in smaller pieces so that my computer can ...
4
votes
1answer
4k views
Unable to do a parallel INSERT using Postgres 9.6.0 & PostGIS 2.3.0
The following PostGIS query executes in parallel using Postgres 9.6's new parallel processing feature:
SELECT pnts.mb_code11, bdys.mb_code11
FROM testing.mb_random_points AS pnts
INNER JOIN testing....
0
votes
0answers
74 views
QGIS: How to speed up raster processing?
Is there any way to speed up raster processing in QGIS? I have seen discussion about speeding up rendering and querying, but not processing.
I am using SAGA tools to calculate morphometrics and the ...
1
vote
1answer
46 views
Multiprocessing ERROR 000210 Cannot create output 'name' Failed to execute Intersect
I am trying to use multiprocessing to process intersects between polygons and points for a list of states and territories (the data sets for the points are large, so my hope was that the ...
2
votes
0answers
55 views
Optimising use of st_within for parallelisation
I'm trying to use st_within() to calculate points within polygons on a fairly complex shapefile. This operation tends to take 4-5 hours which is a bummer for debugging the code. I have access to a ...
6
votes
1answer
439 views
Multi-threaded compression in rasterio?
For writing large raster files (e.g. satellite image GeoTIFFs) to disk, the throughput rate tends to be limited by the compression algorithm rather than by the raw disk IO. (Benchmarks: 1, 2.) This is ...
1
vote
0answers
172 views
QGIS parallel processes
I am processing a bunch of GPS data using a Qgis python script. The processing takes a LONG time to execute and when it is executing I notice that only a single CPU is pinned at 100%, the remaining ...
2
votes
1answer
185 views
Parallel processing on rasterstack using Raster and Parallel Packages
I want to run BFAST on a raster stack (768 Landsat NDVI) in parallel processing. In order to illustrate the problem, I used the following reproducible code:
Load necessary packages;
library(raster)...
5
votes
2answers
438 views
Definitive list of QGIS multiprocessing tools?
This question is not as straightforward as it seems, but can be summed up with the following:
Is there a definitive list of geoprocessing tools for QGIS that currently support multithreaded ...
8
votes
3answers
4k views
How parallelize the extract function for raster files in R?
I´m trying to parallelize a simple "extract" operation like this:
UseCores<-detectCores() -1
cl<- makeCluster(UseCores)
registerDoParallel(cl)
all.test.poly_period.test<-extract(stack.ts, ...
5
votes
1answer
181 views
Parallel processing in QGIS3 on Windows system
I am working on a big vector data set with more than 2.4 million features containing 10x10m squares. I wrote a small script in Python to delete uninteresting fields and perform calculations on the ...
0
votes
1answer
1k views
Parallel processing in Pandas
Using this implementation of parallelization raises an ImportError: cannot import name 'Parallel' from 'multiprocessing' The following code tries parallelization with the "denominator" function and ...
3
votes
0answers
202 views
Running Standalone PyQGIS Script with Multiprocessing?
Is it possible to run a standalone PyQGIS script using multiprocessing for better performance?
I'm creating 1,800 rendered map images and would like to run it across 4 processes to speed it up.
If I ...
2
votes
1answer
523 views
Dividing a large raster into blocks for iteration in R
I have very big rasters for processing (#pixels >1,5*10^10) for this project I am working on. Simple functions such as finding the area of a pixels with a particular value, across the entire raster 1) ...
1
vote
1answer
79 views
Using all processor cores when exporting map to TIFF using ArcGIS Desktop?
By default, ArcGIS Desktop uses only one processor core when you export a map. Esri help talks about the possibility of setting environment parameters for geoprocessing tools, but found nothing about ...
0
votes
0answers
182 views
A Distributed Cost-Distance Algorithm
I'm currently trying to implement a distributed cost-distance algorithm, however I can't seem to invent my own and still make it fast. I've been googling around a bit but haven't found anything that ...
1
vote
0answers
98 views
Arcpy multiprocessing arcpy.Select_analysis
I have the following script:
import arcpy
from arcpy import env
import datetime
import os
env.workspace = r"Q:\\SDE\Direct Connection to Contours.sde"
dataFC = arcpy.ListFeatureClasses("","","...
0
votes
1answer
97 views
Clipping Raster Mosiac Simultaneously Within Separate ArcPy Scripts?
I have a large raster mosaic dataset that I would like to clip within several scripts simultaneously, but whenever I run two instances of a script simultaneously one always fails with the error:
...
5
votes
1answer
390 views
Parallelize Extracting Large Raster Values (one band) to Points
I have sf data with single point geometries (points) and a 23,000 x 18,000 elevation RasterLayer with a single band, loaded from a geotiff (elev). They have the same CRS. I want to extract the ...
5
votes
1answer
2k views
R - multicore approach to extract raster values using spatial points
It might look like this question is duplicated, but I'm asking about data extraction by points, rather than by polygons, and I couldn't find hints about point extraction. Therefore, please bare with ...
1
vote
0answers
518 views
asyncio with multiprocessing
I am trying to run a multiprocessing on a zonal statistic in pure Python. I am new on it. I have to apply a zonal statistic with a shapefile with more than 6 million features on a taster of 6 GBs (...
1
vote
1answer
171 views
Enable “Render layers in parallel using many CPU cores” from python console QGIS [closed]
How can I enable "Render layers in parallel using many CPU cores" option from QGIS python console?
4
votes
0answers
1k views
Multiprocessing with geopandas
I am learning how to implement the multiprocessing with spatial data using the module multiprocessing.
I am trying to implement a spatial intersection between a polygon file and a grid file.
I am ...
2
votes
1answer
197 views
How to use multiprocessing with Make Route Analysis Layer tool
I am using ArcGIS 10.4.1 and Network Analyst extension. As an IDE i use Spyder and Python version 2.7. I created a script where there is a method with the following arguments.
def ra(EndPoint, ...
4
votes
1answer
602 views
Python rtree and parallelized code
I'm trying to parallelize with multiprocessing.Pool() some processes that involve queries of R-trees. In the non-parallel procedure, it works as expected.
But when I try to do it in parallel (even ...
2
votes
0answers
113 views
Parallel optimization for query over LiDAR dataset
I'm using PostGIS + pointcloud to store a huge dataset of LiDAR point (2M patches of 600points)
I query this database to obtain the altimetic profile between two points.
At the moment I have a huge ...
2
votes
0answers
697 views
Spatial analysis and parallel processing in R
I am trying to do a point-in-poly analysis using R. I have 4.5 million points and 5k polygons. This is computationally intensive. I was pointed to parallel processing through another post and tried to ...
1
vote
0answers
31 views
Multiprocessing GALayerToRasters
I'm new to ArcPy and am creating a script to automate some interpolation work.
I have multiple GA Layers outputted by an Empirical Bayesian Kriging interpolation (using arcpy....
2
votes
1answer
237 views
PyWPS with python multiprocessing
I have a python script that runs a set of queries in parallel using multiprocessing.pool and map_async.
The script from @Aaron works perfectly.
Now I want to integrate the same functions into a ...
1
vote
1answer
76 views
Converting polylines to polygons in ArcGIS using multiprocessing
I am attempting to convert 15 feature classes containing roughly 70,000 contours each into polygons record by record. Using my current code, each feature class takes roughly 16 hours on my current ...
6
votes
0answers
1k views
Is it possible to do parallel processing in GDAL and QGIS?
It seems like processing a large raster dataset in QGIS can cause a crash, for example in QGIS failing to complete clip or intersect and crash dump in qgis. Is it possible to use GDAL parallel ...
13
votes
2answers
3k views
Multiprocessing Errors - ArcGIS implementation
I was wondering if anyone else in the community here has attempted to use multi-processing for spatial analyses. Namely I am trying to iterate through a series of rasters, create a multiprocessing job ...
2
votes
2answers
2k views
Full utilization of CPU-cores with gdalwarp
I need to process many large GeoTiff files with gdalwarp.
For this I have used the following command:
gdalwarp -r cubic -s_srs EPSG:25832 -t_srs EPSG:3857 -cutline shapefile.shp -crop_to_cutline -...
3
votes
1answer
1k views
Multiprocessing issues with ArcPy
I am using ArcGIS 'watershed' routine in a script which makes use of multiprocessing.
This works fine, but during execution I get the message:
Unable to remove directory. Possible causes:
1- Not ...
9
votes
5answers
2k views
python.multiprocessing and “FATAL ERROR (INFADI) MISSING DIRECTORY”
While trying to do multiprocessing with arcpy, I am occasionally running into this error:
FATAL ERROR (INFADI)
MISSING DIRECTORY
I have no clue what is triggering this error, and it crashes the ...
7
votes
1answer
5k views
Can multiprocessing with arcpy be run in a script tool?
I have been tinkering around with the multiprocessing module to resolve some processing of data in ArcMap 10.3.
I have created a script which works fine if I run it in IDLE. I see all my cores max ...
0
votes
0answers
103 views
arcpy and multiprocessing: switch workflow to shapefiles to avoid schemalocks
I need to processe a large dataset through a bunch of interwoven scripts using arcpy. On a small test dataset my workflow functions fine so far but when it comes to the large dataset i see no other ...
1
vote
2answers
356 views
Using MultiProcessing With Update Cursor?
I am trying to use multiprocessing to help me run a tool that will clip a feature and sum an attribute in it. The tool needs to iterate through about 24,000 rows of a featureclass while doing this. ...
4
votes
2answers
4k views
Processing multiple files simultanously using python
I have 11000 point shapefiles to run Near tool on against another point shapefile.
I am looking for a way to process 10 files at the same time.
Is there a way to process so many files using Python ...
