16

You need to use the beginCluster and endCluster functions of the raster package. See the example below. library(raster) library(snow) # Make test data # RasterStack r <- raster(ncol=36, nrow=18) r[] <- 1:ncell(r) s <- stack(r, sqrt(r), r/r) # SpatialPolygons cds1 <- rbind(c(-180,-20), c(-160,5), c(-60, 0), c(-160,-60), c(-180,-20)) cds2 <- ...


13

If you change your program to read the file name from the command line and split up your input file in smaller chunks, you can do something like this using GNU Parallel: parallel my_processing.py {} /path/to/polygon_file.shp ::: input_files*.shp This will run 1 job per core. All new computers have multiple cores, but most programs are serial in nature and ...


11

Yes, you can run multiprocessing child processes from a toolbox script. Below is some code to demonstrate in a Python Toolbox (*.pyt). There are a number of "gotchas". Some (but not all) will be applicable to Python script tools in a binary toolbox (*.tbx), but I only use Python Toolboxes these days so have not tested. Some "gotchas"/tips: Make sure each ...


11

Yes, GDAL supports parallel processing, and this support applies to gdalwarp by default. Use the -multi option with gdalwarp to enable multithreading, as opposed to only multiple cores. Details: Without -multi: 33.849s, and the CPU reached 555%. (multiple cores) With -multi: 23.377s, and the CPU reached 700%. (multiple cores and multiple threads) Raster ...


8

Multiprocessing is trying to use ArcMap.exe instead of python.exe to run the child processes. Either disable running the script 'in process' or use multiprocessing.set_executable(os.path.join(sys.exec_prefix, 'python.exe')) Note: don't use sys.executable to get the path to python.exe if running in-process as it will be pointing to ArcMap.exe, use sys....


6

Just to add to @PolyGeo's answer that I also could not find any official documentation regarding multiprocessing (if it even exists!). But there is also another method, described in this blog, which uses multithreading in QGIS which might be useful. Main difference between the two methods are (more of which is discussed here): Multiprocessing allows ...


6

If there is no QGIS official documentation available for multiprocessing QGIS using Python then I recommend reviewing earlier Q&As here that have tags multiprocessing, qgis and python. There are two with answers that would seem relevant to what you are trying to do: Parallelising GIS operations in PyQGIS? Multiprocessing error in QGIS with Python on ...


6

Writing data and reading data is a different story. Modifying data from different parallel resources always needs to be serialized and prepared for changes in source data while writing. It is therefore more complicated and needs to be treated carefully while reading data in parallel is no trouble. Also see the wiki page for parallel queries. Even when ...


6

Yes, although this seems not to be mentioned in the documentation. If you include num_threads=8 or num_threads='all_cpus' as an argument to rasterio.open then multithreading will be enabled (for writing of compressed data). Example: import numpy as np, rasterio size = 16384 chunk = 512 with rasterio.open('test.tiff', 'w', driver='GTiff', nodata=0, ...


5

You're probably getting some sort of exception being raised. Perhaps use a Queue to pass messages back to the parent process. Tested working code: import os, arcpy, arcgisscripting, time, sys from multiprocessing import Process from multiprocessing.queues import SimpleQueue def ConvertCADtoGDB(msgs,in_dgn,out_gdb): try: gp = arcgisscripting....


5

Rather than using the GNU Parallel method you could use the python mutliprocess module to create a pool of tasks and execute them. I don't have access to a QGIS setup to test it on but multiprocess was added in Python 2.6 so provided that you are using 2.6 or later it should be available. There are a lot of examples online on using this module.


5

See this blog post, it should cover it http://blogs.esri.com/esri/arcgis/2012/09/26/distributed-processing-with-arcgis-part-1/


5

If you are using a Linux OS, you could consider using GNU parallel to process multiple files in parallel. For instance, a simple example using gdalinfo in parallel is: cat list_of_images.txt | parallel -j 4 gdalinfo {} Where list_of_images.txt, is a text file containing the filename. To address your above example, you will need to put your command ...


5

I ended up using an approach based on the snowfall package. It is quite simple, works really good and the point extraction function is as fast as the number of cores that you can use. The approach I used was inspired by this post, and here is my reproducible example: library(raster) library(snowfall) # Create date sequence idx <- seq(as.Date("2010/1/1")...


5

The problem is that rtree is a ctypes wrapper around libspatialindex, and pickle has no way of storing the pointer to the rtree in memory. You could instead store the index to disk: from multiprocess.pool import Pool from rtree import index class Test(object): def __init__(self): pass def test(self, a): self.idx = index.Index('./...


5

Thread safety QGIS Processing algorithms can be thread-safe or not. According to the docs, the latter are : [...] algorithms which manipulate the current project, layer selections, or with external dependencies which are not thread-safe. You can run the following code snippet in a QGIS Python console (Ctrl+Alt+P) to obtain a list of QGIS Processing ...


4

The quick answer is that you can't do parallel queries. Not yet, at least. PostgreSQL, the most common database engine that processes PostGIS queries, has parallel query execution as a development priority. Although there has been some progress on the prerequisite background worker API in versions 9.3 and 9.4, the timing of this feature is probably not ...


4

There are several overlapping issues here. ArcGIS Desktop is single-threaded, but can make use of a multi-core machine because it can then get the exclusive use of one core. Unless there's a Direct Connect connection to an enterprise geodatabase, in which case, each connection will be run as an additional thread. ArcGIS Server supports multiple cores ...


4

I found this issue arises when the arcpy.env.workspace and arcpy.env.scratchWorkspace are the same for two different processes. Arc writes almost all intermediate rasters to the workspace (or scratch workspace) in the ESRI GRID format. You can't write two ESRI GRID rasters into the same directory at the same time due to the pseudo-database structure of the ...


4

Once you've got a variogram fitted then kriging is trivially parallelizable. Kriging predictions are independent of each other. So, divide your prediction points (grid) into N sets, where N is your number of cores, and do the predictions for each of the point sets on a separate core. Merge the predictions afterwards. You can use any of the paralleling ...


4

If you are restricted to windows you can write a DOS sript and run it in parallel as decribed parallel execution of shell processes If can run in linux there is a procedure pretty well worked out using gnu-parallel. I did something similar for QGIS described in more detail at How to run processing commands in parallel in QGIS Although the examples I give ...


4

Use this: from PyQt5.QtCore import QSettings, QThread # parallel rendering QSettings().setValue("/qgis/parallel_rendering", True) for use max cores from qgis.core import QgsApplication threadcount = QThread.idealThreadCount() QgsApplication.setMaxThreads(threadcount) And extra for use OpenCL acceleration QSettings().setValue("/core/OpenClEnabled", True)


3

1) resample results in 50% improvement I was able to get about 50% improvement by resampling directly from the cld raster to a new raster with the same extent/resolution as r and a nearest neighbor sampling method: system.time({ mat<-as.data.frame(getValues(r)) mat$landuse<- NA mat$landuse<-getValues(resample(cld,r,method='ngb')) }) user ...


3

Here is the gnu parallel solution. With some care most emabrrassingly parallel linux based ogr or saga algorithms could be made to run with it inside your QGIS installation. Obviously this solution requires the installation of gnu parallel. To install gnu parallel in Ubuntu, for example, go to your terminal and type sudo apt-get -y install parallel NB: I ...


3

Environments aren't propagated from process to process, so changing extent in one won't affect the other at all.


3

I finally found the time to look into this. I don't fully understand the "unpickleable" error message, but a workaround is to pass only strings into the multiprocessor. Something like this: import multiprocessing, arcpy, os def doProcess(fClass): #This function doesn't do anything, it's just to show that accessing arcpy methods is possible print("...


3

Just use the following function def run_MultiPros(function, variables): """<function, variables> Execute a process on multiple processors. INPUTS: function(required) Name of the function to be executed. variables(required) Variable to be passed to function. Description: This function will run the given fuction on to multiprocesser. ...


3

I was just looking for this myself and found some of the answers: 1) Which geoprocessing tools honor the parallel processing environment? I couldn't find a comprehensive list of them other than the ones linked in the other answer, but if you look at the geoprocessing tool reference, you can tell for that tool by the list of Environments it supports ...


3

I was wrong in my comment to original post. Suggested workaround does work. In case someone else has that issue here is what needs to be done in QGIS 2.0 before instantiating Manager(). # OSGeo4W does not bundle python in exec_prefix for python path = os.path.abspath(os.path.join(sys.exec_prefix, '../../bin/pythonw.exe')) mp.set_executable(path) sys.argv = [...


3

The problem you are experiencing is actually twofold. Firstly, and as pointed out by @EdzerPebesma, you are not loading the required packages on each of the 4 nodes separately. You have to use clusterEvalQ to tell each node which packages it is going to need to fulfill the required (spatial) tasks. Secondly, you need to assign a proj4string to the polygon ...


Only top voted, non community-wiki answers of a minimum length are eligible