`multiprocessing` is a python package that supports spawning processes using an API similar to the `threading` module.

learn more… | top users | synonyms (1)

1
vote
0answers
35 views

Multiprocessing in python qgis?

When i try to do mutliprocessing in python Qgis it doesn't work from multiprocessing import Process,Lock import multiprocessing def name1(l,i): print i if __name__=='__main__': lock=Lock() ...
1
vote
2answers
68 views

Python and QGIS multiprocessing documentation?

Where can we find better documention for python QGIS multiprocessing. Especially regarding the multiprocessing of python and QGIS.
0
votes
0answers
22 views

Can we integrate QGis with Java? [duplicate]

Inegrateion of java with Qgis and its extensions? In case, If integration is possible can we use multi-threading qgis and java?
0
votes
2answers
49 views

Really slow extraction from raster even after using crop?

I have a large raster file (245295396) cells and stacks of rasters having 4 layers each which lie in the extent of this large raster. To start with I am trying to get value from one stack (3 channels) ...
1
vote
0answers
35 views

Using Multiprocessing inside a plugin in Qgis

I spent quite a bit of time looking on how to use the multiprocessing package, but couldn't find anything on how to use it inside a plugin in QGIS. I am developing a plugin that does some ...
5
votes
3answers
134 views

How to run qgis scripts in parallel across remote server(s)

I have a stand alone python script using qgis that I wrote with the help of the SE question here The script takes three arguments being; an input file path, an output file path, and another input ...
2
votes
1answer
24 views

problem deleting shapefiles because still in use by arcpy cursor [duplicate]

I have a function outputTblContents which contains def outputTblContents ( path ): rows = arcpy.SearchCursor(path, fields="alpha; beta") for row in rows: print("{0} => ...
0
votes
2answers
146 views

Multiprocessing with arcpy and Solar Analyst (spatial analyst)

I need help getting multiprocessing to work with the spatial analyst tool - Solar Analyst. I have tested the multiprocessing bit and it works fine. This sample is passing it a file containing 2 lines ...
4
votes
1answer
160 views

How to write to a single table via Python multiprocessing

A comment in the post Can multiprocessing with arcpy be run in a script tool? got me thinking, as I often need to do exactly this: Just beware of deadlocking situations (two Insert cursors in the ...
4
votes
1answer
282 views

Can multiprocessing with arcpy be run in a script tool?

I have been tinkering around with the multiprocessing module to resolve some processing of data in ArcMap 10.3. I have created a script which works fine if I run it in IDLE. I see all my cores max ...
2
votes
0answers
86 views

How to use the multiprocessing module on python to execute a spatial join?

I have several shapefiles of the entire Amazon rainforest each with millions of polygons (they are fairly detailed and take some time to draw on ArcMap). I need each of these shapefiles to be ...
0
votes
0answers
53 views

A Distributed Cost-Distance Algorithm

I'm currently trying to implement a distributed cost-distance algorithm, however I can't seem to invent my own and still make it fast. I've been googling around a bit but haven't found anything that ...
1
vote
0answers
90 views

multiprocessing and the ArcGIS in_memory workspace

I am trying a script that makes use of multiprocessing and a number of tools within arcpy. The script works only if I write all input, intermediate, and output from child multiprocessing executions to ...
10
votes
3answers
636 views

How can I parallelise embarrassingly parallel GIS operations in QGIS python?

A common requirement in GIS is to apply a processing tool to a number of files or apply a process for a number of features in one file to another file. Much of these operations are embarrassingly ...
2
votes
1answer
73 views

Setting ArcPy environment variables with Python multiprocessing?

I have a script which uses ArcPy + Python multiprocessing. Are ArcPy environment variables shared between separate Python processes? If I write to arcpy.env.extent from within each process am I ...
2
votes
1answer
201 views

arcpy with multiprocessing throws “UnpickleableError: Cannot pickle <type 'geoprocessing Layer object'> objects”

I'm attempting to speed up a process which is currently running synchronously, by using the python multiprocessing module. I'm having trouble sending a feature layer to a function which is called by ...
1
vote
1answer
52 views

Maintaining reference to feature layer (rather than its name string) throughout Python script? [duplicate]

When creating a Layer in arcpy, the syntax uses a string to denote the layer name, as shown in the Make Feature Layer sample script: import arcpy arcpy.env.workspace = "C:\Program Files ...
1
vote
0answers
31 views

Sending commands into FGDB

I have a large .gdb of all blocks in the United States, available from Census. I would like to create a point shape file of all census tract centroids in the US. I have QGIS with FileGDB enabled, as ...
0
votes
1answer
331 views

How to write output from subprocess to disk (using multiprocessing)

This code was developed from another question I posted about how to kill an arcpy tool if it takes longer than a specified time. See How to cancel tool execution in python. Using the multiprocssing ...
3
votes
1answer
674 views

Multiprocessing issues with ArcPy

I am using ArcGIS 'watershed' routine in a script which makes use of multiprocessing. This works fine, but during execution I get the message: Unable to remove directory. Possible causes: 1- Not ...
3
votes
1answer
764 views

Python Postgres Multiprocessing speed increase

I'm trying to speed up my PostGIS queries using multiprocessing. My current setup is using python and psycopg2 such as below. Although this has given a speed increase, it still seems like there are ...
6
votes
1answer
438 views

How to do parallel queries in postgis [closed]

Is there any way to do queries in postgis using multiple cores in a transparent way, other than splitting my queries among several client connections ?
4
votes
1answer
590 views

Can I increase the number of processes in ArcGis Server 10.1?

I use ArcMap 10.1. I have a map with many Mosaic and Raster layers. They are stored in many gdb-s. I publish this map with option "Create tiles on demand". But this tiles are created very slowly (may ...
2
votes
0answers
208 views

python.multiprocessing and “FATAL ERROR (INFADI) MISSING DIRECTORY”

I have a question which is closely related to this old question here: python.multiprocessing and "FATAL ERROR (INFADI) MISSING DIRECTORY" I have also looked at this, which seems to relate ...
2
votes
0answers
130 views

Python scaling issues and possible subprocess conflicts?

I have always found such great analytical and programming expertise on StackExchange in the past that I hope you can help me. Background: I have a project that involves spatially joining and ...
1
vote
2answers
539 views

Parallel Computing and Raster Operations in ArcObjects gives cryptic error

This is a problem I have been struggling with for awhile. The code below takes 1-3 input feature layers (pFLayer) that were stored into a list called pFeatureLayerCollection, casts them into ...
4
votes
1answer
238 views

How to implement multiprocessing with multiple geoprocesses?

I am relatively new to Python and thought I would give a stab at multiprocessing. I have a script that runs well in IDLE or as an ArcMap toolbox script. After perusing these forums and docs.python, I ...
0
votes
2answers
1k views

Execute multiple process in parallel of Arcpy script

I have seen some difficult ways to do parallel processing, but I wonder if it is possible to simply execute multiple process of the same ArcPy script at the same time. My script makes some changes to ...
5
votes
1answer
227 views

parallelization of raster mosaic

I would like to create a raster mosaic of the national USGS NED 30m DEM (~50 GB). I'd like to leverage parallel processing using python multiprocessing module where possible. I'm using ArcGIS 10.0. My ...
16
votes
2answers
1k views

Arcpy Parallel Processing Environment

I recently discovered the Parallel Processing environment setting in ArcGIS (this must be new to 10.1), however, there is no mention as to what tools are designed to utilize this environment setting. ...
18
votes
1answer
1k views

ArcGIS 10.1 Python AddIn Using Multiprocessing crashes ArcMap?

I would like to run a multiprocessing task from a python add-in tool. My issue is that the process keeps failing. Basically crashes ArcMap. Here is my basic code: def function(startOID, endOID, ...
2
votes
1answer
411 views

Multiprocessing error in QGIS with Python on Windows?

I have this error when I try to use Manager() from python multiprocessing library. (Just start QGIS open Python console and execute the two following lines) >>> import multiprocessing ...
10
votes
2answers
2k views

Methods for optimizing multcore processing in ArcGIS

I am interested in learning methods to utilize the full extent of multicore processing power available on a desktop computer. Arc states that background geoprocessing allows the user to utilize ...
10
votes
1answer
645 views

OGR/GDAL threading results in low core utilization

I'm trying to process some raster data using ogr/gdal and I can't seem to get full utilization of all the cores on my machine. When I only run the process on a single core, I get 100% utilization of ...
14
votes
3answers
2k views

Porting Avenue code for Producing Building Shadows to ArcPy/Python for ArcGIS Desktop?

Whuber provided an answer at Producing building shadows? which required using Avenue code. Any idea how to make it work in ArcGIS Desktop 10?
13
votes
2answers
2k views

Multiprocessing Errors - ArcGIS implementation

I was wondering if anyone else in the community here has attempted to use multi-processing for spatial analyses. Namely I am trying to iterate through a series of rasters, create a multiprocessing job ...
4
votes
1answer
264 views

Splitting data into geographic zones for multiprocessing

I have been looking at multiprocessing and tools like Find Duplicates in Big Data and they seem to simply split the data into sections based on rows/number of records. I am trying to do an intersect ...
7
votes
5answers
804 views

python.multiprocessing and “FATAL ERROR (INFADI) MISSING DIRECTORY”

While trying to do multiprocessing with arcpy, I am occasionally running into this error: FATAL ERROR (INFADI) MISSING DIRECTORY I have no clue what is triggering this error, and it crashes the ...