Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I'm trying to process some raster data using ogr/gdal and I can't seem to get full utilization of all the cores on my machine. When I only run the process on a single core, I get 100% utilization of that core. When I try to split into multicore (in the example below, by chunking the x offsets and putting them in a queue), I get pathetic utilization on each of my 8 cores. It seems like it only adds up to 100% utilization across each core (e.g. 12.5% on each).

I was concerned that using the same datasource was the bottleneck, but I then I duplicated the underlying raster file for each core... and core utilization is still crap. This leads me to believe that ogr or gdal is somehow behaving like a bottleneck shared resource but I can't find anything online about that. Any help would be much appreciated!

This is the "helper" function that runs inside each Worker thread:

def find_pixels_intersect_helper(datasource, bounds_wkt, x_min, x_max):
    bounds = ogr.CreateGeometryFromWkt(bounds_wkt)
    rows_to_write = []
    for x_offset in range(x_min, x_max):
        for y_offset in range(datasource.RasterYSize):
            pxl_bounds_wkt = pix_to_wkt(datasource, x_offset, y_offset)
            pxl_bounds = ogr.CreateGeometryFromWkt(pxl_bounds_wkt)
            if pxl_bounds.Intersect(bounds):
                rows_to_write.append(['%s_%s' % (x_offset, y_offset), pxl_bounds.Centroid().ExportToWkt()])
share|improve this question
    
Unlikely, but did you check if memory is the bottleneck? –  lynxlynxlynx Jun 11 '12 at 20:31
    
@lynxlynxlynx - yep. Memory is definitely not the bottleneck. Been trying to track this thing down all day... this is pretty weird. –  Max Jun 11 '12 at 20:35
    
It may be that the raster driver you are using is simply not designed to be called from more than one thread at a time. Reference: mail-archive.com/gdal-dev@lists.osgeo.org/msg07283.html –  blah238 Jun 11 '12 at 21:35

1 Answer 1

up vote 6 down vote accepted

OK. That was a day of my life that I'll never get back again. Turns out the problem was not in the code I posted above. That's totally fine. Turns out that this was a case of threading.Thread vs. multiprocessing.Process.

As pointed out in the python documentation:

The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads. Due to this, the multiprocessing module allows the programmer to fully leverage multiple processors on a given machine

Thus, threading.Thread is for IO-intensive operations, multiprocessing.Process is for CPU intensive operations. I switched to multiprocessing.Process and everything works great.

Check out this tutorial to learn how to use multiprocessing.Process

share|improve this answer
    
I was just going to suggest that, I wasn't sure which implementation (there are also 3rd party implementations) you were using :) I have used that recently to speed up a neat building shadows tool here: Port “Producing Building Shadows” Avenue code to ArcGIS 10 –  blah238 Jun 12 '12 at 2:17
    
+1 I was about to post that you should have a word on the GDAL-dev mailing list; but I'm now pleased you didn't! This has been squirrelled away for future reference. –  MerseyViking Jun 12 '12 at 9:12
    
FWIW (probably not very much), I read somewhere people are collecting funds to try to fix the global interpreter lock (GIL) problem. I think it will be for 3.x. –  canisrufus Jun 12 '12 at 14:09

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.