I'm trying to process some raster data using ogr/gdal and I can't seem to get full utilization of all the cores on my machine. When I only run the process on a single core, I get 100% utilization of that core. When I try to split into multicore (in the example below, by chunking the x offsets and putting them in a queue), I get pathetic utilization on each of my 8 cores. It seems like it only adds up to 100% utilization across each core (e.g. 12.5% on each).

I was concerned that using the same datasource was the bottleneck, but I then I duplicated the underlying raster file for each core... and core utilization is still crap. This leads me to believe that ogr or gdal is somehow behaving like a bottleneck shared resource but I can't find anything online about that. Any help would be much appreciated!

This is the "helper" function that runs inside each Worker thread:

def find_pixels_intersect_helper(datasource, bounds_wkt, x_min, x_max):
    bounds = ogr.CreateGeometryFromWkt(bounds_wkt)
    rows_to_write = []
    for x_offset in range(x_min, x_max):
        for y_offset in range(datasource.RasterYSize):
            pxl_bounds_wkt = pix_to_wkt(datasource, x_offset, y_offset)
            pxl_bounds = ogr.CreateGeometryFromWkt(pxl_bounds_wkt)
            if pxl_bounds.Intersect(bounds):
                rows_to_write.append(['%s_%s' % (x_offset, y_offset), pxl_bounds.Centroid().ExportToWkt()])
  • Unlikely, but did you check if memory is the bottleneck? Jun 11, 2012 at 20:31
  • @lynxlynxlynx - yep. Memory is definitely not the bottleneck. Been trying to track this thing down all day... this is pretty weird.
    – Max
    Jun 11, 2012 at 20:35
  • It may be that the raster driver you are using is simply not designed to be called from more than one thread at a time. Reference: mail-archive.com/[email protected]/msg07283.html
    – blah238
    Jun 11, 2012 at 21:35

1 Answer 1


OK. That was a day of my life that I'll never get back again. Turns out the problem was not in the code I posted above. That's totally fine. Turns out that this was a case of threading.Thread vs. multiprocessing.Process.

As pointed out in the python documentation:

The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads. Due to this, the multiprocessing module allows the programmer to fully leverage multiple processors on a given machine

Thus, threading.Thread is for IO-intensive operations, multiprocessing.Process is for CPU intensive operations. I switched to multiprocessing.Process and everything works great.

Check out this tutorial to learn how to use multiprocessing.Process

  • I was just going to suggest that, I wasn't sure which implementation (there are also 3rd party implementations) you were using :) I have used that recently to speed up a neat building shadows tool here: Port “Producing Building Shadows” Avenue code to ArcGIS 10
    – blah238
    Jun 12, 2012 at 2:17
  • +1 I was about to post that you should have a word on the GDAL-dev mailing list; but I'm now pleased you didn't! This has been squirrelled away for future reference. Jun 12, 2012 at 9:12
  • FWIW (probably not very much), I read somewhere people are collecting funds to try to fix the global interpreter lock (GIL) problem. I think it will be for 3.x.
    – canisrufus
    Jun 12, 2012 at 14:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.