Return to Revisions

Here is the gnu parallel solution. With some care most emabrrassingly parallel linux based ogr or saga algorithms could be made to run with it inside your QGIS installation.

Obviously this solution requires the installation of gnu parallel. To install gnu parallel in Ubuntu, for example, go to your terminal and type

sudo apt-get -y install parallel

NB: I couldn't get the parallel shell command to work in Popen or subprocess, which I would have preferred, so I hacked together an export to a bash script and ran that with Popen instead.

Here is the specific shell command using parallel that I wrapped in python

parallel ogr2ogr -skipfailures -clipsrc tile_{1}.shp output_{1}.shp input.shp ::: {1..400}

Each {1} gets swapped out for a number from the {1..400} range and then the four hundred shell commands get managed by gnu parallel to concurrently use all the cores of my i7 :).

Here is the actual python code I wrote to solve the example problem I posted. One could paste it in directly after the end of the code in the question.

import stat
feature_count=tile_layer.dataProvider().featureCount()
subprocess_args=["parallel", \
"ogr2ogr","-skipfailures","-clipsrc",\
os.path.join(output_folder,"tile_"+"{1}"+".shp"),\
os.path.join(output_folder,"output_"+"{1}"+".shp"),\
input_file,\
" ::: ","{1.."+str(feature_count)+"}"]
#Hacky part where I write the shell command to a script file
temp_script=os.path.join(output_folder,"parallelclip.sh")
f = open(temp_script,'w')
f.write("#!/bin/bash\n")
f.write(" ".join(subprocess_args)+'\n')
f.close()
st = os.stat(temp_script)
os.chmod(temp_script, st.st_mode | stat.S_IEXEC)
#End of hacky bash script export
p = Popen([os.path.join(output_folder,"parallelclip.sh")],\
stdin=PIPE, stdout=PIPE, stderr=PIPE)
#Below is the commented out Popen line I couldn't get to work
#p = Popen(subprocess_args, stdin=PIPE, stdout=PIPE, stderr=PIPE)
output, err = p.communicate(b"input data that is passed to subprocess' stdin")
rc = p.returncode
print output
print err

#Delete script and old clip files
os.remove(os.path.join(output_folder,"parallelclip.sh"))
for i in range(feature_count):
    delete_file = os.path.join(output_folder,"tile_"+str(i+1)+".shp")
    nosuff=os.path.splitext(delete_file)[0]
    suffix_list=[]
    suffix_list.append('.shx')
    suffix_list.append('.dbf')
    suffix_list.append('.qpj')
    suffix_list.append('.prj')
    suffix_list.append('.shp')
    suffix_list.append('.cpg')
    for suffix in suffix_list:
        try:
            os.remove(nosuff+suffix)
        except:
            pass

Let me tell you it's really something when you see all the cores fire up to full noise :). Special thanks to Ole and the team that built Gnu Parallel.

It would be nice to have a cross platform solution and it would be nice if I could have figured out the multiprocessing python module for the qgis embedded python but alas it was not to be.

Regardless this solution will serve me and maybe you nicely.