How to run qgis scripts in parallel across remote server(s)

Question

I have a stand alone python script using qgis that I wrote with the help of the SE question here

The script takes three arguments being; an input file path, an output file path, and another input file path. a la:

myscript.py input1.shp output.shp input2.shp

As per advice from this question the script now happily runs in parallel using gnu parallel and gives the correct output files with the following command:

parallel myscript.py {.}.shp {.}_output.shp /tmp/input.shp  :::  /tmp/tile_{1..443}.shp 

I want to do the work in parallel on a remote server (or servers). So I enabled a passwordless login to an available server as per this question, I installed QGIS on the remote server, copied over the script, and made it executable from bash. If I manually copy the required input files over to the server the same parallel command as above runs fine.

The problem I have is when trying to run the remote parallel work from the local machine.

There is an example for running a remote process described here

One could imagine that the equivilent gis type line as per my example might look as follows:

parallel --trc {.}_output.shp -bf /tmp/input.shp -S 192.168.88.222,: \
'myscript.py {.}.shp {.}_output.shp /tmp/input.shp'  :::  /tmp/tile_{1..443}.shp

Unfortunately, there's a fundamental problem with this approach. If a GIS program calls the path to a "shapefile" as an argument; while /path/to/my/file.shp will work most of the time, the "shapefile" actually consists of a collection of files with a common filename prefix. When run, the program actually locates whichever file it needs from the collection from the path to the ".shp" file.

This issue then becomes a problem when the gis script is operating in the remote environment since it doesn't have access to the full collection of files.

The question is then how do we solve this?

Answer 1

Your problem is that --return takes an argument: GNU Parallel can guess which file to transfer, because it assumes you give the files to transfer as argument. But it cannot guess which file to return, as you could be creating all sorts of files. In your case the file to return is probably:

/tmp/output_{1}.shp

so:

--return  /tmp/output_{1}.shp

But maybe you want the input and output file removed on the remote server, too?

--transfer --return /tmp/output_{1}.shp --cleanup

which can be shortened to:

--trc /tmp/output_{1}.shp

Answer 2

Here is a general solution which should work for most any stand alone qgis script; It solves the "shape file collection" problem by wrapping the call to the qgis script with tar pipes as follows:

(parallel 'touch {.}.tar && rm {.}.tar && tar -cf {.}.tar {.}.* && echo {.}.tar' ::: tile_{1..445}.shp)|\
(parallel --bf input.tar --trc {.}_out.tar -S $SERVER1,: 'tarwrap.sh myscript.py {.}.tar {.}_output.tar input.tar')
(parallel 'tar -xf {.}_out.tar && rm {.}_out.tar && rm {.}.tar' ::: tile_{1..445}.shp)

I know it looks horrific but let me break it down: The first parenthetic portion uses parallel locally to tar the shapefile collections into single files (first deleting any existing tar files of the same name) and then echos the resultant list of tar files as the argument to the second parenthetic portion.

The second parenthetic portion uses gnu-parallel to carry out the qgis script on the remote servers and in a sence mirrors the non-working example remote use of parallel given in the question. Note especially here the addition of the "tarwrap" function and the exchange of the script's shape file arguments for tar file arguments. Now, since the arguments represent single files the gnu-parallel program will work as desired to pass the shape file collections to the remote servers.

Finally the third parenthetic portion (which is a seperate command) untars the result and deletes the temporary tars from the local PC

The purpose of the tarwrap.sh bash script is to build a clean temporary folder in each of the remote servers and untar the shape file collections into them, then run the qgis script with its shape arguments as required - changing the "tar" arguments back to there respective "shp" arguments as required. Finally the tarwrap.sh script retars ALL the shape files back up and delivers them to the gnu-parallel working directory for return and cleanup.

The code for the tarwrap.sh script is given below:

#!/bin/bash
tmpdir=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 8 | head -n 1)
mkdir /tmp/$tmpdir
execstring=""
for var in "$@"
do
    if [ "${var: -4}" == ".tar" ]
    then
    B=$(basename "$var"); D=$(dirname "$var");
    subvar="$D/${B%.*}.shp" #change the command 'tar's to 'shp's
    execstring="${execstring}$subvar " 
    if [ -f "$var" ]
    then
        tar -xf "$var" -C "/tmp/$tmpdir" #untar the supplied shp files
    fi
    else
    execstring="${execstring}$var "
    fi
done
runscript=${execstring::-1}
(cd "/tmp/$tmpdir" && $runscript) #Execute the supplied command with shp instead of tar
for var in "$@"
do
    if [ "${var: -4}" == ".tar" ]
    then
    cwd=$(pwd)
    godir="/tmp/$tmpdir"
    inshp="${var::-4}"
    cd $godir
    for fname in $(ls | grep $inshp); do tar -uf "${inshp}.tar" "$fname"; done
    cd $cwd
    mv "$godir/$var" "$cwd/$var"
    fi
done
rm -r "/tmp/$tmpdir"

This is tested and now working for me to run my qgis scripts across multiple remote hosts.

Points to note: both your python qgis script and the tarwrap.sh script need to be accessible from the parallel working directory of your remote servers AND executable from there.

Have a good look at how your "expected" remote parallel command for the qgis script as exemplified in the question translates into the three part wrapped command in this answer.

Finally please note that this process should work for *any qgis script. You just need to remember to replace the shp arguments with tar arguments and let the tarwrap.sh script handle untaring and retarring the results. Any other arguments to your qgis script can be left as is.

Answer 3

A more conventional (and simpler) cluster computing solution is to have a shared directory in the master node mounted in each slave node and let the networking protocols handle all the file passing. As per this reddit post

Rather than building a network folder mounting script I just did it all manually using nfs. First, install nfs on the master and create the shared folder

sudo apt-get install nfs-kernel-server
mkdir -p /home/richard/netshare

Then, set up a permanent tmpfs folder in order to make access to the master data as fast as possible. Add the following line to the end of /etc/fstab

tmpfs /home/<user>/netshare tmpfs defaults 0 0

Replace <user> with a user which exists in each node. This creates a tmpfs drive half the size of your RAM and has very fast access for the rest of your network.

Next mount the tmpfs drive as an nfs share as per SettingUpNFSHowTo. Add the following line to etc/exports

/home/<user>/netshare       192.168.88.0/24(rw,fsid=0,insecure,no_subtree_check,async)

Again where <user> is a user on each system and 192.168.88.0/24 is your local network. (in CIDR notation).

Now restart your nfs server.

sudo service nfs-kernel-server restart

Setup the slave nodes

sudo apt-get install nfs-common
mkdir -p /home/<user>/netshare

Then when you want your slave node to have access to the shared directory you can run the following line on the slave node

sudo mount -t nfs -o proto=tcp,port=2049 192.168.88.239:/ /home/<user>/netshare

where 192.168.88.239 is the ip address of the master node

If you dont want to have to manually mount the shared directory you could try something like the sshfs script described in the first link or you could try autofs. Neither of which worked for me however. Or you could modify the sshfs script to mount nfs drives as described here.