1

I need help getting multiprocessing to work with the spatial analyst tool - Solar Analyst.

I have tested the multiprocessing bit and it works fine. This sample is passing it a file containing 2 lines used as parameters that get parsed and fed to the main function (final) as vars to the Solar Analyst tool. The function works on its own, but when altogether with the multiprocessing code I get the traceback below.

I think that the fact the raster DEM is trying to be accessed by multiple child processes is what is tripping the traceback error, as I get the error after some time, close to what would be the second run from the pool. Perhaps it is because I do not have unique scratch workspaces for each child process, as is mentioned in the following links: link1 link2. Again, if this is the case I need help implementing a unique workspace for each child process.

However, as suggested by a colleague I have successfully run the Solar Analyst tool using the same input raster on two instances of ArcMap being open, at the same time. So I am not convinced it is having a hard time accessing the raster because the other child process is as well. Thoughts?

In general, please feel free to offer any pointers you might have for optimizing this code. 

import multiprocessing
import time
import arcpy

def final(x):
    outputfile=open("outputfile.txt","a")
    starttime= time.time()
    # Check out any necessary licenses
    arcpy.CheckOutExtension("Spatial")
    arcpy.env.workspace = r"C:\data\LiDAR\working\V2\FINAL_TEST\final_test"
    arcpy.env.overwriteOutput = True

    # Parse params
    the_split= x.split(",")
    DOY = the_split[0]
    k_daily = the_split[1]
    diffuse = the_split[2]


    # Set local variables
    inRaster = r"C:\data\LiDAR\working\V2\best_dems_05_1\oak_idwnat_flt.tif"
    latitude = 48.4222
    skySize = 400
    timeConfig = arcpy.sa.TimeWithinDay(DOY,0, 24)
    dayInterval = 14
    hourInterval = 0.5
    zFactor = 1
    calcDirections = 32
    zenithDivisions = 16
    azimuthDivisions = 16
    diffuseProp = diffuse
    transmittivity = k_daily
    outDirectRad = ""
    outDiffuseRad = ""
    outDirectDur = ""

    # Execute AreaSolarRadiation
    outGlobalRad = arcpy.sa.AreaSolarRadiation(inRaster, latitude, skySize, timeConfig,
       dayInterval, hourInterval, "NOINTERVAL", zFactor, "FROM_DEM",
       calcDirections, zenithDivisions, azimuthDivisions, "UNIFORM_SKY",
       diffuseProp, transmittivity, outDirectRad, outDiffuseRad, outDirectDur)

    # Save the output 
    outGlobalRad.save(r"C:\data\LiDAR\working\V2\FINAL_TEST\output\global_"+str(DOY))
    end_time = time.time()
    outputfile.write("batch job" + str(DOY) + "ended well")
    outputfile.write(" %d " % (end_time-starttime) )
    outputfile.close()
#To test this script only pass the below
#if __name__ == "__main__":
#   final("266,0.4269347,0.771440152")

if __name__=='__main__':

    start= time.time()

    filename = "k_daily_diffuse_2.txt"
    readfile = open(filename, "r")
    lines_from_filename = []
    for i in readfile:
        lines_from_filename.append(i)
    p = multiprocessing.Pool(2)
    p.map(final, lines_from_filename)
    end = time.time()
    print "batch job ended well"
    print end-start

    p.close()
    p.join()

----------------------------------------OUTPUT BELOW----------------------------------------

Python 2.7.5 (default, May 15 2013, 22:44:16) [MSC v.1500 64 bit (AMD64)] on win32 Type "copyright", "credits" or "license()" for more information.
>>> ================================ RESTART ================================
>>> 

Traceback (most recent call last):   
File "C:\data\LiDAR\working\V2\FINAL_TEST\multi_test_FINAL.py", line 63, in <module>
        p.map(final, lines_from_filename)   
File "C:\Python27\ArcGISx6410.2\lib\multiprocessing\pool.py", line 250, in map
        return self.map_async(func, iterable, chunksize).get()   
File "C:\Python27\ArcGISx6410.2\lib\multiprocessing\pool.py", line 554, in get
        raise self._value ExecuteError: 
ERROR 999999: Error executing function. 
Failed to open raster dataset 
Failed to execute (AreaSolarRadiation).

These sites report the same traceback errors, if they offer any clues: GeoNet from 2010 , GeoNet from 2014 , GIS StackExchange

-----------------------UPDATE-------UPDATE--------UPDATE------------------------------------

A clue! Following a suggestion by Nathan at Riverside I opened up two instances of ArcCatalog and ran the solar analyst tool using the same input raster at the same time. I got the following error: enter image description here This explains the "Cannot open raster" traceback.

My initial thought is to have a unique copy of the input raster available for each run of the child process. Thoughts?

Improve this question
2
  • 3
    Consider reducing this to the smallest non-working example.
    – phloem
    Commented Apr 10, 2015 at 8:05
  • @phloem Good suggestion. I shall whittle away at it; though suffice to know a large part of the script is just for the Solar Analyst tool, which runs fine on its own, as does the multiprocessing bit.
    – geokrowding
    Commented Apr 10, 2015 at 9:10

2 Answers 2

Reset to default
2

Ok, so after many trials and tribulations, I have wrangled arcpy into submitting to multiprocessing for running the Solar Analyst tool. Behold:

import multiprocessing 
import time
import arcpy
import os
import shutil

def final(x):
    # Check out any necessary licenses
    arcpy.CheckOutExtension("Spatial")
    arcpy.env.overwriteOutput = True
    temp_directory = r"C:\yourWD\temp_" + str(DOY)
    arcpy.env.scratchWorkspace = temp_directory + "\unique_folder_" + str(DOY)
    #OR use this method to create unique folder paths:
    #os.path.join(temp_directory, "unique_folder" + str(DOY))
    #documentation on this method can be found here: [os.path][1], [pythoin.org][2], [ESRI MP page][3]. Note the last link has the scratch going in main(), in our case needs to be in child process.

    # Parse list of atmospheric params 
    the_split= x.split(",")
    DOY = the_split[0]
    k_daily = the_split[1]
    diffuse = the_split[2]

    # Set local variables
    # First make a temp working dir to avoid tripping the "Failed to open raster dataset"
    inRaster = temp_directory+ "\the_raster_copy.adf"
    latitude = (your lat here)
    skySize = 400
    timeConfig = arcpy.sa.TimeWithinDay(DOY,0, 24)
    dayInterval = 7
    hourInterval = 0.5
    zFactor = 1
    calcDirections = 64
    zenithDivisions = 32
    azimuthDivisions = 32
    diffuseProp = diffuse
    transmittivity = k_daily
    outDirectRad = ""
    outDiffuseRad = ""
    outDirectDur = ""

    # Execute AreaSolarRadiation
    outGlobalRad = arcpy.sa.AreaSolarRadiation(inRaster, latitude, skySize, timeConfig,
       dayInterval, hourInterval, "INTERVAL", zFactor, "FROM_DEM",
       calcDirections, zenithDivisions, azimuthDivisions, "STANDARD_OVERCAST_SKY",
       diffuseProp, transmittivity, outDirectRad, outDiffuseRad, outDirectDur)

    # Save the output 
    outGlobalRad.save(r"C:\yourWD\global_"+str(DOY))


def main():
    workspace = r"C:\yourWD"
    arcpy.env.workspace = workspace
    filename = "list_of_atm_params.txt"
    readfile = open(filename, "r")
    lines_from_filename = []
    for i in readfile:
        lines_from_filename.append(i)
    p = multiprocessing.Pool(2)
    p.map(final, lines_from_filename)
    p.close()
    p.join()
if __name__=='__main__':
    main()

Having a unique scratch workspace for each child process is in fact very important, as mentioned in other posts concerning MP using arcpy (link1,link2, link3).

Note that this script takes a file named "list_of_atm_params.txt" and parses it for the atmospheric params that are essential for the model to produce accurate results. Much attention should be paid to attaining representative values here.

Results from initial testing of multiprocessing are:

Processes    Time    Ratio (min per process)
1            11min   11/1  = 11
6            15min   15/6  = 2.5
12           23min   23/12 = 1.91

So I hope this helps those of you looking to multiprocess the Solar Analyst tool. It has reduced my computation time from 92 days to about 2.5 weeks, so well worth the time getting it to work for me.

Improve this answer
0
1

A couple of ideas/notes, though not a solution probably. I haven't done anything with python multiprocessing, but have done a decent amount with regular old fork/exec, so read these with that in mind

1) outputfile=open("outputfile.txt","a")

It looks like each process is going to try to open and append to the same output file at the same tile. I expect that this is going to end poorly.

2) arcpy.CheckOutExtension("Spatial")

Will this try to check out multiple Spatial extensions? If so, do you have multiple licenses, or can you check out the extension before doing multiprocessing? This is going to be checked out for every line in your input file, I think.

3) "Perhaps it is because I do not have unique scratch workspaces for reach child process, as is mentioned in the following links"

One option is to do the following:

Have a base scratch directory, eg. C:\temp\scratch

Get the individual processes pid os.getpid() and make the real temp dir, eg os.makedirs("c:\temp\scratch\scratch_" + str(os.getpid()))

Then use the directory you just created.

You could also use the pid to specify a different output file for each process.

Improve this answer
1
  • Thanks for your input. 1) point taken, though it works fine, and is just in place for debugging. I removed any writing to the outfile to rule out it contributing to the problem. 2) I believe this is fine, and is required for the tool to work. I am curious how that works being in a child process, and I guess will find out once I get past these bugs first. 3) I like the idea of using the PID to individualize each dir, but would you mind providing a working example of the code for completion, please. Also, this solution might come in handy to address my update. Looking forward to your reply!
    – geokrowding
    Commented Apr 10, 2015 at 21:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.