2

I'm running a fairly small script to get height differences. The scripts gets the height in a point based on a DTM. Running the script in the ArcMap python window uses 1 min 30 seconds for 1000 points, while running it in the python window outside Arcmap (C:\Python27\ArcGIS10.5\python.exe) uses 2 min 40 seconds for the same 1000 points. What causes this big difference? There can be many points for each dataset, so I want it to be able to run outside ArcMap so it doesn't run out if memory.

import arcpy 
import datetime

arcpy.env.workspace =  r'\\path\Laserdata.gdb'
arcpy.env.scratchWorkspace =  r'\\path\Temp_data.gdb'
# laserline 
laser = 'L0380_2m_Vertice_selection'

# 1 x 1 m raster
raster_1m = r'\\path\Connection to sde.sde\DTM1'

def main():
    i = 1
    print "Starttime: " + str(datetime.datetime.now())
    start = datetime.datetime.now()
    fields = arcpy.ListFields(laser)

    if len(arcpy.ListFields(laser,"DTMZ"))>0:  
        print "Field exist"  
    else:  
        print "Does not exist"
        arcpy.AddField_management(laser, "DTMZ", "FLOAT")

    with arcpy.da.UpdateCursor(laser, ['SHAPE@X','SHAPE@Y' ,'DTMZ']) as curs:
        for row in curs:
            X = str(row[0]).replace(".",",")
            Y = str(row[1]).replace(".",",")
            XY = str(X + " " + Y)

            hoydeDTM  = arcpy.GetCellValue_management(raster_1m, XY)
            if str(hoydeDTM) == 'NoData':
                hoydeDTM = -9999
            else:
                hoydeDTM = hoydeDTM

            row[2] = str(hoydeDTM)
            i = i + 1 
            if i % 1000 == 0:
                print i
                print datetime.datetime.now() - start
            curs.updateRow(row)

    print "Endtime: "  + str(datetime.datetime.now())
    print  datetime.datetime.now() - start

if __name__ == '__main__':
    main()
  • 1
    It's unclear whether you mean that one process took 40 seconds or 40 minutes longer. You'd likely have better performance if your DEM was in a local GeoTIFF instead of an Enterprise geodatabase. I wouldn't expect to see performance or memory savings with a 32-bit Python. – Vince Oct 31 '18 at 11:41
  • 3
    A general piece of advice - move your raster out of ArcSDE geodatabase into a file geodatabase (best if local one, stored on your SSD drive). I guess the reason in performance difference is that you might have your raster added into ArcMap session so the GetCellValue operates faster on the raster layer in your map rather than connecting to the ArcSDE and reading the raster each time you run your GetCellValue. Depending on the size of your raster, you may boost the performance drastically by copying the raster into the special in_memory workspace once before going into for loop – Alex Tereshenkov Oct 31 '18 at 11:41
  • In_memory desktop.arcgis.com/en/arcmap/10.3/analyze/modelbuilder/…. Check whether your raster can fit into the RAM you have on your machine – Alex Tereshenkov Oct 31 '18 at 11:42
  • @AlexTereshenkov Thank you for the advice. The problem here is that the raster is huge. Its uncompressed size is 5,72 TB and covers a whole country. I will check if I can clip out parts of it to use for the analysis and add it to the FGDB. – birks Oct 31 '18 at 11:53
  • 1
    Running a script is a singular process. You may have your script editor (IDE) open, but that doesn't keep the raster loaded. Every time you execute the script new, it needs to go fetch/open the raster. When doing "performance tests" or simple compares, its better to do them on individual tool or small part of the workflow. Comparing the execution of something inside ArcMap to a complete script execution isn't an apples-apples compare. – KHibma Oct 31 '18 at 13:04
4

@AlexTereshenkov idea that it could be down to it being a layer may be the source of this problem, if you look at the help file for the Get Cell Value Tool what does the Syntax section tell you that it wants as input? A raster layer, you are not providing it with a layer but a string which is the full path to a raster dataset. I would imagine internally it is having to convert that to a layer object on every cycle of the loop.

That being said you have written a whole load of unnecessary code which can all be done in just one tool! I suggest you explore the Extract Values to Points tool? I would expect this to be much faster as it is a system tool (compiled code).

  • The extract values to point tools was so much faster. Thank you for showing me. – birks Oct 31 '18 at 12:48
  • @birks good news! Glad that has worked, out of interest how fast? – Hornbydd Oct 31 '18 at 13:44
  • Also get into the habit of reading the help file syntax section to understand what the inputs are, layers and datasets may look like the same thing but they are very different beasts (for good reasons). – Hornbydd Oct 31 '18 at 13:50
  • 1
    I think it was something like a minute for the whole 36 k points! – birks Oct 31 '18 at 14:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.