1

arcpy/python beginner here.

I am trying to write a script that uses the Cell Statistics function to derive annual rainfall data for an entire continent from 1983 to 2016. The original data is made up of slightly over 12,000 individual tiff files (dimensions 751 x 801 pixels, 1 degree/11km spatial res.) measuring rainfall for every single day for the entire period. The filenames include a consistent time stamp in the form of *name.yyyymmdd. The annual aggregates I seek are the rainfall totals from June to October (5 months) of each year. I set up my script with the monthly loop nested within the annual loop as follows:

    import arcpy
    arcpy.CheckOutExtension("Spatial")
    arcpy.env.workspace = r'C:/data/rainfall'
    arcpy.env.overwriteOutput = True
    years = range(1983,2017)
    months = ["06","07","08","09","10"]
    yearlist = []
    for year in years:
        for month in months:
            monthlist = arcpy.ListRasters("*"+str(year)+month+"*","TIF")
            yearlist.append(monthlist)
        #convert list of monthly lists to simple yearly lists and sum
        ylist = [j for i in yearlist for j in i]
        annualsum = arcpy.sa.CellStatistics(ylist, statistics_type = "SUM")
        annualsum.save(r'C:/data/rainfall/aggr/'+str(year)+'.tif')
        print str(y)+" annual rainfall total done!"
        del annualsum
     print "ALL DONE!"

The code runs fine initially but crashes while saving the output raster ("annualsum") at the '1988' loop point. No specific error message, just a pop up that says "pythonw.exe has stopped working." When I alter the begin date to restart after '1988' it would crash again at the '1994', rinse repeat with 2000, 2006 and 2012 points - exactly 6 year intervals. I was convinced it was a memory issue since I am using 32 bit python/ArcGIS but I thought deleting the raster object after saving would solve the problem. While I finally got it all done by restarting after each crash, and doing the 'crash' years separately.

I use 32 bit because I only have the 1 year student/evaluation license for ArcGIS. My pc is however an i5 with 16GB of RAM.

closed as off-topic by PolyGeo May 27 '17 at 5:32

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This problem cannot or can no longer be reproduced. Changes to the system or to the asker's circumstances may have rendered the question obsolete, or the question does not include a procedure to enable potential answerers to reproduce the same symptoms. Such questions are off-topic as they are unlikely to help future readers, but editing them to include more details can lead to re-opening." – PolyGeo
If this question can be reworded to fit the rules in the help center, please edit the question.

  • I don't think you're quite a beginner.. Perhaps the problem is you're continually appending so continually building up a much bigger list. Try moving yearlist = [] inside the for year in years so you're not adding previous years to the current year. I don't think 32bit has anything to do with it as ArcGis caches to temp and processes in chunks; is your system drive filling up? Multiple crashes may cause your %tmp% or %temp% to fill up with old data that is not cleaned up... with no Esri product running look for ArcXXXX folders which can be deleted and your arcpy.env.workspace for GRIDS. – Michael Stimson May 25 '17 at 0:03
  • Excellent! worked! Even seemed to speed up the code. Thanks a lot. Just so you know, I wasn't being falsely modest when I called myself a beginner. True I am not entirely green but I still struggle with basic syntax (have to keep looking them up!) and I am generally unable to spot code flaws like the case in point. Hence the conservative self-assessment. But Cheers, :-). – Talsakun Sauvage May 25 '17 at 2:16
  • Take a look at 64-bit background geoprocessing. Will give you some more breathing room. – Paul May 25 '17 at 17:29