3

I've built a model in arcgis 10.5 ModelBuilder to represent a process that I need to run for thousands of polygons and input raster combinations. The problem is that to this model, for one polygon, takes approximately 8min 57 seconds to complete. When I work out how many iterations I will need, that is around 24 days of running time! Way too long. The input raster sizes are approximately 0.008 decimal degrees cell size covering the entire globe.

Here is the model currently: enter image description here

For each of the zonal statistics and raster calculator outputs I tried to change the output from writing to a geodatabase to "in_memory". But the model crashes after the first zonal statistic on the first polygon with a message about "out of memory".

Here is what the model is doing:

  1. For each country, calculated the total population (population raster)
  2. weight average the PM_raster based on the polygons total populations (=(pm_raster * (Population_raster/total_pop))
  3. Run zonal statistics to sum the weighted PM_weighted%n% and that is the average PM weighted on the countries population.

So my questions are:

  1. Can I make this model run quicker by using "in_memory"? Why did it crash at the first use of it in the model?
  2. Is there a quicker way to get these results away from using this model?
  • Unless you have 64-bit background GP and copious amounts of RAM, storing a global dataset in memory won't work well. How large is the dataset on disk? – Paul Jan 5 '17 at 0:02
  • the population data is ~480mb and the PM data is ~250mb – andrewdawes07 Jan 5 '17 at 0:07
2

So you are trying to run a set of zonal stats on a base raster that is global in coverage? That's your problem you are processing a lot of data, several billion cells on each operation, so it is not surprising that it will take that long.

Looking at your model it appears you are processing at a country level? So for an individual country you are not interested about population outside it?

The solution to this problem is that you need to set the extent of you zonal/calculate tools to the extent of the country. This will massively reduce the amount of cells it needs to process during each iteration.

What you need to do is tweak your model by doing the following:

  • On each iteration copy the country polygon to a new dataset, this will provide a dataset for which you can use it's extent.
  • Link the copied polygon (now a separate dataset) to you zonal stats tool as an extent environment property.
  • Ensure snap raster environment is set to the base population raster

You should observe a marked improvement in processing time.

  • This reduced the running time to a sufficient amount. On average around <10 minutes for all 189 countries. Thanks for the advice! – andrewdawes07 Jan 6 '17 at 3:43
  • 24 days to less than 10 minutes = awesome! – Hornbydd Jan 6 '17 at 15:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.