0

Been attempting (with no success so far) to implement the random forest classification in ArcMap on a reasonably large image. If I throw a raster with say 20-30 bands in it, works fine, just takes time. However my aim is really to get the tool to classify a 250-band raster, to really see how well it works on our data. Splitting the 250-band raster into multiple smaller band rasters defeats the purpose of what we're trying to achieve.

However this means our input raster (ESRI GRID stack format) is over 80Gb in size, and ArcGIS can't seem to handle it. So I am basically wondering if anyone has actually managed to implement any of the image classification tools on relatively large rasters? If not, is anyone aware of what the practical limitations of these tools are?

I have access to both Desktop 10.7 and Pro, run into problems with the large rasters on both and am running a desktop PC with decent specs.

Out of frustration, I have also tried to implement random forests using just a python script outside of the ESRI environment on a TIFF version of the GRID stack. Code works fine on small'ish inputs, but computer said "no" for the big files (the TIFF file is nearly 70 times the size of the GRID stack...).

Any advice? No access to a supercomputer at the moment, but if that's your advice, and you've successfully got it to work on one, happy to hear about it!

  • Python is under-the-hood with the ESRI implementation so, no surprise that both approaches failed. I have successfully adressed problems this large in R but, it is slow. Rather than subsetting bands you could tile your raster into subsection that can run. – Jeffrey Evans Oct 22 '19 at 1:41
  • It could be a problem with your environment %TEMP% and/or %TMP%. Each instance of Catalog or Map creates a folder in your temp folders.. you could be running out of space on your system drive. If you have another working drive installed try changing your operating system TEMP and TMP to point to folders on a non-system drive. Implicitly set your scratchWorkspace to os.environ.get('TEMP') to be sure that the process isn't writing to your system drive. According to desktop.arcgis.com/en/arcmap/latest/manage-data/… the size limit is 4 million squared. – Michael Stimson Oct 22 '19 at 1:49
  • I suspect tiling the image is a problem due to the nature of the both the training data distribution and the actual input raster to be classified. Will investigate the R implementation though! Don't care if it is slow, as long as it works :) – Arianne Ford Oct 22 '19 at 1:58
  • I empty my scratch workspaces pretty frequently and I still have several Tb free space on the relevant drive, so that's unlikely to be the cause. – Arianne Ford Oct 22 '19 at 2:00
0

In situations such as these, you have three options:

  1. Tile your input imagery into smaller pieces
  2. Choose a computer system with increased specs
  3. Attempt to use different software which hopefully implements a version of #1 under the hood

I always opt to solve these problems by tiling the imagery into smaller pieces (option 1). Many folks in the geospatial machine learning field use R or Python to perform the land cover classification, where you can run parallel operations on cloud resources such as AWS EC2 instances if needed. Here is a an example of using a random forests implementation in the Python scikit-learn library for land cover classification of satellite imagery.

| improve this answer | |
  • I seem to have successfully implemented it in R. Slow, but functional, which is all I really need. Attempts to do it in python kept failing. – Arianne Ford Oct 23 '19 at 18:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.