I have a large (1GB+) raster that I am using as the base for the generation of an equipotential surface. This is to assess the accuracy of DEMs derived from another source. In situ observation is impossible and the equipotential surface allows me to model in error.

I have an ArcGIS model that generates a single realization of the surface. The model is outputting beautifully and generates a 1GB raster realization. I am going to generate 100 of these surfaces to start to build a statistically significant pool of realizations (Monte Carlo simulation).

How would you go about storing the rasters? How would you visualize them to allow for the selection of the most likely realization (the one at the center of the monte carlo significance envelope)?

Here are the options I have considered:

  1. Convert to point and store each realization as a column in the attribute table. Here I am concerned about the shapefile size limit (2GB).

  2. Convert to CSV using gdal2xyz. I tried the conversion and the output of a single raster was 12GB! This is way too inefficient.

  3. I considered using R but am not sure which package would facilitate this type of research (spatstat perhaps).

  4. Graph either the histograms or cumulative distribution functions of each raster realization for visual identification.

closed as too broad by PolyGeo Sep 4 '16 at 0:31

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • With this question still on the unanswered list, I think we are overdue in trying to narrow its focus. By spreading the "responsibility" for answering across multiple subcommunities within GIS SE (ArcGIS for Desktop, R, GDAL, etc) it does not focus attention on any single of those subcommunities to either say "this is how to do it" or "it cannot be done" or "this is our best effort towards it" or ... Can you choose which GIS product you think is the most likely to yield an answer and focus this question on that, please? – PolyGeo Sep 4 '16 at 0:28
  • In the meantime I am going to place it on hold for being too broad. If you can identify the GIS product as I have suggested then I will be happy to re-open it and add a bounty to it. – PolyGeo Sep 4 '16 at 0:31
  • @PolyGeo - The question is ~4 years old. Fine to have on hold / removing it. The 1GB size is not longer any issue - this is easily solvable using an HDF (or NetCDF) data store. – Jay Laura Sep 7 '16 at 18:01

As commented by @jbosq:

I've done the inverse in R, sampling from a known distribution around the values from a raster. Mine was much smaller and some of the cell values were the same so I was able to generate matrices with each unique value for cells (column) along the probabilities for those values (row) and then join those back to the original locations. Another possibility might be to discretize your results to store counts for ranges rather than unique values.

Not the answer you're looking for? Browse other questions tagged or ask your own question.