2

I have a question on speeding up “Mosaic to New Raster” or finding an alternative. After completing a suitability analysis we are trying to define movement corridors. We used a modified CircuitScape that left 4 folders, each containing 1080 raster tiles across our study area. We now need combine all 1080 rasters in each folder then combine these 4 combined rasters into a final raster which will be analyzed for paths.

I tried a python script that just wouldn’t work so I went on to Model Builder. It iterates through the grids in the folder, collects the values (raster names), then mosaics all to a new raster. It has been running for 5 days now and the results window indicates it is working. I’m hoping it will conclude in the next day or 2. But, we still have 3 more folders, each with 1080 rasters, that need to be mosaicked. So….is there a way to speed up Mosaic to New Raster or is there an alternative I could use to produce the final grid?

The cell size is 270 meters, 540 x 540 columns and rows, 1 band, 32 bit floating.

Improve this question
6
  • In the middle of a processing run isn't the best time to experiment with this, but mosaicking across before down has often given me the best possible performance, since the "across" is more expensive, I/O wise (due to interleave), while down is often simple file concatenation. The trick is organizing your tiles for piecewise assembly, then mosaicking the intermediate down bands for greatest efficiency.
    – Vince
    Jan 12, 2016 at 19:11
  • I think it is going to take a long time one way or another. Either waiting for each mosaic or in setting up the data in a more organized structure beforehand as you suggest. Right now the mosaic is jumping around rather than proceeding smoothly from one tile to the next.
    – Charleen
    Jan 12, 2016 at 21:18
  • The jumping around is increasing the I/O. If you don't have sufficient independent disks, you can collect some metadata on the remaining folders' tiles, and organize them while waiting. Running concurrently without independent disks would be slower still, increasing fragmentation of the intermediate files.
    – Vince
    Jan 12, 2016 at 21:31
  • Thank you. I think this is probably the wisest action. It may take some time to sort out the structure and set up a mosaic plan but in the long run it will save time.
    – Charleen
    Jan 12, 2016 at 22:12
  • Is it critical to keep 32 bit floating point raster data? Can the raster data you wish to mosaic be 8 bit unsigned integer values (i.e. 0-255)? Do you have access to Erdas Imagine?
    – Aaron ♦
    Jan 13, 2016 at 3:03

1 Answer 1

Reset to default
2

Assuming you have a regular collection of tiles, distributed in two dimensions like a grid, and your mosaic function only takes pairs of images to produce a new concatenated image, there are four ways to compile a mosaic:

  1. Random order
  2. Row-major order
  3. Column-major order
  4. Hybrid row-major

With random order, the target image will get large fast, with large NODATA holes using space, and slowing the compilation.

With simple row-major order, the target image will increase by one image, interleaved by the I/O cost of reading the intermediate image (1+1=2, 2+1=3, 3+1=4, ..., 7+1=8). An optimized form would be to pair neighbors (1+1=2, 1+1=2, ..., 2+2=4, 2+2=4, 4+4=8), which for eight neighboring images would require the data in each to be read completely three times (24 reads vice 35). Once the next row is breached, you have a partial row of empty space to move until filled.

Column-major order is really consecutive concatenation, which simplifies access (less head movement), but if you do it first, then you've used up your best trick too soon.

Finally, there's the hybrid approach, where row-major sets are assembled in pairs (optimized form), then the row sets are paired with concatenation. This approach uses more space, since you have to keep more large intermediate products nearby, but it should minimize I/O operations.

Improve this answer
1
  • Thanks for the input. It seem the obvious message is that I need to examine the file naming convention and determine a plan of action to limit the processing.
    – Charleen
    Jan 13, 2016 at 16:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.