Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I'm trying to create some time estimates for imagery caching in ArcGIS 10. Does anyone know of any forumlas or ratios to calculate the time it takes to generate overviews for a mosaic dataset?

Does it have anything to do with the area, resolution, file size etc of the actual imagery, or is there even a way to calculate a time estimate for overview generation?

Thanks

UPDATE: Does anyone have experience in building overviews on massive mosaic dataset (upwards of 9,000 images) in arcGIS 10?

I'm wondering if there is a way to speed up the process ... right now it looks like arcGIS is only using 1 core to build the overviews. I have a 12 core machine. Obviously things would go much faster if we can utilize more cores.

My original estimates are blown out of the water ... I'm looking at over 3 weeks to complete this process.

Any ideas?

share|improve this question
    
I am curious as to what are you trying to do with the mosaic dataset once it is done and why are you choosing to use a mosaic dataset for this specific use case? –  Ragi Yaser Burhum Feb 23 '12 at 15:24
    
Ragi - I will be caching the imagery. The mosaic dataset allows me to manage the imagery spanning multiple years. I can adjust the footprints to eliminate bounding black pixels and also ensure that only the most recent imagery will be displayed. –  Gshock Feb 23 '12 at 15:38
    
Mmm. OK. I was trying to figure out if you could use GDAL directly instead (since it is what ArcGIS uses internally for raster access) to paralellize the process somehow. The operation sounds mostly I/O bound though. Is this correct? If so, for two weeks, you may be better of buying an SSD and processingthe dataset there (if it fits) –  Ragi Yaser Burhum Feb 23 '12 at 15:51
    
Unfortunately I don't think the boss is going to approve the purchase of an SSD! Definitely reading a lot with this process. Building overviews is all about resampling tiles. So at the lowest overview level we read in the affected tiles first, then compute the resampling of the pixels, and output a new image. Then repeat for the next set of images. Do you know of any links directing to the use of GDAL in generating overviews? Anything is worth a try at this point. –  Gshock Feb 23 '12 at 16:46
    
40+hrs of your time creating a parallel process vs, say, 8hrs of your time walking to the store and buying one and configuring it. Works every time :) –  Ragi Yaser Burhum Feb 23 '12 at 16:51
show 6 more comments

1 Answer 1

up vote 2 down vote accepted

We saw overview processing times of similar magnitude when reading/writing to/from a network drive. Switching to any local drive, even an external USB 2.0, reduced what had been a 2-week process to more like 8 hours. You shouldn't need an SSD. It takes some time to transfer all that data locally and back again when you're done, and you have to do a "repair" to get the paths right on the Mosaic Dataset if you move things around this way, but it's still basically an overnight process and a few minutes of setup on either end rather than weeks long.

share|improve this answer
    
Thanks somemaps. Yea, I've found that the bottleneck in my overview creation was read/write speed rather than CPU speed. I moved from a network drive to a 10,000rpm hard drive and saw overview creation go from 13 days down to under 4 days. However, I was still on the hook for transferring the data to an external drive after that. I will definitely recommend sata/firewire connections for anyone dealing with large volumes of imagery data –  Gshock Apr 26 '12 at 16:16
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.