Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I would like to find out what it takes to create a custom implementation of an ArcGIS function. In particular, I would like to implement GeoAnalyst.ISurfaceOp2.Visibility() in order to get it to run faster. Right now, it is taking ~3 seconds per call to Visibility(). From my limited understanding, the bottleneck is the writing of temporary rasters to the file system. If this could be done in-memory, I suspect the processing time would decrease significantly. I am doing this in a .NET project, but solutions in any language are welcome.

share|improve this question
    
Why not just have the temporary rasters written to a RAM disk? Then you won't have to code the visibility op from scratch, which comes with its own risks and costs. –  whuber Nov 28 '11 at 20:41
    
That sounds great. How would I go about doing that? Is that not what @Radar states is not possible in the answer below? –  The Talking Walnut Nov 28 '11 at 20:52
4  
You install a RAM disk on your system; details vary with the OS. Then you point the ArcGIS scratch folder at it. You could do almost as well using an SSD especially if your rasters are huge or if you don't have much RAM. –  whuber Nov 28 '11 at 21:24
    
I like the RAM disk idea. An SSD is risky since they aren't great for constant read/write operations as you could wear it out quickly. –  Nick Ochoski Nov 28 '11 at 22:04
2  
A flash-based SSD will survive 1 - 5 million writes, @Radar, but DRAM SSDs won't "wear out" at all. Visit storagesearch.com for (lots) more info. –  whuber Nov 28 '11 at 22:33
show 2 more comments

2 Answers

up vote 5 down vote accepted

This answer memorializes and expands on some of the discussion in the comments. A RAM disk emulates an external disk drive using some of the RAM in a computing system. It can read and write at speeds comparable to in-memory caching, minus a little overhead for the translation protocols to convert disk-oriented commands to memory-oriented commands. A RAM disk is created by running special operating system-level software, a "device driver." Open source and free RAM disks are available for many operating systems, including Windows.

Therefore, one way to speed up a bottleneck due to intermediate disk I/O is to set up a RAM disk (buying the additional RAM if necessary) and place the scratch folder there. (This is usually a software setting.)

Another option is to install a high-end DRAM solid state device (SSD), which essentially is a block of RAM in separate packaging with an electronic interface to act like a disk drive. It installs in a computing system in place of a disk drive and will behave exactly like another disk drive without any additional software, but will read and write almost as fast as memory accesses. These are relatively expensive, but likely only a small one is needed even for very large intermediate raster storage.

Before taking any of these steps, it is important to profile the process to ascertain where the bottleneck really is. (Windows has been delivered with increasingly powerful profiling and monitoring apps in recent years, available in Win 7 as the Task Manager/Resource Monitor pair, and of course many similar apps are available for other OSes too.) Many systems are automatically configured, or can be configured, to cache disk reads and writes in RAM for short periods. Caching works almost the same as a RAM disk, but is probably even faster: the software thinks it is writing intermediate files to disk, but the OS writes them temporarily to RAM first, not accessing the disk, in the hope that soon the same data will be read back and deleted, in which case a physical write will never be necessary. Profiling the physical disk operations while the bogged-down software is running will indicate whether or not disk I/O is the bottleneck.

Given the amount of computation needed for any full visibility calculation (in a naive algorithm, every cell has to be inspected for visibility once for each viewpoint), one should at least suspect that computing speed, not disk I/O, may be the problem here. If that is the case, RAM disks or SSDs will be a waste of time and money. Instead, effort should be directed at analyzing and improving the underlying algorithm.

Some discussion of whether RAM disk performance helps ArcGIS has appeared in another thread.

share|improve this answer
    
+1 Great response. I think GIS folks should also keep an eye on general purpose use of the graphics processing unit, or GPGPU. I suspect most first person shooter games leverage the GPU to determine visibility of object from the shooter's location. Would be great to see an OpenCL implementation of this problem. Maybe if GIS used the GPU more, then it might catch the attention of kids who grew up playing video games. –  Kirk Kuykendall Nov 29 '11 at 16:27
    
Also I meant to mention the GPU is optimized to do things like hidden surface determination, which is similar to what ISurfaceOp2.Visibility does. –  Kirk Kuykendall Nov 29 '11 at 16:45
    
Good ideas, @Kirk. Manifold uses the NVIDIA GPUs (and has done so for several years). There's also independent research on this front; I have been part of some grant proposal efforts to develop GPU-based map algebra implementations. –  whuber Nov 29 '11 at 17:13
    
This is an excellent answer. I did the suggested profiling, and it appeared that there might be a disk I/O bottleneck. So, I implemented a solution that used a RAM disk (I chose IMDisk). In the end, this did not reduce the time to complete the process. –  The Talking Walnut Mar 6 '12 at 14:50
    
I'm sorry to hear there was no improvement. Visibility is an expensive calculation: in principle, every cell in the DEM has to be inspected for each observer and the inspection process may have to look at an entire line of cells between the observer and the DEM cell: that can mean billions of calculations per observer even on a modest (megapixel) grid. If computation time is the problem, your two best options are likely (1) parallelize the work: split observers across workstations, compute visibilities, combine the results; and (2) write your own visibility code. –  whuber Mar 6 '12 at 15:34
add comment

Unfortunately, you cannot set the IRasterAnalysisEnvironment to work with an in-memory workspace.

This thread addresses the issue.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.