4

In one of my arcobjects-based GIS tools I have a process which takes an array of shapefile paths and uses them as inputs with the ESRI.ArcGIS.DataManagement.Merge tool. I run it via a Geoprocessor instance. I use this setup for various other gp tasks (such as Get Count and Calculate Field), but I have found that a Merge process takes a much longer period to process than if I had run it in ArcMap or ArcCatalog. For example, a process which I have clocked at around 3 hours took over 27 to complete when performed with this method in ArcObjects.

Has anyone experienced this before, and are there any known methods for expediting the process?

EDIT:
To be more explicit about the workflow taking place here:

  1. Create a backgroundworker object to do this in a separate thread to avoid UI lockup.
  2. List shapefile paths using Directory.GetFiles and append them to a semicolon-delimited path string for the merge operation
  3. Perform a Get Count on each Shapefile
  4. Use the path string to perform a merge operation.

I've tried using IGeoProcessor and Geoprocessor to see if there is any noticeable increase in performance, but to no avail. There are no tight loops involved (unless you consider rapid-fire Get Counts), and I certainly have no use for an EditSession. For some reason, however, the process just plods along at an amazingly slow pace.

It is worth noting that I am working with unreasonably large data sets (the road network of North America, for example), so I doubt any method that makes use of RAM as a workspace will be appropriate.

Improve this question
8
  • I wonder if you have a memory leak somewhere. If you monitor resource consumption in your tool and in arcmap, how does it compare? Are you calling IGpUtilities3.ReleaseInternals anywhere? – Kirk Kuykendall Dec 8 '11 at 3:44
  • My memory and process use never goes above a certain amount, but I have not measured it versus ArcMap. It certainly stays below what usually constitutes a memory crash of the program. I'm wondering if the heavy COM marshaling might be stacking up to slow me down. Have you ever run into a problem like that, Kirk? – Nathanus Dec 8 '11 at 15:17
  • Did you start an EditSession (i.e called StartEditing), that for sure would do it. Take a look at this answer I gave before: gis.stackexchange.com/questions/8186/… – Ragi Yaser Burhum Dec 12 '11 at 21:43
  • Let me be more explicit about the process of the code-- but as a quick answer, no I did not start an edit session. I've yet to use an edit session when working with ArcObjects. – Nathanus Dec 12 '11 at 21:58
  • Have you considered using Append instead of Merge? – blah238 Dec 12 '11 at 23:11
2

You could try setting an in-memory workspace to reduce I/O write times, as this may be slowing your process.

See an example of the SetEnvironmentValue() method below for using an in-memory workspace, assuming you've already set it up.

        Dim GP As New Geoprocessor
        Dim StreamLink As New StreamLink

        StreamLink.in_stream_raster = inStreamRaster
        StreamLink.in_flow_direction_raster = inFlowRaster
        StreamLink.out_raster = outRaster

        GP.SetEnvironmentValue("workspace", WorkspacePath)
        GP.TemporaryMapLayers = False
        GP.AddOutputsToMap = False
        GP.ClearMessages()
Improve this answer
3
  • At what point would an in-memory workspace improve the time? I'm merging dozens of gigs of data, so how would I put them in the workspace? Or are you suggesting I create the workspace for an output? Again, I do not believe my RAM, large as it is, could even begin to handle this file. – Nathanus Dec 6 '11 at 19:00
  • I've edited my answer to show an example of how you might go about implementing an in-memory workspace using the Geoprocessor in ArcObjects. Your RAM very well may be a limitation. You could always go the hardware route and look into a solid-state drive. – Radar Dec 6 '11 at 19:05
  • That is an idea, and it is a bit sad that we're still using HDDs, but the storage cost would be monumental. All that aside, I still wish I knew why running the process inside ArcWhatever makes it go so much faster. – Nathanus Dec 6 '11 at 21:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.