Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

This question has already been posed to the ArcGIS Geoprocessing forum but I have had no responses there.

I have a 3.5 million polygon layer (cadastre) stored in Oracle via ArcSDE. I use a polygon feature chosen by an end user from a second layer to copy out the cadastral polygons that intersect it (via Copy Features). I can get this to work fine for most polygons, and get the few seconds to few minutes response I need, by using the polygon the user chooses to set the geoprocessing extent (arcpy.env.extent) prior to executing Copy Features.

However, for a few polygon features that the user might choose (think petroleum or gas pipeline corridor), the number of polygons that would be copied out can number up to about 200,000. These can be copied in about 15 mins, but rather than get all of these back so that I can write them into a very bloated PDF report, I would like to save time and pages by having the option to set a maximum number of features that can be copied out i.e. be able to set a MaxRecordCount = 500 that would stop the Copy Features once 500 had been copied.

Is there a way to tell Copy Features to only copy up to a maximum number of features?

Alternatively, is there a fast way to get a count of how many features in an ArcSDE layer are within the current geoprocessing extent?

UPDATE: No real progress made but the results and code below may illustrate better how long it takes to use SelectLayerByLocation and GetCount (at least at ArcGIS 10.0) against the 3.5 million polygon ArcSDE layer even when the tenement chosen only overlaps 9,000 polygons (original example takes 10x as long).

This is the output:

Copying one tenement to in_memory takes 2.87037144971 seconds
Setting arcpy.env.extent to that one tenement (in_memory)
SelectLayerByLocation takes 64.4129756363 seconds to intersect cadastre with this one tenement (in_memory)
8961 cadastral parcels found in Search Area
GetCount takes 206.410587153 seconds to come up with that number

This is the code:

import arcpy
import time

mxd = arcpy.mapping.MapDocument("C:\\MXDS\\CustRep\\Maps\\Search_layers_V1.mxd")
df = arcpy.mapping.ListDataFrames(mxd,"Layers")[0]
searchAreaLayer = arcpy.mapping.ListLayers(mxd,"Exploration Permits Petroleum (EPP)",df)[0]
searchAreaLayer.definitionQuery = "TENURENO = 644"
start = time.clock()
arcpy.CopyFeatures_management(searchAreaLayer,"in_memory/SearchArea")
elapsed = (time.clock() - start)
print "Copying one tenement to in_memory takes " + str(elapsed) + " seconds"

print "Setting arcpy.env.extent to that one tenement (in_memory)"
arcpy.env.extent = "in_memory/SearchArea"

cadastreLayer = arcpy.mapping.ListLayers(mxd,"Lot and Plan",df)[0]

arcpy.MakeFeatureLayer_management("in_memory/SearchArea","SearchArea")

start = time.clock()
arcpy.SelectLayerByLocation_management(cadastreLayer,"INTERSECT","SearchArea")
elapsed = (time.clock() - start)
print "SelectLayerByLocation takes " + str(elapsed) + " seconds to intersect cadastre with this one tenement (in_memory)"

start = time.clock()
result = arcpy.GetCount_management(cadastreLayer)
numRowsInSearchArea = int(result.getOutput(0))
print str(numRowsInSearchArea) + " cadastral parcels found in Search Area"
elapsed = (time.clock() - start)
print "GetCount takes " + str(elapsed) + " seconds to come up with that number"
share|improve this question
add comment

3 Answers

Why not just do a Select Layer by Location instead of setting the geoprocessing extent? Alternatively, you could use a SearchCursor and stop after the 500th row.

share|improve this answer
    
These are useful thoughts, and I will need to do some timings later in the week to be sure that I have good reasons for no longer using SelectLayerByLocation, but I went to the CopyFeatures within env.extent method to speed up performance (sometimes dramatically) albeit on much smaller extents/datasets. I think it is the time of opening a SearchCursor on 200,000 rows from 3.5 million that is the killer (hopefully will be much better in 10.1). The CopyFeatures is done to an in_memory workspace so that a subsequent Intersect flies. If SelectLayerByLocation worked that might not be necessary. –  PolyGeo Feb 26 '12 at 3:55
    
If your data is stored in a native spatial storage format such as ST_Geometry or Oracle Spatial it may be worth looking into how to write a SQL query to get the data you need for your report and avoid the Geoprocessor object bloat. –  blah238 Feb 26 '12 at 6:46
    
I think my client's data is stored in SDE Binary but understand that it is destined to end up in either ST_Geometry or SDO_Geometry so that may be an option in the future, if not already. At the moment the issue in this question has been raised but as yet it does not appear to be critical. Your advice is much appreciated. –  PolyGeo Feb 26 '12 at 11:09
    
I'm keen to explore whether a simple SQL expression can be passed with the extent values from Python/ArcPy to Oracle Spatial and have it return how many features are found to overlap it within a second or two. I'll happily award the Answer to some code that explains the steps/code to do that. –  PolyGeo Mar 21 '12 at 6:50
add comment
up vote 2 down vote accepted

This has now been requested as an enhancement at ArcGIS Ideas

UPDATE: Kim Ollivier just pointed me in the direction of what will be a viable workaround for me, inspired by ArcStorm, in the ArcGIS Geoprocessing forum. Many thanks Kim!

share|improve this answer
add comment

When using arcpy, you are not going for performance, you are going for convenience. This is explained further in a previous answer about arcgis scripting

share|improve this answer
    
Thanks Ragi, I agree that the convenience of arcpy can compromise performance, and your post makes many excellent points, but in this case I have accepted arcpy's performance and am just looking for a parameter to be added to a tool to enable it to stop copying once a maximum number or features has been reached. I'm not asking for it to copy faster, just a way to stop it copying more records than I know that I want to see. –  PolyGeo May 30 '12 at 5:27
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.