1

I'm using a version of Solution 2 of the selected answer here to build a new shapefile of all points that fall within a given polygon object (i.e. a clip).

When run against 27K points, it takes about 9 minutes. In arcpy it takes < 10 seconds.

I cant find any documentation on the Clip_analysis() code (is it proprietary?) so I am curious as to how it can execute the checks of 27K so fast?

Is it multi-processed? Is it calling some type of map() on the iterable of points?

Improve this question
18
  • All Esri code is proprietary, you will not find the source anywhere unless you are a developer working for Esri. However, I have found OGR2OGR -clipsrc to be even faster, especially with polylines. It is no surprise to me that assembly instructions (executables) are faster than python (an interpretative language), it just highlights the efficiency of low level code (C, C++, fortran, assembly) vs high level languages (python, VBA). You also have to be aware that Esri has been around since the 80s' and has optimized their base code through years of coding.
    – Michael Stimson
    Sep 5, 2017 at 22:10
  • Very common reason for such big difference in speed is that the fast solution is using spatial index but the slow one makes Cartesian product. Differences between languages are usually nominal compared with good vs bad general logic.
    – user30184
    Sep 6, 2017 at 4:48
  • @user30184 difference between similar generation languages is usually nominal when you compare interpreter vs interpreter (4GL) but not so much when comparing compiled vs compiled (3GL). C++ performs better than C#/VB.net under CRT. The performance difference between python/Ruby/Perl and C++ is enormous; It's the price you pay for simplicity in code as more overhead is required to interpret and validate during run time.
    – Michael Stimson
    Sep 6, 2017 at 21:15
  • 2
    I mean that it also happens that programmer is using fast language but writes ineffective code. Then the primary solution is not to change the language. This difference 10 sec vs. 9 minutes can't be explained by using something else vs. Python.
    – user30184
    Sep 6, 2017 at 21:23
  • 2
    The Python solution in the link takes one multipolygon at a time and tests each point "is it within me", thus 27000 operations for each multipolygon. The solution that utilises spatial index would take a multipolygon, select only points which fall inside its bounding box from the index which is very fast, and runs the "within" test only for those candidates. Much less to test, much faster, in any language.
    – user30184
    Sep 7, 2017 at 5:46

2 Answers 2

Reset to default
3

My understanding is that the polygon intersecting algorithms used by the Clip tool are written at a much lower level than Python.

The Clip tool is then made available within the ArcPy site package via a Python wrapper.

Improve this answer
2

Lack of a spatial index. You're evaluating every point for intersection whether or not it's likely to be a candidate.

If you want to do it in Python, you should use something like Rtree to quickly weed out which points might intersect your clipping polygon. For a pure Python spatial index you could try PyQuadTree.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.