3

I have the following table with distance values (NEAR_DIST) after using the "Near" tool in ArcGIS:

enter image description here

I am looking for a method of getting the following statistical information/statement, for example: "80% of the values from the field "NEAR_DIST" fall inside the range of 40-500, 90% in the range of 30-600,..."

With the normal statistical information i do not get much information, just the sum, min, etc...:

enter image description here

Is there a tool in ArcGIS to achieve that or how would you deal with that?

I would like to remove the "outliers" (for example the 868 and 528 (which are too high, and also the very low records), to get a better frequency distribution (histogramm)...therefore i would like to automatically find the "break values" which i can use to filter my table.

Is there a way in ArcGIS to do this, or any statistically method?

10
+50

Here is a python solution, using arcpy to access the data and numpy to calculate the statistical values.

Using arcpy.da.SearchCursor() write the values to a list. Use python.numpy.percentile() to find the threshold percentile values that you want to use to identify outliers, lets take your example and drop the lowest 10% and highest 10% of values.

If you actually want to remove the records, include the last block of code, otherwise the print statement may be enough.

import arcpy, numpy as np

# input shape file or feature class
in_shp = r'd:\near_features.shp'

# write distance vals to list
vals = []
with arcpy.da.SearchCursor(in_shp, ['NEAR_DIST']) as cur:
    for row in cur:
        vals.append(cur[0])

# identify 10th and 90th percentiles (leaving 80% of values remaining)
percentile_10 = np.percentile(vals, 10)
percentile_90 = np.percentile(vals, 90)

# print percentiles
print '80% of records are between {} and {} distance'.format(percentile_10, percentile_90)

# delete records that exceed these threshold values
with arcpy.da.UpdateCursor(in_shp, ['NEAR_DIST']) as cur:
    for row in cur:
        if row[0] <= percentile_10:
            cur.deleteRow()
        elif row[0] >= percentile_90:
            cur.deleteRow()
  • 2
    +1, great use of numpy. Instead of building a list and then appending each record to it, you can list a list comprehension to build it all in 1 go: vals = [row[0] for row in arcpy.da.SearchCursor(in_shp, ['NEAR_DIST'])] – Paul Jun 23 '16 at 15:44
2

If you are looking for an automated way to do this, or just a way to do this at all, you could:

  1. Perform a query on your table WHERE: NEAR_DIST >= 40 AND NEAR_DIST <=500

  2. Divide how many records are returned (e.g. 800) by how many total records you have (e.g. 1000). For example 800/1000 * 100 = 80%

If you want to automate this you could do this in ModelBuilder. Just perform each query,count results, do the math and write it to a text file.

  • Thank you for your feedback. For a better understanding: I would like to remove the "outliers" (for example the 868 and 528 (which are too high, and also the very low records), to get a better frequency distribution (histogramm)...therefore i would like to automatically find the "break values" which i can use to filter my table... Is there a way in ArcGIS to do this, or any statistically method? – Duddel Jun 23 '16 at 7:26
  • I see in the edits that you added the outlier issue to your post. Looks like somebody has an automated solution so I'll just refer you to that :-) – alexGIS Jun 24 '16 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.