Arcpy Get Unique Value

Question

I am trying to get (print out) all unique values of a field named PhaseAtt in a group of shapefiles which all have that field with different unique values

import arcpy
from arcpy import env
print "Satrted"
env.workspace = "C:/Phases"

for feature in arcpy.ListFeatureClasses():
    print feature

Can you please let me know how to do that?

Answer 1

Easiest way to use set:

>>> row=[x[0] for x in arcpy.da.TableToNumPyArray("Fitting","PROCESS")]
>>> set(row)

set([u'AIR RELEASE', u'Air Release', u'FLOW CONTROL', u'FLOW MEASURE', u'JOINT', u'ISOLATION', u'REDUCER', u'SIPHON', u'ANTIVACCUM', u'INCOMING SEWER', u'SCOUR', u'BYPASS', u'NON RETURN', u'INSPECTION'])

Answer 2

There's at least two ways to do this, first just python:

import arcpy
from arcpy import env
print "Satrted"
env.workspace = "C:/Phases"

for feature in arcpy.ListFeatureClasses():
    print feature
    with arcpy.da.SearchCursor(feature,"PhaseAtt") as SCur:
      bList = []
      for row in SCur:
        if not row[0] in bList: # if not in list then add to list
          bList.append(row[0])
    for val in bList:
      print val

Using a searchcursor loop through all the features and add new values to the list if they're not found, this will take a long time as the list grows.

Secondly using Summary Statistics to create a temporary table with a Case Field of the one you want the unique values of.. the statistic isn't important but count works for all simple field types (int, double, text) then cursoring through this with the case field and listing:

import arcpy
from arcpy import env
print "Satrted"
env.workspace = "C:/Phases"
env.overwriteOutput = True

for feature in arcpy.ListFeatureClasses():
    print feature
    arcpy.Statistics_analysis(feature,'in_memory\\sumtable',[["PhaseAtt",'COUNT']],"PhaseAtt")
    with arcpy.da.SearchCursor('in_memory\\sumtable',"PhaseAtt") as SCur:
      for row in SCur:
        print row[0]

I'm using in_memory as a convenient place to put the temporary table but you could easily put it somewhere else; overwrite is on so it will be overwritten each time.

In either case bad things will happen if the field PhaseAtt is not found in the feature class, it would be good to test for this before proceeding.

Case sensitivity:

Both of these methods are case sensitive, 'hello' != 'HELLO', if you want to find the unique text case insensitive then modify the first method:

import arcpy
from arcpy import env
print "Satrted"
env.workspace = "C:/Phases"

for feature in arcpy.ListFeatureClasses():
    print feature
    with arcpy.da.SearchCursor(feature,"PhaseAtt") as SCur:
      bList = []
      for row in SCur:
        val = row[0].lower() # convert value to lower case
        if not val in bList: # if not in list then add to list
          bList.append(val)
    for val in bList:
      print val

Which could be a good reason to do it the hard way despite being slower.

Answer 3

Here is more information regarding the performance of the provided solutions:

• Using TableToNumPyArray as below:

Code:

row=[x[0] for x in arcpy.da.TableToNumPyArray("Fitting","PROCESS")]

may give you the memory error if using Python 32bit (which comes with ArcGIS Desktop):

MemoryError: cannot allocate array memory

The number of features you will be able to convert into a numpy array would depend on the type of column you will be counting unique values from. Speaking of a middle size unicode string field, you may hit the limits somewhere around 500,000 features.

If you get this error, you need to use Python 64bit which will let you create a larger arrays in memory.

• If your shapefile is not that big (mind that it has a .dbf attribute table limit of 2GB), you can use the da.SearchCursor which will outperform TableToNumPyArray:

The timing for ~3 mln points feature class (running on ~3 mln shapefile gave the same results - 10secs for SearchCursor and 15secs for numpy array):

import timeit

print timeit.timeit(stmt="print list(set(f[0] for f in arcpy.da.SearchCursor(fc, 'municipality')))",
              setup='''import arcpy; fc = r'C:\GIS\GDB_GeocodingOperational.gdb\PointAddress' ''', number=1)

>> 24.381339184

Doing the same thing using TableToNumPyArray:

import timeit
print timeit.timeit(stmt="print list(set(f[0] for f in arcpy.da.TableToNumPyArray(fc,'municipality')))",
                    setup='''import arcpy; fc = r'C:\GIS\GDB_GeocodingOperational.gdb\PointAddress' ''', number=1)
>> 38.08295354

• Whenever you will construct sequences such as lists, consider using generators instead of lists constructed in a list comprehension or by iteratively appending values. This is because when you construct a list, it will be stored in your machine's memory. Constructing a large list may give you the out of memory error.

Code (rows doesn't have any significant memory footprint; initialized instantly):

>>> rows = (f[0] for f in arcpy.da.SearchCursor(fc, 'municipality'))
>>> rows
<generator object <genexpr> at 0x197CBE68> 

Code (rows have eaten a good part of your machine RAM, almost 500MB for ~3mln features feature class; it also takes time to construct):

>>> rows = [f[0] for f in arcpy.da.SearchCursor(fc, 'municipality')]
>>> type(rows)
<type 'list'>

• If you will work with other data sources that are database based (file geodatabases or DBMS), you might like using sql_clause parameter of the SearchCursor which can use the DISTINCT prefix to get unique values from the database table column. In this way, you won't need to construct a list or an iterator and constructing a set from it.

However mind that using DISTINCT prefix will take longer time to run (2-3 times longer), however, if performance is not critical, you might like using it to write a cleaner code as I think it's unnecessary to pull all the data and then filter it if you can provide a filter already when pulling the data.