1

I am trying to get (print out) all unique values of a field named PhaseAtt in a group of shapefiles which all have that field with different unique values

import arcpy
from arcpy import env
print "Satrted"
env.workspace = "C:/Phases"

for feature in arcpy.ListFeatureClasses():
    print feature

Can you please let me know how to do that?

  • 1
    Use summary statistics resources.arcgis.com/en/help/main/10.2/index.html#//… then print that table, it's bound to be quicker than building a list and checking.. or do you want to do this all in arcpy the hard way to learn? – Michael Stimson Mar 16 '17 at 21:50
  • Hi Michael , I am actually talking about some shapefiles(over 60) and not table. so which one you thinks is faster to do? – user1106951 Mar 16 '17 at 21:52
  • 1
    Definitely summary statistics but I'm working on both.. get back to you in 5 minutes. – Michael Stimson Mar 16 '17 at 21:53
  • Could each feature class have the same set of unique numbers, or are you looking for unique numbers among the combined feature classes? – Aaron Mar 16 '17 at 23:03
4

There's at least two ways to do this, first just python:

import arcpy
from arcpy import env
print "Satrted"
env.workspace = "C:/Phases"

for feature in arcpy.ListFeatureClasses():
    print feature
    with arcpy.da.SearchCursor(feature,"PhaseAtt") as SCur:
      bList = []
      for row in SCur:
        if not row[0] in bList: # if not in list then add to list
          bList.append(row[0])
    for val in bList:
      print val

Using a searchcursor loop through all the features and add new values to the list if they're not found, this will take a long time as the list grows.

Secondly using Summary Statistics to create a temporary table with a Case Field of the one you want the unique values of.. the statistic isn't important but count works for all simple field types (int, double, text) then cursoring through this with the case field and listing:

import arcpy
from arcpy import env
print "Satrted"
env.workspace = "C:/Phases"
env.overwriteOutput = True

for feature in arcpy.ListFeatureClasses():
    print feature
    arcpy.Statistics_analysis(feature,'in_memory\\sumtable',[["PhaseAtt",'COUNT']],"PhaseAtt")
    with arcpy.da.SearchCursor('in_memory\\sumtable',"PhaseAtt") as SCur:
      for row in SCur:
        print row[0]

I'm using in_memory as a convenient place to put the temporary table but you could easily put it somewhere else; overwrite is on so it will be overwritten each time.

In either case bad things will happen if the field PhaseAtt is not found in the feature class, it would be good to test for this before proceeding.

Case sensitivity:

Both of these methods are case sensitive, 'hello' != 'HELLO', if you want to find the unique text case insensitive then modify the first method:

import arcpy
from arcpy import env
print "Satrted"
env.workspace = "C:/Phases"

for feature in arcpy.ListFeatureClasses():
    print feature
    with arcpy.da.SearchCursor(feature,"PhaseAtt") as SCur:
      bList = []
      for row in SCur:
        val = row[0].lower() # convert value to lower case
        if not val in bList: # if not in list then add to list
          bList.append(val)
    for val in bList:
      print val

Which could be a good reason to do it the hard way despite being slower.

  • Thanks Michael, is there any way to skip some feature classes which has not the PhaseAtt field? – user1106951 Mar 17 '17 at 4:23
  • Yes there is.. use if arcpy.ListFields(feature,'PhaseAtt'): on the line after print feature which will return True if the field is found and False if the field is not found. – Michael Stimson Mar 19 '17 at 21:48
6

Easiest way to use set:

>>> row=[x[0] for x in arcpy.da.TableToNumPyArray("Fitting","PROCESS")]
>>> set(row)

set([u'AIR RELEASE', u'Air Release', u'FLOW CONTROL', u'FLOW MEASURE', u'JOINT', u'ISOLATION', u'REDUCER', u'SIPHON', u'ANTIVACCUM', u'INCOMING SEWER', u'SCOUR', u'BYPASS', u'NON RETURN', u'INSPECTION'])
  • Agreed that set is the way to go here. I use it in my scripts to get unique values. – Fezter Mar 16 '17 at 22:20
  • related: gis.stackexchange.com/a/215629/7424 – Fezter Mar 16 '17 at 22:22
  • 2
    Nice and compact, get all the values and condense to the unique ones with set(). Is this faster that Summary Statistics? It might be. It's certainly a lot less lines of code. Also an opportunity to 'sanitize' x on-the-fly with lower(), upper(), trim() or replace()... – Michael Stimson Mar 16 '17 at 22:34
  • @Fezter it is most definitely duplicate question. – FelixIP Mar 16 '17 at 22:37
  • Nicely done! I was going to add this precisely as I've used this quite recently. – jbchurchill Mar 17 '17 at 19:57
0

Here is more information regarding the performance of the provided solutions:

  • Using TableToNumPyArray as below:

Code:

row=[x[0] for x in arcpy.da.TableToNumPyArray("Fitting","PROCESS")]

may give you the memory error if using Python 32bit (which comes with ArcGIS Desktop):

MemoryError: cannot allocate array memory

The number of features you will be able to convert into a numpy array would depend on the type of column you will be counting unique values from. Speaking of a middle size unicode string field, you may hit the limits somewhere around 500,000 features.

If you get this error, you need to use Python 64bit which will let you create a larger arrays in memory.

  • If your shapefile is not that big (mind that it has a .dbf attribute table limit of 2GB), you can use the da.SearchCursor which will outperform TableToNumPyArray:

The timing for ~3 mln points feature class (running on ~3 mln shapefile gave the same results - 10secs for SearchCursor and 15secs for numpy array):

import timeit

print timeit.timeit(stmt="print list(set(f[0] for f in arcpy.da.SearchCursor(fc, 'municipality')))",
              setup='''import arcpy; fc = r'C:\GIS\GDB_GeocodingOperational.gdb\PointAddress' ''', number=1)

>> 24.381339184

Doing the same thing using TableToNumPyArray:

import timeit
print timeit.timeit(stmt="print list(set(f[0] for f in arcpy.da.TableToNumPyArray(fc,'municipality')))",
                    setup='''import arcpy; fc = r'C:\GIS\GDB_GeocodingOperational.gdb\PointAddress' ''', number=1)
>> 38.08295354
  • Whenever you will construct sequences such as lists, consider using generators instead of lists constructed in a list comprehension or by iteratively appending values. This is because when you construct a list, it will be stored in your machine's memory. Constructing a large list may give you the out of memory error.

Code (rows doesn't have any significant memory footprint; initialized instantly):

>>> rows = (f[0] for f in arcpy.da.SearchCursor(fc, 'municipality'))
>>> rows
<generator object <genexpr> at 0x197CBE68> 

Code (rows have eaten a good part of your machine RAM, almost 500MB for ~3mln features feature class; it also takes time to construct):

>>> rows = [f[0] for f in arcpy.da.SearchCursor(fc, 'municipality')]
>>> type(rows)
<type 'list'>
  • If you will work with other data sources that are database based (file geodatabases or DBMS), you might like using sql_clause parameter of the SearchCursor which can use the DISTINCT prefix to get unique values from the database table column. In this way, you won't need to construct a list or an iterator and constructing a set from it.

However mind that using DISTINCT prefix will take longer time to run (2-3 times longer), however, if performance is not critical, you might like using it to write a cleaner code as I think it's unnecessary to pull all the data and then filter it if you can provide a filter already when pulling the data.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.