3

Currently I am using this code to extract the "Number of fields" and "Feature Count" for the shapefiles listed in the input csv

import csv
import arcpy
from os import path
csvpath = r"D:\QAQC\List2.csv"
output = r'D:\Test\Test_2.csv'
csvdir = path.split(csvpath)[0]

with open(csvpath, "r") as csvfile:
    fcs = [r[0] for r in csv.reader(csvfile)]
    with open(output, 'wb') as f:
        for fc in fcs:
            count = len(arcpy.ListFields(fc))
            fcount = arcpy.GetCount_management(fc)
            csvwriter = csv.writer(f,delimiter=',')
            csvwriter.writerow([fc,count,fcount])

    f.close()

Along with this I would like to extract number of attributes of each shapefile. For eg; if Africa1.shp has 4 fields and 40 features, theoretically there can be 160 attribute values to Africa1.shp (but some features might not have few attributes).

So I want the output like, Africa1.shp 4 40 160

Don't know whether such a function exists in ArcPy

I want the actual number of attribute values. To explain things, i've mentioned 40 * 4 = 160 attributes

Edit:

Based on the answers below and through this Q&A Whats a Python Script/Method to use to get the number of fields with data in an attribute table?, i've almost got what i want. Two issues. 1) I don't want FID & Geometry field to get calculated. For this, ive referred to this one - https://geonet.esri.com/thread/75885 - and it works.

Only issue is couldn't able to save it to a csv file. If i remove this one piece of code

field_list = [f.name for f in arcpy.ListFields(fc) if f.type not in types and f.name.lower() not in other_geom]

and use count = arcpy.ListFields(fc), i can able to save it as csv. Don't know how to figure this out

2) Count results for fields with no null/blanks values are like, row after row. It's getting listed for all the fields. Instead how to sum the count results ?

Here is the code

import csv
import arcpy
from os import path
csvpath = r"D:\RTXQAQC\Open.csv"
csvdir = path.split(csvpath)[0]
output = r'D:\Test\Test_323.csv'
with open(csvpath, "r") as csvfile:
    fcs = [r[0] for r in csv.reader(csvfile)]
    with open(output, 'wb') as f:
        for fc in fcs:

            types = ['OID','Geometry']  
            other_geom = ['shape_area','shape_length']  

            field_list = [f.name for f in arcpy.ListFields(fc) if f.type not in types and f.name.lower() not in other_geom]      

            # Get list of fields

            count = len(field_list)
            fcount = arcpy.GetCount_management(fc)

            # Set up dictionary for the fields
            countdict = {}
            for field in field_list:
                countdict[field] = 0
            # Loop through Table and see which fields are NULL or Blank
            cur = arcpy.SearchCursor(fc)
            for row in cur:
                for field in field_list:
                    if not row.getValue(field) in [None, " ", ""]:
                        countdict[field] = countdict[field] + 1

            # Print out results - shows field name and number of non NULL/Blank fields
            for f in countdict.keys():

                print (fc,count,fcount,f,countdict[f])

            csvwriter = csv.writer(f,delimiter=',')
            csvwriter.writerow([fc,count,fcount,f,countdict[f]]

And the results are

File - No of Fields(Without FID, Geometry) - Feature count - Field name - Field vaules (excluding null/blank)

D:\\Test\\shp\\sheet41.shp, 14, 87, Y, 87
D:\\Test\\shp\\sheet41.shp, 14, 87, X, 87
D:\\Test\\shp\\sheet41.shp, 14, 87, Sample No, 87
D:\\Test\\shp\\sheet41.shp, 14, 87, Cu_Method, 72
D:\\Test\\shp\\sheet42-6.shp, 7, 333, Sample No., 333
D:\\Test\\shp\\sheet42-6.shp, 7, 333, Comments, 10
D:\\Test\\shp\\sheet42-6.shp, 7, 333, Source, 0
D:\\Test\\shp\\sheet42-6.shp, 7, 333, Shape, 333
  • Do you want the theoretical max number of attribute values? Then it is int(count) * int(fcount). Or do you want the actual number of unique values? If this is the case you would have to summarize each attribute. – Dowlers Sep 8 '16 at 3:41
  • @Dowlers I want the actual number of values. – joseph_k Sep 8 '16 at 4:21
  • What is classified as not having a few attributes is it an empty string, null value, 0, etc.? – JamesLeversha Sep 8 '16 at 13:33
  • It varies. At times we have files with both "0"'s and nulls's. – joseph_k Sep 9 '16 at 1:17
  • @JamesLeversha your query made me to think in another way. I've edited and added the code. Have a look – joseph_k Sep 9 '16 at 2:35
2

As I said, not near a computer so hopefully this is ok, this is how I would go about it, I will check to see if it works in a few weeks. Essentially this stores all values for all keys in a list that is then stored in a dictionary under that key, you can then analyse each list such as counts etc.

import csv
import arcpy
from os import path
csvpath = r"D:\QAQC\Open.csv"
output = r'D:\Test\Test_32.csv'
csvdir = path.split(csvpath)[0]

with open(csvpath, "r") as csvfile:
    fcs = [r[0] for r in csv.reader(csvfile)]
with open(output, 'wb') as f:
    message = ['fc', 'fieldCount', 'rowCount', 'totUniCount', 'totCount']
    a = csv.writer(f)
    a.writerows([message])


for fc in fcs:
    # Get list of fields
    fields = [f.name for f in arcpy.ListFields(fc)]
    #get number of fields
    fieldCount = len(fields)
    #get field index in the list so you know what row to search for 
    fieldIndex = {k: v for v, k in enumerate(fields)}
    rowCount = 0
    # this is a dictionary to store all values under a key in each field
    # if the files are large or you haven't got much memory this
    # might fall over
    fieldInfo = {}
    with arcpy.da.SearchCursor(fc,fields) as cursor:
        for row in cursor:
            # count the rows
            rowCount += 1
            # get the value for each item in the row
            for key, value  in fieldIndex.iteritems():
                #add fields here that you don't want to count
                if key not in ['oid']:
                    rowValue = row[value]
                    # get the existing list of values for that field
                    # create empty list if it doesn't
                    listOfFieldData = fieldInfo.get(key,[])
                    if rowValue not in [None, " ", ""]:

                        # append the value to the list and store it under the 
                        # the appropriate field
                        listOfFieldData.append(rowValue)
                    fieldInfo[key] = listOfFieldData
    totCount =0
    totUniCount = 0
    # analyse your dictionary here
    # you can even get unique for each field as well
    for key, value in fieldInfo.iteritems():
       # print field name and length
        print key, len(value), len(set(value))
        totCount += len(value)
        totUniCount += len(set(value))
    #write it
    message =[fc, fieldCount, rowCount, totUniCount, totCount]
    with open(output, 'ab') as f:
         a = csv.writer(f)
         a.writerows([message])
  • Thanks a lot mate. With slight modifications your code worked perfectly. I was not in town and couldn't able to confirm the code. – joseph_k Sep 30 '16 at 6:23
2

A lot of this has been covered in Summarising all fields in attribute table using ArcPy?

I think the the Summary Statistics tool will do what you want but other approaches are mentioned in the first link. Summary Statistics will create a table with the unique counts of one or more attribute fields. You can then loop through the table and add up all the counts. Code below is not tested but will get you started. (I used the code in the answers in the question I linked to so credit to the original authors.)

import csv
import arcpy
from os import path
csvpath = r"D:\QAQC\List2.csv"
output = r'D:\Test\Test_2.csv'
csvdir = path.split(csvpath)[0]
intable = "intable"
outtable = "sumstats"
casefield = "Name"
stats = []
with open(csvpath, "r") as csvfile:
fcs = [r[0] for r in csv.reader(csvfile)]
with open(output, 'wb') as f:
    for fc in fcs:
        count = len(arcpy.ListFields(fc))
        fcount = arcpy.GetCount_management(fc)
        csvwriter = csv.writer(f,delimiter=',')
        csvwriter.writerow([fc,count,fcount])
        for field in arcpy.ListFields(fc):
            # Just find the fields that have a numeric type
            if field.type in ("Double", "Integer", "Single", "SmallInteger"):
            # Add the field name and Sum statistic type
            #    to the list of fields to summarize
            stats.append([field.name, "Sum"])
            # Correct formatting of stats [["Field1", "Sum"], ["Field2", "Sum"], ...]
         # Run the Summary Statistics tool with the stats list
         arcpy.Statistics_analysis(fc, outtable, stats, casefield)
         #Loop through the statistics and sum them
         for sumfield in arcpy.ListFields(outtable):
             mycount= sum([row[0] for row in arcpy.da.SearchCursor(fc, sumfield)])
         csvwriter.writerow([fc,count,mycount])
f.close()
  • Getting this error // Field Name does not exist within table // – joseph_k Sep 8 '16 at 6:09
  • @saravanaganesh18 Does the error specify which line it's failing at, or any other info? – Midavalo Sep 8 '16 at 6:16
  • Sorry. At line 22 which is // arcpy.Statistics_analysis(fc,outtable,stats,casefield) // – joseph_k Sep 8 '16 at 6:19
  • OK. Problem is with the casefield. Since it's only optional i've removed it. Again ran the code and getting this one (for the same line) // 'field_names' must be string or non empty sequence of strings // – joseph_k Sep 8 '16 at 6:48
1

The following code reads each shapefile, counts the features and fields, counts the unique values per field, and outputs them to a csv file. It assumes List2.csv has a column headed Shapefilename where it gets the list of shapefiles.

import arcpy, csv, os

csvpath = r"D:\QAQC\List2.csv"
output = r'D:\Test\Test_2.csv'

csvdir = os.path.split(csvpath)[0]

f = open(output, 'wb')
csvwriter = csv.writer(f, delimiter=',')
csvwriter.writerow(['Shapefile', 'Feature Count', 'Field Count', 'Possible Values', 'Unique Values'])

shpList = list()

with arcpy.da.SearchCursor(csvpath, 'Shapefilename') as cursor:
    for row in cursor:
        shpList.append(row[0])

for shpFile in shpList:
    shpPath = os.path.join(csvdir, shpFile)
    featCount = int(arcpy.GetCount_management(shpPath).getOutput(0))
    fieldList = arcpy.ListFields(shpPath)
    fieldCount = len(fieldList)

    valCount = 0

    if fieldCount > 0:
        for field in fieldList:
            uniqueValues = set()
            with arcpy.da.SearchCursor(shpPath, [field.name]) as cursor:
                for row in cursor:
                    if row[0] not in uniqueValues:
                        uniqueValues.add(row[0])
            valCount += len(uniqueValues)

        print "Shapefile = {}, Features = {}, Fields = {}, Possible Values = {}, Unique Values = {}".format(shpFile, featCount, fieldCount, featCount * fieldCount, valCount)

        csvwriter.writerow([shpFile, featCount, fieldCount, featCount * fieldCount, valCount])

f.close()
  • Yet to check this code. Let you know once i check with this one. Thanks – joseph_k Sep 8 '16 at 6:10
  • I've added "Shapefilename" in the csv. Still getting this error. // line 15, in <module> RuntimeError: A column was specified that does not exist // Line 15: // for row in cursor: // – joseph_k Sep 8 '16 at 6:55
  • @saravanaganesh18 in your List2.csv that lists your shapefiles to inspect, is there just a single column? And you've added Shapefilename as a header (first row) above the values in that column? – Midavalo Sep 8 '16 at 6:59
  • CSV file will have the complete file_path - Eg: D:\Test\Africa1.shp - row after row. I've added Shapefilename as header on the first row – joseph_k Sep 8 '16 at 7:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.