Making GIS inventory using ArcPy

Question

My office will be seeing a big change in its GIS section. This section has been operational since the 1980's and has a huge collection of GIS data (i.e., shapefiles, raster files, data, etc) but never been through any inventory. Now it will happen.

Is there any automated way to extract all the information about the GIS data (i.e., shapefile, arc-info coverage, layer file, *.mxd, gdb, raster file, and more) from a PC to an Excel file? The information may include date of creation, date last edited, folder or container name, etc.

Answer 1

This works for me, using the arcpy.da.Walk function at ArcGIS 10.1 SP1:

import arcpy, csv, os

workspace = r"c:\GISData"
output = r"C:\temp\test.csv"

with open(output, 'wb') as csvfile:
    csvwriter = csv.writer(csvfile)
    for dirpath, dirnames, filenames in arcpy.da.Walk(workspace):
        for filename in filenames:
            desc = arcpy.Describe(os.path.join(dirpath, filename))
            csvwriter.writerow([desc.catalogPath, desc.name, desc.dataType])

The csv module is also used to simplify writing the output file. Excel can open CSV files so you can view them as spreadsheets.

See the arcpy.Describe function for additional properties you can include in the output.

If you are specifically looking to parse out information from the actual metadata, see the script in this answer: Creating a table containing all filenames (and possibly metadata) in a File Geodatabase

Answer 2

When you use Python, you must use the correct modules to do what you want. To find all files in a directory with extension shp, for example, there are much simpler solutions that was presented without the break, which is awful...(as the solution presented by Nathan W, but there are many, many others, just search on Internet)

Some examples with relevant modules:

1) with the glob module:

shapefiles only:

import glob
import os
os.chdir("mydir")
for files in glob.glob("*.shp"):
    print files

shapefiles and geodatabases:

import glob
types = ('*.shp', '*.gbd') # the tuple of file types
files_grabbed = []
for files in types:
     files_grabbed.extend(glob.glob(files)) #files_grabbed = the list of shp and gbd files

if you want to search also in the subdirectories:

import glob
for f in glob.iglob("/mydir/*/*.shp"): #search immediate subdirectories 
    print f

2) with os.listdir and list comprehension (in two lines) -> list of results

path = 'mydir'
shape_files = [f for f in os.listdir(path) if f.endswith('.shp')]
gdb_files = [f for f in os.listdir(path) if f.endswith('.gdb')]

3) with fnmatch module:

import fnmatch
for file in os.listdir('path'):
    if fnmatch.fnmatch(file, '*.shp'):
        print file

and many others solutions, recursive etc.

Answer 3

Thanks artwork21 and Nathan W for your response. And yes Nathen's code made the magic.

import os, arcpy

#create blank text file
with open("C:\\Temp\\GISlayers.txt", "w") as txt:
for root, dirs, files in os.walk("C:\\Temp\\temp"):
    for f in files:
        #look for shapefiles
        if f.endswith('.shp'):
            desc = arcpy.Describe(root + "\\" + f)
            #write info to text file
            txt.write(desc.name + "," + desc.catalogPath + "\n")

        #look for file geodatabases
        if f.endswith('.gdb'):
            desc = arcpy.Describe(root)
            for child in desc.children:
                #write info to text file
                txt.write(child.name + "," + child.path + "\n")

        #look for layer files
        if f.endswith('.lyr'):
            desc = arcpy.Describe(root + "\\" + f)
            #write info to text file
            txt.write(desc.name + "," + desc.catalogPath + "\n")

        #look for img file
        if f.endswith('.img'):
            desc = arcpy.Describe(root + "\\" + f)
            #write info to text file
            txt.write(desc.name + "," + desc.catalogPath + "\n")

Only file name, and location. The pc i'll be working with has lots of coverage (the arc-info file) file, will it work on them too?

Answer 4

If you have ArcGIS Desktop 10.0 (or any of its service packs), I think your best bet is writing a python script that uses os.walk to look through a defined GIS directory and searches for common GIS file extensions such as .shp, .gdb, .mdb, etc... and writes the result to a comma delimited text file. You can then bring the text file into excel, see code example below:

import os, arcpy

#create blank text file
txt = open("C:\\Temp\\GISlayers.txt", "w")

for root, dirs, files in os.walk("C:\\Temp\\temp"):
    for f in files:

        #look for shapefiles
        foundSHP = f.find(".shp")
        if foundSHP >0:
            checkEXT = f[-3:]
                if checkEXT <> "xml":
                    desc = arcpy.Describe(root + "\\" + f)
                    #write info to text file
                    txt.write(desc.name + "," + desc.catalogPath + "\n")

        #look for file geodatabases
        foundGDB = f.find(".gdb")
        if foundGDB >0:
            desc = arcpy.Describe(root)
            for child in desc.children:
            #write info to text file
            txt.write(child.name + "," + child.path + "\n")
        break
txt.close()

If you are using ArcGIS 10.1 (or later) for Desktop then there is another Answer here that uses arcpy.da.Walk which was not available at 10.0 or earlier.

Answer 5

If you want to avoid programming, this might be the easiest and fastest method.

There is an add-on for Excel called ASAP Utilities. There is a 90-day free trial but after that, it's $49 USD for business use. It's free for student or personal use. The add-on adds a lot of useful functions. One of which is creating a list of files in a folder structure. It also provides file properties. You can limit the results by file type if you wish.

Here's a video of how to do this.

I've used this add-on before and the results are very quick.

Note, I am not affiliated with this software company.

Answer 6

I couldn't get the other answers to work fully.

In the first example, in a directory with both geodatabases and shapefiles, I only got a listing of the feature classes in the geodatabase, but when I commented out the geodatabases portion of the script, I got a list of shapefiles.

In the second example, the geodatabases portion didn't work at all, so I copied in the geodatabases portion of the first example. Again, I got a list of only geodatabases.

Then it hit me: geodatabases are read before shapefiles, and the script stops at the break in the geodatabases part.

Being a python newbie I don't know why the break is needed, but without it the script seems to go in an endless loop, but since the break is needed it occurred to me that putting the geodatabases part in its own loop, after the other file types are listed, would solve the problem:

#create blank text file with open("C:\\Temp\\GISlayers.txt", "w") as txt: for root, dirs, files in os.walk("C:\\Temp\\temp"): for f in files: #look for shapefiles, etc. [code...] for f in files: #look for geodatabases [code...]

When I did that I got my full listing.