Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I'm interested in a way to identify rasters (e.g., ESRI GRID format) or shapefiles that are corrupt that would prevent them from being used in an analysis tool. I have a large number in structured folder tree and would like to be able to script something that will walk the file tree and be able to ID rasters that, for example if you tried to view it in ArcCatalog it would not allow it to be displayed and accordingly would make any geoprocessing tool fail. Same deal for shapefiles or feature classes.

My preference would be somehow to do this in Python (and I wouldn't be opposed to a non-ESRI python module or other) but I'm open for anything. I guess you could hack something together by trying to validate geometry or capitalizing on some aspect of the file design, but what would make the most sense? Or is there another way to do it that is specifically designed for such a purpose?

My goal is to just be able to run a simple Quality Control mechanism on some large week-long preprocessing files to make sure they're all sound before I start doing the actual batch processing.

thanks, Tom

share|improve this question

4 Answers 4

up vote 1 down vote accepted

For the rasters, why not just use a simple script to output the properties (cell size, extent etc.) of all the rasters you are going to use. If getting the properties fails then anything more advanced will also fail.

Plus you can write the raster properties to a log file and check that the extents, cell sizes, value types etc. are all what you expect.

import arcgisscripting, sys, string, os
gp = arcgisscripting.create()

try:
    # Set local variables
    InRaster = "D:/Data/elevation"
    InPropertyType = "MAXIMUM"

    # Process: GetRasterProperties
    zmax = gp.GetRasterProperties (InRaster, InPropertyType)
    # log these to a file or apply logic here to 
    # make sure values are in the expected ranges


except:
    # Print error message if an error occurs
    # likely to be an invalid raster
    print gp.GetMessages()

Also your mention of week long processing is somewhat worrying. It is very likely you will have to rerun the processes, so is there any way you can break it down into several smaller intermediate steps. This way you can check the results at each stage, and if something is incorrect you only need to resume from the previous step rather than the beginning.

share|improve this answer
    
thanks for the suggestion on getting raster properties, I figured utilizing some characteristic(s) of the data type would probably be my best bet given there are no validation tools that I was aware of. Regarding week-long processing, no problem there the time length is just due to the number of rasters and their size. I have a robust python program handling it, but part of the problem is the data are on our network so I'm losing some speed since they're not local.I actually built in functionality to remove from the processing list those that have already been run automatically –  turkishgold May 10 '11 at 12:12
    
Furthermore - I'll probably just do something similar for shapefiles in terms of just checking the geometry (@Craig Williams). The intent of my post was to explore if there something out there like a non-ESRI python module or other...but outlook not so good based on these responses. –  turkishgold May 10 '11 at 12:16
    
@turkishgold - You could use the same technique of reading shapefiles and rasters using GDAL and the Python bindings gdal.org/gdal_tutorial.html –  geographika May 10 '11 at 15:26

As far as vector shp files I would use something like the shape file checker on arcscripts.
shape file checker

For Raster The most common error during geoprocessing is invalid field type.
You might consider discovering the field types in your geoprocessing modules and then run the check on field types in your rasters.
I guess other types of erros or some other things to consider might be the pallette being indexed, or color depth being unsupported for particular processing.

Actually being corupt seems a bit more rare. But sure it can happen. Here might be some things to look for.
esri forums
corrupt dted
I think that most of the time these cases could be that using another software to open the file and save it back out to correct format/paramters fixes a lot of these issues.

share|improve this answer

For the vector data sources, use Check Geometry/Repair Geometry in tandem.

share|improve this answer

There are no tools to do validation in ArcToolbox. Have your identified anything within ArcObjects that would do this? I suspect that a Python solution will not work since it is tied into ArcToolbox tools, so short of loading each file into ArcMap, I suspect you are out of luck.

share|improve this answer
    
I came across this yesterday: code.google.com/p/pyshp and was hoping maybe someone had used something like that to perform a similar task (the reason I mentioned being open to non-ESRI options). But I realize the latter wouldn't do anything for me with rasters... –  turkishgold May 10 '11 at 2:19

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.