Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

Q_PART_I

I need to merge many (MANY) tables in one unique output table by means of the arcpy.Merge_management. Anyway, I'd like to avoid the creation of tons of intermediate files as my script running, having only my final big table rather than a number of "mini-table" for each file I have. I'm converting raster attribute table to table views (by arcpy.TableToTable_conversion), and then using them to run the Merge. How can I avoid having a table for each raster in the intermediate process? Hope someone could help me. I've been reading something about the creation of an InMemoryWorkspace, but I couln'd manage it so far. Thanks in advance.

Q_PART_II

Ok, I tried both Your suggestions, but so far I couldn't make it. It seems the Merge function cannot recognize the temporary files as input. Infact, I receive this error: ERROR 000735: Input Datasets: Value is required Failed to execute (Merge) Here is my script (the important part is at the end, I can always avoiding this problem by simply having a lot of files, but I'd prefer to be more "clean"):

#import relevant modules, create geoprocessing dispatch object
import win32com.client, sys, string, os, arcpy

gp = win32com.client.Dispatch("esriGeoprocessing.gpDispatch.1")

# Remember to change this to wherever your files are stored
gp.workspace = 'G:\\Chile_2012\\MODIS10A2.V005\\provvisoria\\ATTRIBUTE_TABLE'
# Remember to change the path to point where you want to store the final tables
out_dir = 'G:\\Chile_2012\\MODIS10A2.V005\\provvisoria\\ATTRIBUTE_TABLE'

try:
    rasters = gp.ListRasters("*", "all")
    rasters.reset()
    rst = rasters.Next()

    while rst:
        # Create the new field
        gp.AddField_management (rst, "FILENAME", "text", "", "", "50")

        # Apply the filename to all entries       
        gp.CalculateField_management (rst, "FILENAME", '"' + rst + '"')
        rst = rasters.Next()

except:
    print gp.GetMessages ()

listRaster = arcpy.ListRasters ()

# Create a table view from each raster to be used as input for the next Merge function:
for raster in listRaster:
    arcpy.TableToTable_conversion (raster, "in_memory", 'Tab_'+ raster)

listTable = arcpy.ListTables ()

# Change the name of the output table as needed!!!
arcpy.Merge_management (listTable, 'Cl_Rio_Aconcagua.dbf')

Hope this could help You helping me in turn. Many thanks in advance!

share|improve this question
add comment

2 Answers 2

up vote 4 down vote accepted

Sadly, the in_memory workspace in ArcGIS is one of the least-known, least used, and least advertised, yet most powerful functions in arcpy scripting. It really needs to be more highly touted than it is...

What you really should consider doing is copying the data into an memory layer using the in_memory workspace. Arc will treat it just like a normal file-based table, but it will automatically clean itself up after you exist your python script.

So your script would be something like:

arcpy.TableToTable_conversion(inTable, "in_memory", outTable, expression)
share|improve this answer
    
My guess is this will take longer and use more memory than the Table View approach. The in_memory workspace is good for a number of things but using Layers and Table Views properly can be just as if not more powerful. –  blah238 Aug 31 '12 at 20:44
    
Thank You. I solved clearing the environment and setting the new one to the "in_memory" path. Then at the very end of the script I removed all the temporary files by using the "arcpy.Delete_management" function. Thank You again, now I can use this way to produce intermediate files without problems! –  umbe1987 Sep 3 '12 at 12:58
add comment

A few points:

  • Table To Table creates a Table, not a Table View. This creates new data, in this case probably unnecessarily.

  • Make Table View creates a Table View.

  • A Table View is a reference to the original data with a few additional properties such as visible fields and a definition query, as well as selection support. But, much like a Layer, it does not actually create new data, hence why it's called a View.

  • If you want to avoid intermediate data, use Make Table View.

Is there some reason you are using Table to Table specifically?

Update:

Here is a code example to hopefully steer you in the right direction. I have not tested it, however.

This picks up after your listRaster = arcpy.ListRasters() line.

tableviews = []
for raster in listRaster:
    tableviews.append(arcpy.MakeTableView_management(raster, 'TabView_' +
        os.path.splitext(os.path.basename(raster))[0]).getOutput(0))
arcpy.Merge_management(tableviews, 'Cl_Rio_Aconcagua.dbf')
share|improve this answer
    
Thank you, I didn't know about the MakeTableView tool. Anyway, I use the TableToTable because I don't need to merge the rasters, but only their Attribute Table. They have only one row each, and rasters in my case represents a cloud mask derived from a MODIS image over a specific area for different days of the year each. –  umbe1987 Aug 31 '12 at 20:09
    
I think Make Table View will work just fine in this case then. It should work with Merge. –  blah238 Aug 31 '12 at 20:17
    
Thank You blah238. One question: how can I access the temporary files I create with the MakeTableView as the input to the Merge tool later? That was my clue made me ask my first question. I imagine there should be like a "temp" statement to make them temporary, still knowing where the workspace is and use this path to make a ListTables to use all the created temporary tables as my new new inputs for the Merge. –  umbe1987 Aug 31 '12 at 20:27
    
As I said, there are no intermediate files created by MakeTableView. The only thing it creates is a Table View within the current process that you access by name. Iterate over the input raster tables and uniquely name each Table View, such as by appending a counter variable, and put them into a list, then pass the list into Merge_management. This should work, but I haven't tried it myself. –  blah238 Aug 31 '12 at 20:36
    
Thank You again blah238, I've just red Your example, I didn't see it before. I didn't try it yet because I managed with the 'in_memory' way, which I can use in many other occasions. Anyway, I'll try Your method which seems really useful as well to me, and which I can understand. Thank You so much for Your efforts! –  umbe1987 Sep 3 '12 at 17:44
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.