Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I'm trying to import a great number of csv files of about 250MB each. I wrote a python script for that as follows:

import gc
import glob
tables = glob.glob("C:\\Users\\emportella\\Documents\\ArcGIS\\2517")
for table in tables:
    arcpy.TableToGeodatabase_conversion(table, "C:\\Users\\emportella\\Documents\\ArcGIS\\2517\\Tables.gdb")
    gc.collect()

It works for the first 4 to 6 files and without giving any warning is closes. i've payed attention to memory usage in Windows Task Manager and the process cosumes more and more memory until it craches.

I've already tried using some different aproches like using:

env.workspace = "C:\\Users\\emportella\\Documents\\ArcGIS\\2517\\*.csv"
tables = arcpy.ListTables()
arcpy.TableToGeodatabase_conversion(tables, "C:\\Users\\emportella\\Documents\\ArcGIS\\2517\\Tables.gdb")

The TableToGeodatabase_conversion accepts list of tables but it craches the same way.

So there is the question is there a way to manage the memory used in arcpy framework?

share|improve this question
1  
are you cleaning your vars as you iterate? ie "del tables" Is the code you posted the entire script? –  Tom Apr 24 '13 at 20:48
add comment

2 Answers 2

up vote 4 down vote accepted

Another approach might be to use the subprocess module to farm out each iteration to a separate process, each with its own memory space that is freed up as soon as the process exits. See this answer for an explanation: Work-around for slow processing times in ArcGIS 9.3 with Spatial Analyst functions

share|improve this answer
add comment

I can't speak to the memory consumption / crashes without actually profiling it. I can offer you an alternative workflow which works quite well for me in arcpy 10.1 (10.1 as it uses da.InsertCursors). The following code is meant to be used as a script tool, but it looks like you know enough about Python to modify it to loop through files and output unique ones.

import arcpy, os, csv

infile = arcpy.GetParameterAsText(0)
outtable = arcpy.GetParameterAsText(1)

outpath = os.path.dirname(outtable)
outname = os.path.basename(outtable)

temptable = "in_memory\\tempcsv"
temppath = os.path.dirname(temptable)
tempname = os.path.basename(temptable)

if arcpy.Exists(temptable):
    arcpy.Delete_management(temptable)

arcpy.CreateTable_management(temppath, tempname)

rfile  = open(infile, 'rb')
csvreader = csv.reader(rfile)

index = 0
fieldnames = []
oidfieldnames = ["OID","FID","OBJECTID"]

for csvrow in csvreader:
        if index == 0:
                for column in csvrow:
                        if column in oidfieldnames:
                column = column + "_ORIG"
            arcpy.AddField_management(temptable, column, "TEXT")
            fieldnames.append(column)
        arcpy.CreateTable_management(outpath, outname, temptable)
        arcpy.Delete_management(temptable)
                tablerows = arcpy.da.InsertCursor(outtable, fieldnames)
        else:
                tablerows.insertRow(csvrow)
        index = index + 1

rfile.close()
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.