1

I am trying to convert ~270k csv-files with lat/long-data to layerfiles. The files are split into folders by year, i.e. ~30 folders with ~9k csv-files in each.

I have written a script to iterate through the files and it works well for test-samples of 100+ split into a couple of different folders. However, when I try to run it on the full set of files, it will either:

  1. work well for the first folder (~9k files) and then when moving on to the second folder it will give the following error for all subsequent files:
    arcgisscripting.ExecuteError: ERROR 000223: Cannot extract event table properties
    Failed to execute (MakeXYEventLayer).

  2. on my other computer it will usually stop after an arbitrary amount of file-conversions before it finishes the first folder (typically at +-8.5k of the conversions) and then give the same error for all subsequent files

The error seems to be similar to this: http://forums.arcgis.com/threads/80161-Make-XY-Event-Layer-Error ,but my error pops up after an arbitrary amount of files has been processed.

I've included my code for reference below. Note that I have also tried to split the list of files per folder into chunks of 100 and defined the layercreation as a function that is being called for each chunk, but the error still persisted in the same way. I suspect that it might be some setting in ArcGIS or on my computers that causes it, but I don't know what. Does anybody have any idea where I might start to solve this?

#setting paths for the project 
path="C:/project"
savepath = path + "/outputlayers"
csvpath = path+ "/csv"

#lists the directories in the csv directory
yearlist = os.listdir(csvpath)

#checks the directory of each year:
for year in yearlist:
    indir = csvpath + "/" + year
    hourlist = os.listdir(indir)

        #runs through each file and converts it to a .lyr file in the outputdirectory
        for infile in hourlist:
            try:
                # Set the local variables
                in_Table = indir + "/" + infile
                x_coords = "longitudex"
                y_coords = "latitudey"
                z_coords = ""
                out_Layer = infile[:-4]
                saved_Layer = savepath + "/" + year + "/" + infile[:-4] +'.lyr'

                # Set the spatial reference
                spRef = arcpy.SpatialReference(4301)

                # Make the XY event layer...
                arcpy.MakeXYEventLayer_management(in_Table, x_coords, y_coords, out_Layer, spRef, z_coords)

                # Print the total rows
                print arcpy.GetCount_management(out_Layer)

                # Save to a layer file
                arcpy.SaveToLayerFile_management(out_Layer, saved_Layer)

            except:
                # If an error occurred print the message to the screen
                print arcpy.GetMessages()

2 Answers 2

3

First off, I don't think you want to be saving these as layer files (.lyr). A layer file is only a pointer to data. You need to save the data to a feature class or shapefile. The output from the Make XY event layer tool is a "in memory layer" and it is gone once the session is over. That needs to be converted to feature class to save it to your computer.

Try something like this:

arcpy.CopyFeatures_management(out_Layer, "C:/project/year_hour.shp")

The error your getting is probably related to either running out of memory, or because your trying to create an "in-memory" layer that already exists. try deleting the in memory layer after your are done with it:

arcpy.Delete_management(out_Layer)
0
0

Just to provide some further color on the solution to this. The first answer worked well to a certain point, but I still had problems with memory leak from ArcGIS.

The final working solution was to split my code into two scripts, where the first code split the files into chunks of 1000 and then fed this into ArcGIS via a subprocess.call-process. This way ArcGIS was 'closed' between each instance and the used memory was freed up.

Code in 'main' script:

import sys, string, os, arcgisscripting, subprocess, argparse


# Setting paths for the directories 
path="C:/project/"
savepath = path + "/outputlayers"
csvpath = path+ "/csv"

#lists the directories in the csv directory
yearlist = os.listdir(csvpath)

#checks the directory of each year and defines the workspace of arcgis
for year in yearlist:
    indir = csvpath + "/" + year
    hourlist = os.listdir(indir)
    py_path = "C:/Python27/ArcGIS10.1/python.exe"
    py_script = "C:/script/layer.py"

    for i in xrange(0, len(hourlist), 1000):
        chunklist = hourlist[i:i+1000]
        subprocess.call([py_path, py_script, chunklist, year])

Then I had a 'child' script called layer.py. It isn't pretty, since chunklist is read as a string, but it does the trick. I believe argparser might do a better job.

# import system modules 
import arcpy
import os
import sys

# Setting paths for the directories 
path="C:/project/"
savepath = path + "/outputlayers"
csvpath = path+ "/csv"
year = "/" + str(sys.argv[2])
indir=csvpath + year
hourlist = sys.argv[1]
hourlist=hourlist.split('.csv')
for i,v in enumerate(hourlist):
    hourlist[i] = str(v) + '.csv'

del hourlist[len(hourlist)-1]

#runs through each file and converts it to a .lyr file in the outputdirectory
for infile in hourlist:

the following part is very similar to the code in my original question

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.