1

I have scripted a series 10 files that are run daily in order to generate an interpolation of data. However, I periodically have encountered errors where not all of the .csv data is being translated into the .shp file. I have already verified that the data within the .csv file is correct by verifying it against the others I introduce into ArcPy by plotting the data in ArcMap using "Display XY Data".

The script is designed to be dynamic in the sense that all shapefiles are created and saved into a unique YMD folder. The script used to generate .shp files is as follows:

import arcpy
import sys
from arcpy import env
from datetime import datetime, timedelta

date0 = datetime.strftime(datetime.now(), '%Y%m%d')

# Set environment settings
env.workspace = r"C:/somepath/data/"+date0+"/"
#Overwrite Old Files (0=No, 1=Yes)
arcpy.env.overwriteOutput = 1

try:
    # Set the local variables
    in_Table = "C:/somepath/data/"+date0+"/"+date0+"00_d0_tempfile.csv"
    x_coords = "lon"
    y_coords = "lat"
    z_coords = "maxt"
    out_Layer = "d0points_layer"
    saved_Layer = "d0points.lyr"

    # Set the spatial reference
    spRef = arcpy.SpatialReference("WGS 1984") 

    # Make the XY event layer
    arcpy.MakeXYEventLayer_management(in_Table, x_coords, y_coords, out_Layer, spRef, z_coords)

    # Save to a layer file
    arcpy.SaveToLayerFile_management(out_Layer, saved_Layer)

    #===============================================================================
    # Copy features to shapefile

    arcpy.CopyFeatures_management("C:/somepath/data/"+date0+"/d0points.lyr", "C:/somepath/data/"+date0+"/d0points.shp")
    #===============================================================================


except:
    print arcpy.GetMessages()

print 'd0 Data Conversion Complete'

sys.exit()

A key question as I sort this out is would it be beneficial to convert the .csv to a .gdb, then proceed with the conversion to .shp?

  • 1
    How are you measuring beneficial: efficiency, convenience, etc? My initial thought is that it would be an unnecessary, extra processing step. – Aaron Nov 21 '17 at 21:48
  • 2
    Where does the CSV come from? Just because you see the rows now does not mean they were there when the script ran, especially if it is a log file of some sort... File operations are usually buffered and until the file is closed by the application writing to it the data isn't guaranteed to be there it's in a buffer waiting to be there. – Michael Stimson Nov 21 '17 at 21:50
  • @MichaelStimson Thanks for getting back to me! The .csv file is copied from a separate machine using a wget process. I can confirm that all rows populate before the script runs, given that download is ordered before the .csv to .shp process runs! – E. Weglarz Nov 22 '17 at 1:36
  • @Aaron I'd prefer efficiency more than anything. I too would find it unnecessary, my initial thought was upon diagnosis that the step could be required, hence why I thought to ask here! – E. Weglarz Nov 22 '17 at 1:37
  • I don't see how wget confirms the file buffer is flushed before downloading, but anyway, if you say you're sure the file is fully committed I'll take your word for it. Instead of trying to bounce between formats you can open the file, read line by line and create point objects with an arcpy.da.InsertCursor - CSV is only text. Normally it's not worth the extra effort, as you're experiencing technical difficulties it might be worth the effort. – Michael Stimson Nov 22 '17 at 1:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.