Using pyshp to convert .csv file to .shp?

Question

I am trying to understand how I can use the csv module in python to open a csv file in the same folder as the python script, and then create a shapefile using the shapefile module pyshp.

The csv file looks like this, but can have a couple of thousand rows of records:

id_nr;date;target;start_lat;start_lon
1;2012-05-21;navpoint 25x;55.123654;13.456954
1;2012-05-23;navpoint 11f;55.143654;12.456954

Answer 1

The pyshp module is a bit tricky to get the hang of, but really useful once you get it going. I've written a script that reads in a csv of the example data and writes out a shapefile with the data stored as attributes of the correct datatypes. The pyshp/xbase datatyping has always been tricky for me until I found this user guide for the xbase format and as a result of this question I have written a small note on my blog regarding the relevant pyshp datatypes, part of which I have pasted below:

• C is ASCII characters
• N is a double precision integer limited to around 18 characters in length
• D is for dates in the YYYYMMDD format, with no spaces or hyphens between the sections.
• F is for floating point numbers with the same length limits as N
• L is for logical data which is stored in the shapefile's attribute table as a short integer as a 1 (true) or a 0 (false). The values it can receive are 1, 0, y, n, Y, N, T, F or the python builtins True and False

The full listing is as follows:

import shapefile as shp
import csv

out_file = 'GPS_Pts.shp'

#Set up blank lists for data
x,y,id_no,date,target=[],[],[],[],[]

#read data from csv file and store in lists
with open('input.csv', 'rb') as csvfile:
    r = csv.reader(csvfile, delimiter=';')
    for i,row in enumerate(r):
        if i > 0: #skip header
            x.append(float(row[3]))
            y.append(float(row[4]))
            id_no.append(row[0])
            date.append(''.join(row[1].split('-')))#formats the date correctly
            target.append(row[2])

#Set up shapefile writer and create empty fields
w = shp.Writer(shp.POINT)
w.autoBalance = 1 #ensures gemoetry and attributes match
w.field('X','F',10,8)
w.field('Y','F',10,8)
w.field('Date','D')
w.field('Target','C',50)
w.field('ID','N')

#loop through the data and write the shapefile
for j,k in enumerate(x):
    w.point(k,y[j]) #write the geometry
    w.record(k,y[j],date[j], target[j], id_no[j]) #write the attributes

#Save shapefile
w.save(out_file)

I hope this helps.

Answer 2

As an alternative you do not need to hold the data in lists.

# import libraries
import shapefile, csv

# create a point shapefile
output_shp = shapefile.Writer(shapefile.POINT)
# for every record there must be a corresponding geometry.
output_shp.autoBalance = 1
# create the field names and data type for each.
# you can insert or omit lat-long here
output_shp('Date','D')
output_shp('Target','C',50)
output_shp('ID','N')
# count the features
counter = 1
# access the CSV file
with open('input.csv', 'rb') as csvfile:
    reader = csv.reader(csvfile, delimiter=',')
    # skip the header
    next(reader, None)
    #loop through each of the rows and assign the attributes to variables
    for row in reader:
        id= row[0]
        target= row[1]
        date = row[2]
        # create the point geometry
        output_shp.point(float(longitude),float(latitude))
        # add attribute data
        output_shp.record(id, target, date)
        print "Feature " + str(counter) + " added to Shapefile."
        counter = counter + 1
# save the Shapefile
output_shp.save("output.shp")

You can find a working example of this implementation here.

Answer 3

I did not have success with any of the solutions here but I was able come up with a solution that worked using Python's shapely and fiona modules. It uses a tab-delineated .ascii file (my preference as opposed to .csv) but can easily be adapted to use a .csv as in the question posed. Hopefully this is helpful someone else trying to automate this same task.

# ------------------------------------------------------
# IMPORTS
# ------------------------------------------------------

import os
import pandas as pd
from shapely.geometry import Point, mapping
from fiona import collection

# ------------------------------------------------------
# INPUTS
# ------------------------------------------------------

# Define path
path = os.path.abspath(os.path.dirname(__file__))

# Set working directory
os.chdir(path)  

# Define file to convert
file = 'points.ascii'

# Define shp file schema
schema = { 'geometry': 'Point', 'properties': { 'LocationID': 'str', 'Latitude': 'float', 'Longitude': 'float' } }

# Read in data
data = pd.read_csv(file, sep='\t') 

# Define shp file to write to
shpOut = 'points.shp'

# Create shp file
with collection(shpOut, "w", "ESRI Shapefile", schema) as output:
    # Loop through dataframe and populate shp file
    for index, row in data.iterrows():
        
        # Define point
        point = Point(row['Longitude'], row['Latitude'])
        # Write output
        output.write({
            'properties': {'LocationID': row['LocationID'], 'Latitude': row['Latitude'], 'Longitude': row['Longitude'] }, 
            'geometry': mapping(point)
        })