4

I noticed something weird when messing with the geospatial python's shapefile module (no complaints here, this module is awesome and the only pure python way to write shapefiles).

When adding date fields, the values appear to be written in the table correctly, but when I view them in any ArcGIS application, the dates appear wrong. Using this simple code, I am adding a new date:

import shapefile
from datetime import datetime

test = r'C:\TEMP\Date_test3.shp'

w = shapefile.Writer(1)
w.point(-93.1, 44.1)
w.field('Date', 'D')

# todays date, but appears in table as 12/8/2014....Weird?
today = datetime(2015, 8, 7) #hard coded so the example is always the same
w.records = [[today]]
w.save(test)

When I view it in ArcGIS, it looks wrong:

enter image description here

I have also tried to input this as a string with no luck. However, the value that is stored is correct. When I read the shapefile:

# however, when I read it, it is indeed the correct value in the database
r = shapefile.Reader(test)

print r.records()[0][0] 
>>> 
2015-08-07 00:00:00
>>> 

Is there a different way I should be passing in dates for the shapefile module? Is this a bug specific to ArcGIS, or is something funky happening with the way shapefile is encoding the data. I also opened up the .dbf in Excel, and it appears correct there too (I suppose this might indicate a bug with ArcGIS). Does anyone know what may be going on here?

I am using ArcGIS 10.3.

  • 1
    Please edit your question to specify the version of ArcGIS in use. – Vince Aug 7 '15 at 22:27
  • 2
    I can't reproduce this issue when I use tools I have written to generate dBase files. An od dump of the .dbf might indicate if it has been encoded correctly. There may be something happening with charset encoding as well. This is probably something to take to Esri Tech Support. – Vince Aug 8 '15 at 1:45
  • It does seem strange. I will test on another machine and if I can reproduce I will contact Esri support. – crmackey Aug 8 '15 at 3:02
  • 1
    Can you generate od -t x1 output? After the two 32-byte headers, I have 0D 20 32 30 31 35 30 38 30 37 1A -- OD is "no more fields", 20 is space (not deleted) and 32 30 31 35 30 38 30 37 is "20150807", and 1a is an EOF marker; My 10.3 reads it fine. If not, GeoNet will let you post a zip file, so you could upload a zip of your Date_test3.shp set there (use the GIS>Managing Data place) – Vince Aug 8 '15 at 4:10
  • @Vince I was able to reproduce this on a colleague's machine (also ArcGIS 10.3) by running the exact same code. I am not familiar with the od command you are using. What Python module are you using to write .dbf files? The shapefile module uses pure Python to write out the DBF headers (using the built in struct module) so I am wondering if there is something wrong in the source code that isn't properly handling date fields. – crmackey Aug 10 '15 at 12:39
3

I took a look at the source code for the shapefile and found this (line 507):

    elif typ == b('D'):
        if value.count(b('0')) == len(value):  # QGIS NULL is all '0' chars
            value = None
        else:
            try:
                y, m, d = int(value[:4]), int(value[4:6]), int(value[6:8])
                value = [y, m, d]
            except:
                value = value.strip()

The shapefile module is just doing a simple slice on a date string, so I am able to get a correct date by entering in this format as a string: '20150810'.

Apparently this is the only way to pass in dates. I do not think this is very useful so I may add some code and make a pull request to allow for datetime objects and a few other string options.

UPDATE:

I just made a fork of this repo on GitHub with the following proposed changes (included here in case someone wants to add it to the code until the pull request is approved/or never approved):

#proposed lines 20-21
import datetime
import re

#proposed line 41
DATE_EXP = re.compile('\d{4}[-/]\d{2}[-/]\d{2}')

if PYTHON3:
    xrange = range
    izip = zip
    basestring = str #proposed line 46

# proposed lines 917-928
elif fieldType == 'D':
    if isinstance(value, datetime.datetime):
        ymd = [str(v).zfill(2) for v in [value.year, value.month, value.day]]
        value = '{0}{1}{2}'.format(*ymd)[:size].ljust(size)
    elif isinstance(value, basestring):
        if DATE_EXP.match(value):
            try:
                value = DATE_EXP.findall(value)[0].replace('/','').replace('-','')[:size].ljust(size)
            except IndexError:
                value = str(value)[:size].ljust(size)
    else:
        value = str(value)[:size].ljust(size)

With these date input improvements, dates can now be added as datetime.datetime() objects and strings matching these formats: '2015/08/10', '2015-08-10'.

So now, I can run this new test and all is good:

import shapefile
from datetime import datetime

test = r'C:\TEMP\Date_test.shp'

w = shapefile.Writer(1)
w.point(-93.1, 44.1)
w.field('Date_obj', 'D') #datetime.datetime() object
w.field('Date_dsh', 'D') #date with a dash
w.field('Date_slsh', 'D') #date with a slash


# now I can input 3 differnt formats
today = datetime.today() 
w.records = [[today, str(today), '2015/08/07']]
w.save(test)

# now all is good!
r = shapefile.Reader(test)

print r.records()[0] 

This printed:

[[2015, 8, 10], [2015, 8, 10], [2015, 8, 7]]

And finally, when I open the table in ArcGIS Desktop:

enter image description here

FINAL UPDATE:

Just re-examined my code, this is better:

ymd = ''.join([str(v).zfill(2) for v in [value.year, value.month, value.day]])
value = ymd[:size].ljust(size)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.