2

I have taken this shapefile (warning > 1gB), and making my first steps towards decoding it in Python.

I had thought that for each field there would be a corresponding item in each record. Is that correct?

I have 21 fields, but only 20 items in each record(!). They almost, but not quite, align. What am I doing wrongly?

Fields

result = {list} <class 'list'>: [('DeletionFlag', 'C', 1, 0), ['ID', 'N', 10, 0], ['SOURCE_ORG', 'C', 25, 0], ['SOURCE', 'C', 30, 0], ['EL_SURFACE', 'C', 15, 0], ['NORTH', 'N', 6, 0], ['SOUTH', 'N', 6, 0], ['WEST', 'N', 7, 0], ['EAST', 'N', 7, 0], ['X_SRCE_RES', 'N', 19, 
 __len__ = {int} 21
 00 = {tuple} <class 'tuple'>: ('DeletionFlag', 'C', 1, 0)
 01 = {list} <class 'list'>: ['ID', 'N', 10, 0]
 02 = {list} <class 'list'>: ['SOURCE_ORG', 'C', 25, 0]
 03 = {list} <class 'list'>: ['SOURCE', 'C', 30, 0]
 04 = {list} <class 'list'>: ['EL_SURFACE', 'C', 15, 0]
 05 = {list} <class 'list'>: ['NORTH', 'N', 6, 0]
 06 = {list} <class 'list'>: ['SOUTH', 'N', 6, 0]
 07 = {list} <class 'list'>: ['WEST', 'N', 7, 0]
 08 = {list} <class 'list'>: ['EAST', 'N', 7, 0]
 09 = {list} <class 'list'>: ['X_SRCE_RES', 'N', 19, 4]
 10 = {list} <class 'list'>: ['Y_SRCE_RES', 'N', 19, 4]
 11 = {list} <class 'list'>: ['HORZ_UNIT', 'C', 15, 0]
 12 = {list} <class 'list'>: ['COORD_SYS', 'C', 25, 0]
 13 = {list} <class 'list'>: ['HORZ_DATUM', 'C', 25, 0]
 14 = {list} <class 'list'>: ['VERT_DATUM', 'C', 15, 0]
 15 = {list} <class 'list'>: ['VERT_UNIT', 'C', 15, 0]
 16 = {list} <class 'list'>: ['MIN_ELEV', 'N', 10, 0]
 17 = {list} <class 'list'>: ['MAX_ELEV', 'N', 10, 0]
 18 = {list} <class 'list'>: ['MEAN_ELEV', 'N', 19, 3]
 19 = {list} <class 'list'>: ['SDEV_ELEV', 'N', 19, 3]
 20 = {list} <class 'list'>: ['PROD_DATE', 'C', 10, 0]

First record

result = {tuple} <class 'tuple'>: (b' ', b'       268', b'Univ of Bristol          ', b'Ant Radar and Laser Alt DEM   ', b'Reflective     ', b'   -55', b'   -90', b'   -180', b'    180', b'          1000.0000', b'          1000.0000', b'Meter          ', b'Polar Sterograph
 __len__ = {int} 20
 00 = {bytes} b' '
 01 = {bytes} b'       268'
 02 = {bytes} b'Univ of Bristol          '
 03 = {bytes} b'Ant Radar and Laser Alt DEM   '
 04 = {bytes} b'Reflective     '
 05 = {bytes} b'   -55'
 06 = {bytes} b'   -90'
 07 = {bytes} b'   -180'
 08 = {bytes} b'    180'
 09 = {bytes} b'          1000.0000'
 10 = {bytes} b'          1000.0000'
 11 = {bytes} b'Meter          '
 12 = {bytes} b'Polar Sterographic       '
 13 = {bytes} b'WGS 84                   '
 14 = {bytes} b'WGS 84         '
 15 = {bytes} b'Meter          '
 16 = {bytes} b'       -82'
 17 = {bytes} b'      4211'
 18 = {bytes} b'           2152.694'
 19 = {bytes} b'           1127.631'

You probably don't need the code, which is simple enough anyway, but here it is:

data = shapefile.Reader(arguments.data_path + '\\' + fileName)
fields = data.fields
records = data.records()

Btw, is there any advantage to using data.shapeRecords()?

4
  • 2
    Fields in a .dbf shapefile table may or may not have a value (value not required). I'm guessing that is why they do not line up as you noted in your print statment. You may open the .dbf in excel or like program to view the data in table view to get an idea what records/fields have data (just make sure you do not edit anything while open b/c it could corrupt the shapefile)
    – artwork21
    Sep 22 '16 at 15:41
  • 3
    Welcome to gis.stackexchange! Please edit the title of your question to include enough information for future visitors to be able to find this thread when looking for the same problem.
    – underdark
    Sep 22 '16 at 15:54
  • @artwork21 I know that you are trying to help me, but I am just becoming confused. I would prefer to use only the Pysmp module to get the information. Does the info in the DBF file correspond to the shapefile records? Where can I fidn a good description of Pyshp? pypi.python.org/pypi/pyshp is helpful, but not helpful enough to complete my code Sep 22 '16 at 16:25
  • 2
    Just a side comment about data type, since you shapefile is very large I would recommend converting it to a database/table with the geospatial extension enabled (e.g. Esri File Geodatabase, Open Geodatabase, spatiallite). This way when you are doing querying and iterating on data you can take advantage of database indexing.
    – artwork21
    Sep 22 '16 at 17:59
5

The pyshp docs are not quite the worst I've seen, but they're in the wrong order. And have some errors, too.

pyshp is a pretty low-level module, and assumes you know quite a bit about the internals of a shapefile and dBASE files (yes, those old things). The reason you have 21 fields yet only 20 items is that DBFs have a hidden DeletionFlag field to mark whether a row has been deleted or not. This module leaves interpretation of this field up to the user, where higher level modules would silently skip those rows.

You're working with a large file here, so the recommendation to use the iterShapeRecords() method to iterate through large files is a good one. Here's a demo that trundles through a shapefile specified on the command line, printing some stuff about every row:

#!/usr/bin/env python
# pyshp test - show something about every row in a shapefile
#              specified on the command line - scruss - 2016-09

import shapefile
import sys
from pprint import pprint

sf = shapefile.Reader(sys.argv[1])

# get field names, skipping deleted flag
field_names = []
for f in sf.fields[1:]:
    field_names.append((f[0]))

count=0
for sr in sf.iterShapeRecords():
    geom = sr.shape             # get geo bit
    rec = sr.record             # get db fields
    fields_of = dict(zip(field_names, rec))
    print '### Record #', count, ':'
    pprint(fields_of)
    print ' Shape #', count, ' bounding box: ', geom.bbox
    print
    count=count+1

Running it on a file of local city ward data:

### Record # 0 :
{'CREATE_ID': 63519,
 'GEO_ID': 14630026,
 'LCODE_NAME': 'EA41',
 'NAME': 'Scarborough-Rouge River (41)',
 'OBJECTID': 1,
 'SCODE_NAME': '41',
 'SHAPE_AREA': '0.00000000000e+000',
 'SHAPE_LEN': '0.00000000000e+000',
 'TYPE_CODE': 'CITW',
 'TYPE_DESC': 'Ward'}
 Shape # 0  bounding box:  [320686.25, 4848286.583, 325808.25, 4854989.0]

### Record # 1 :
{'CREATE_ID': 63519,
 'GEO_ID': 14630028,
 'LCODE_NAME': 'EA44',
 'NAME': 'Scarborough East (44)',
 'OBJECTID': 2,
 'SCODE_NAME': '44',
 'SHAPE_AREA': '0.00000000000e+000',
 'SHAPE_LEN': '0.00000000000e+000',
 'TYPE_CODE': 'CITW',
 'TYPE_DESC': 'Ward'}
 Shape # 1  bounding box:  [329138.699, 4846015.756, 335747.204, 4852573.609]
....
7
  • 1
    Sounds great(!), BUT ... for shape, record in shapeRecs: TypeError: '_ShapeRecord' object is not iterable Sep 22 '16 at 17:28
  • By the way, I am not married to Pyshp - it is just the first module whic h I found. Perhaps I ought to be using one of those higher level modules of which you speak? Sep 22 '16 at 17:35
  • 1
    yeah, the docs were wrong. I've edited my answer to be actually something useful
    – scruss
    Sep 22 '16 at 22:14
  • Apologies for the delay in getting back to you; I was pulled to another project. Your code is excellent, but I seem to have a fundamental inability to understand a shapefile. Each shape consists of a record plus a geometry (bounding box). However, each record also has “north, south, west, east” items, which have a granularity of one degree. And the min/max/mean elevations never change over that one degree. Every record in that one degree has the same elevation values, although each has a different bbox. I need more detail, and had hoped/expected that those elevations would reflect the bbox. Sep 27 '16 at 14:25
  • 1
    it looks like you're looking at at a shapefile of metadata, not the data itself. Those fields look like the extent of the entire data set, so it's no surprise they don't change. You've picked a bad example to use to understand shapefiles. You should also have said that you're looking for elevation data, which are most often rasters and thus not shapefiles. Search for gmted instead
    – scruss
    Sep 28 '16 at 18:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.