Creating a really large shapefile without eating all the virtual memory

Question

I am doing something similar to below to create a really large (large than memory on my machine) shapefile:

shapes = shapefile.Writer(shapefile.POINT)
shapes.autoBalance = 1
shapes.field('MyId','N')
shapes.field('OtherData','C', 20)

for item in largeIteration():
    . . . . .
    shapes.record(item['id'], item['desc'])
    shapes.point(item['lat'], item['lon'])

The problem is while I'm using yield in largeIteration() to not hold onto memory there, the entire shapefile seems to remain open in memory, and therefore shapes grows with each call to shapes.record() and shapes.point(). Is there any way I can have pyshape not hold the entire shape in memory? I see OGR and its virtual file systems. However, I don't see a "stream to disk" vfs option.

Answer 1

To process large files, you need to use a generator which only read/write one line at a time and Python has a command that does that: with.

The with statement handles opening and closing the file, including if an exception is raised in the inner block. The for line in f treats the file object f as an iterable, which automatically uses buffered IO and memory management so you don't have to worry about large files. (How to read large file, line by line in python)

In short

with open("my_large_file.shp", "w") as f:
    for line in f:
       (code here)

Illustration with a Python module that allows that, Fiona, with data from where do I append point values with pyshp

data = [(33.21, -122.15, 'France'), (35.31, -122.15, 'Germany'), (35.41, -123.15, 'Hawaii'), (30.51, -122.15, 'Philippines'),(32.30, -122.15, 'Texas')]

import fiona
# schema of the shapefile
schema = {'geometry': 'Point', 'properties': {'location':'str:20'}
with fiona.open('my_shapefile.shp', 'w', driver='ESRI Shapefile', schema=schema) as output:
    for i in data:
        point = {'type': 'Point', 'coordinates': (l[0],l[1])}
        prop = {'location': l[2]}
        output.write({'geometry': point, 'properties':prop})

The equivalent solution with Pyshp leads to an error AttributeError: Writer instance has no attribute '__exit__ :

 with sf.Writer(sf.POINT) as w:
    w.field('location','C', 20))
    for l in data:
       w.point(l[0], l[1])
       w.record(l[2])
       w.save("my_shapefile.shp")

Unfortunately, you cannot use directly with here because PyShp,by design, "constructs' the entire shapefile in memory before saving it (but can use with or yieldto read a shapefile, look at PyShp as Fiona (with the GeoJSON-like protocol)).

  w = sf.Writer(sf.POINT)
  w.field('location','C', 20)
  for l in data:
      w.point(l[0], l[1])
      w.record(l[2])

and w grows, grows ...(it remain open in memory)
Then

   w.save("my_shapefile.shp")

You may be try to append each feature to an existing shapefile as in Appending using PyShp but it is more complicated.

Answer 2

this may be old - and i'm likely way off base, but pretty sure i did this a few days ago with OGR to create a shapefile larger than 2Gigs (by checking memory usage during writing, it never seemed to exceed about 40 megs).

so, it would be something like:

drv = ogr.GetDriverByName('ESRI Shapefile')
ds = drv.CreateDataSource('myshapefile.shp')
lyr = ds.CreateLayer('myshp', geom_type=ogr.wkbPoint25D)

fieldDef = ogr.FieldDefn('MyID', ogr.OFTInteger)
lyr.CreateField(fieldDef)
fieldDef = ogr.FieldDefn('OtherData', ogr.OFTString)
lyr.CreateField(fieldDef)

for item in largeIteration():
    feat = ogr.Feature(lyr.GetLayerDefn())
    feat.SetField('MyID',item['id'])
    pt = ogr.Geometry(ogr.wkbPoint25D)
    pt.AddPoint(item['lon'], item['lat'])
    feat.SetGeometry(pt)
    lyr.CreateFeature(feat)

    feat = None

i think the 'CreateFeature' writes to disk during each iteration, thus keeping memory usage down.