1

I am working on scripting some data processing using QGIS and Python, but I'm running across issues that I need to debug. Since my data is really large, and run-times can be lengthy, I would like to be able to write out results periodically and then append to those results so I don't have to re-run my scripts.

I'm outputting a polygon shapefile, and I am using the following Python code currently to work with the shp file:

poly_provider = poly_layer.dataProvider()    
fields = { 0 : QgsField("STAT_NO", QVariant.String) }

writer = QgsVectorFileWriter("/localstore/data/shps/contrib_area.shp", "CP1250", fields, poly_provider.geometryType(), poly_layer.crs(), "ESRI Shapefile")

if writer.hasError() != QgsVectorFileWriter.NoError:
  print "Error when creating shapefile: ", writer.hasError()

#processing...

def collect_geoms(geometries):
  if len(geometries) > 0:
    union_geoms = geometries[0]
    for geometry in geometries:
      union_geoms = union_geoms.combine(geometry)

    fet = QgsFeature()
    fet.setGeometry(union_geoms)
    fet.addAttribute(0, QVariant( point_feat_dict[point] ))
    writer.addFeature(fet)

The collect_geoms() function is then called in a loop, and when I'm done, I use del writer and I get the shapefile written to the disk.

Is there a way that I can write the shapefile out incrementally and append new polygons to it? So for instance I can run 10 iterations of the loop, write the shapefile, and then run the next ten without overwriting the first set?

Improve this question

1 Answer 1

Reset to default
2

This is how you could do it:

shp = "/localstore/data/shps/contrib_area.shp"
writer = QgsVectorFileWriter(shp, "CP1250", fields, poly_provider.geometryType(), poly_layer.crs(), "ESRI Shapefile")
# Just nuke the writer because we have already made our empty shapefile on disk.
del writer

# Open a reference to the new layer
layer = QgsVectorLayer(shp, "contrib_area","ogr")
dp = layer.dataProvider()

# in your process block
features = get10Features()
dp.addFeatures(features)

QgsVectorDataProvider::addFeatures() will write directly to the disk/datasource so that is the only method you should need to call.

Improve this answer
3
  • I hadn't thought about that. I spent a lot of time digging through the writer documentation trying to find a good solution. I'll try this out and see if it works, might even reduce my memory footprint if addFeatures() writes to disk.
    – Nelz11
    Commented Jun 18, 2013 at 3:29
  • QgsVectorFileWriter is a bit ugly IMO. There really should just be a save() method on the QgsVectorLayer class.
    – Nathan W
    Commented Jun 18, 2013 at 5:24
  • I agree. On a similar note, @Nathan, do you know how efficient combining geometries is with Qgis? Seems like when I get a large number of geometries it tries to hang (or it just takes a long time--several days!) This is what is leading me to try to only do parts of my loop at a time to figure out where and why it is taking so long...
    – Nelz11
    Commented Jun 18, 2013 at 15:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.