Why does writing large shapefiles crash QGIS?

Question

I'm trying to write large shapefiles with QGIS 2.0 API, using Python, which crashes QGIS. The data is composed of simple two point lines and a data table of five columns. A thousand features or so works fine, but at 10k the program crashes for sure. Perhaps some sort of writing in chunks is a solution, or the problem lies in my code :

def write_lines (file_name, data_list, coordinate_ref_system):

    fields = QgsFields() 
    fields.append(QgsField("Data_1", QVariant.String ))
    fields.append(QgsField("Data_2", QVariant.String))

    writer = QgsVectorFileWriter( file_name + ".shp", "CP1250", fields,
                          QGis.WKBLineString, coordinate_ref_system)
    if writer.hasError() != QgsVectorFileWriter.NoError:
        QMessageBox.information(None, "ERROR!", "Cannot write file (?)")
        return 0

    for r in data_list:
        # create a new feature
        feat = QgsFeature()

        l_start=QgsPoint(r[0],r[1])
        l_end =QgsPoint(r[2],r[3])
        feat.setGeometry(QgsGeometry.fromPolyline([l_start, l_end]))

        feat.setFields(fields)
        #feat.setAttributes([ str(r[4]), str(r[5]) ]) DOESN'T WORK?

        feat['Data_1'] = r[4]
        feat['Data_2'] = r[5]

        writer.addFeature(feat)
    del writer

Thanks!

Answer 1

Adding content of comment against question as an answer to allow this to be closed

In the loop, the construction of the feature

    for r in data_list:
        # create a new feature
        feat = QgsFeature()
        # do something with the new feature
        ....

Is allocating memory for the feature, but it isn't being released anywhere. Conceptually the python memory manager should be able to tell that the feature could be garbage collected, but that is often difficult in applications that wrap C/C++ calls.

The solution to running out of memory in this case is to provide an explicit "I've finished with that feature, thanks" indication, using del

So the code structure will look like:

    for r in data_list:
        # create a new feature
        feat = QgsFeature()
        # do something with the new feature
        ....
        del feat

Note: This technique is generally applicable (so it might be regarded as off-topic for GIS.SE), but since you wouldn't need to do this in most python programming, I think its "close enough"