What is the best approach to create a create a very large shapefile (close to the max of 2GB) using python? Best approach means: robust and fast. Above all fast.
I want to take the chance of promoting OGR's virtual file system that writes geometries to a in-memory dataset.
Using it is simple as @Luke demonstrated in this post
drv = ogr.GetDriverByName( 'ESRI Shapefile' ) ds = drv.CreateDataSource(r'/vsimem/virtual.shp')
This works just great. Creating a point shape file with ~300.000 geometries and two attribute fields takes less than two seconds. Impressive.
A slow method is updating the shapefile, on disk, for each record. I would look for a Python library that allows you to create an in-memory layer, then write the shapefile to disk in a single function or transactions. Fione/shapely may offer something. Also, ogr2ogr will create a shapefile from csv, or another shapefile, etc. Ogr2ogr is written in C or C++ so it should be fast, but is not a pure Python solution. You'll want to define the shapefile attributes as tight as possible to keep the shapefile as small as possible.