Suggestions for vector format for contour map

Question

I've been trying to create a contour map of a DEM and I've been running into some technical problems. I want to avoid artifacts, so I'm using a DEM of the entire area. This means the size is rather large, over 40 gb. I then run gdal_contour to get a shapefile.

I think gdal started throwing out errors when the .shp file reached over 4gb. This is over the filesize limit of 2gb, so I did not get a finished map or valid shapefile. So I was wondering if anyone had experience working with large raster sets, and had suggestions for an appropriate file format.

Other considerations:

Space is a small consideration. I created a few contour maps of a small sample section, and noticed that the other formats were over two and a half times the size of the shapefile.

Ram is an issue. I'd like to keep ram usage minimized. I only have 16gb of ram, and I saw gdal_contour go up to 12gb of ram used. My fear is that going to a more space intensive format will increase ram usage. I can go up to 32gb of ram, but I'm also wanting to create another contour map with a higher definition, which will increase ram usage.

Answer 1

If portability is not an issue, set up a postgis database. That can take the whole world (in the form of openstreetmap data).

If you want the data to be portable, think of a spatialite database. I have not yet used such large amount of data with it, so I will be pleased to hear your experience.

To speed up processing, spatial indices can help on both databases.