I am wondering what storage method will result in the fastest reading of the map vectors for rendering. SHP? PostGres? SQLite? (They do not change often and I do not need spatial functions for these vectors).
There are some very speed tests of shapefiles versus database (PostGIS) for MapServer in this presentation (from 2007).
- For a dataset of 3 million features running requests for 30 features one after another PostGIS was faster than shapefile (although this may have since changed by a fix to reading the shapefile index)
- For a dataset of 10,000 features shapefile was slightly faster.
- For concurrent requests shapfile was faster
And the times in detail, which can also help to decide if the storage format is an important factor.
PostGIS Shapefile Start mapserv process 15ms 15ms Load mapfile 3ms 3ms Connect to DB 14ms n/a Query 20ms n/a Fetch 7ms n/a Draw 11ms 28ms Write Image 8ms 8ms Network Delay 3ms 3ms
Always use FastCGI in MapServer if using a database, as the database connections can be reused, otherwise a new connection must be created on every request.
Implementations for Shapefile Readers
The speed of reading a shapefile (and data from a database) depends on the specific coding implementation.
The source code for MapServer opening a shapefile can be seen here. Following the comments you can see how important it is to have an index. Normally you can only read a file in one direction to get a record, but with an index you can read in two directions.
345 /* Read the .shx file to get the offsets to each record in */ 346 /* the .shp file.
Another is example of opening a shapefile can be seen in the Python source for PyShp. Again you can see how an index is used to find specific shapes directly.
Other Factors to Condsider
The limitations of the DBF format (limits on field size, no null support, limits on text storage), should also be taken into consideration when deciding on whether or not to use a database.
A database also offers means of securing data, easier joining and creation of views, logging and many other features you won't get with a standalone file.
Contrary to what dariapra says, my experience in developing Maperitive tells me that the greatest bottleneck is in actual loading of the data before rendering. It all very much depends on how large the overall stored dataset is and how large is the dataset you are trying to render in one go. If you can load it all up into memory, then shapefiles are probably better than using a database engine.
Which program will you use for rendering? This may influence the results. Anyway, having a shapefile with a spatial index (eg http://mapserver.org/utilities/shptree.html ) which is used is often the fastest technique. Apart from that: it depends on you application, but caching your rendered results is often much more useful for improving performance.
Shapefile will be the quickest and probably your best bet. There is overhead for any SQL database, then there is managing the return of large result sets (conversion from database datatypes to native datatypes will also slow things).
Try using an open source package from maptools.org to do your reads. The ArcGIS tools, though purpose built, do have a bit of overhead to get started and are expensive.
Hope this helps