I am wondering what storage method will result in the fastest reading of the map vectors for rendering. SHP? PostGres? SQLite? (They do not change often and I do not need spatial functions for these vectors).
-
1How many features are you talking?– Roger D.Commented Jul 8, 2011 at 18:30
-
2Your question seems to imply that the bottleneck is the process of reading data. My experience is that actual load is in the rendering process itself, at least with MapServer. That is, I do not see that using shapefiles or an spatial database will make a great difference.– dariapraCommented Jul 8, 2011 at 18:41
-
@Roger - I'm not sure. Its a Map of the US using the highest resolution (states, counties, roads, rivers). I got the data from naturalearthdata.com– NateCommented Jul 8, 2011 at 20:26
-
@Nate, that being the case, why note read in OSM, or Google for your map? Then overlay on top of them? To render the entire US live each-and-every time is a large process. Then you need to consider network performance, disk-io performance, CPU. There are many things that can and will impact performance.– D.E.WrightCommented Jul 11, 2011 at 3:32
Add a comment
|
5 Answers
Speed Tests
There are some speed tests of shapefiles versus database (PostGIS) for MapServer in this presentation (from 2007).
In summary:
- For a dataset of 3 million features running requests for 30 features one after another PostGIS was faster than shapefile (although this may have since changed by a fix to reading the shapefile index)
- For a dataset of 10,000 features, shapefile was slightly faster.
- For concurrent requests, shapfile was faster
And the times in detail, which can also help to decide if the storage format is an important factor.
PostGIS Shapefile
Start mapserv process 15ms 15ms
Load mapfile 3ms 3ms
Connect to DB 14ms n/a
Query 20ms n/a
Fetch 7ms n/a
Draw 11ms 28ms
Write Image 8ms 8ms
Network Delay 3ms 3ms
Always use FastCGI in MapServer if using a database, as the database connections can be reused, otherwise a new connection must be created on every request.
Implementations for Shapefile Readers
The speed of reading a shapefile (and data from a database) depends on the specific coding implementation.
The source code for MapServer opening a shapefile can be seen here. Following the comments you can see how important it is to have an index. Normally you can only read a file in one direction to get a record, but with an index you can read in two directions.
345 /* Read the .shx file to get the offsets to each record in */
346 /* the .shp file.
Another is example of opening a shapefile can be seen in the Python source for PyShp. Again you can see how an index is used to find specific shapes directly.
Other Factors to Consider
The limitations of the DBF format (limits on field size, no null support, limits on text storage), should also be taken into consideration when deciding on whether or not to use a database.
A database also offers means of securing data, easier joining and creation of views, logging and many other features you won't get with a standalone file.
answered Jul 9, 2011 at 10:44
Contrary to what dariapra says, my experience in developing Maperitive tells me that the greatest bottleneck is in actual loading of the data before rendering. It all very much depends on how large the overall stored dataset is and how large is the dataset you are trying to render in one go. If you can load it all up into memory, then shapefiles are probably better than using a database engine.
-
4+1 for pointing out that the form of the bottleneck matters and providing information from actual experience. Good stuff.– whuberCommented Jul 8, 2011 at 19:34
-
Where is the bottleneck seems an interesting and open question that depends on a number of variables, perhaps the kind of workload is one important. I remember that once I indexed some shapefiles with shapetree for getting a faster rendering of some vector layers with MapServer, and I did not get a significative performance gain.– dariapraCommented Jul 8, 2011 at 21:17
-
2@dariapra True, it very much depends on the use case. If he's able to load all of the data in one go, then the spatial index is not really necessary, but loading of data from a shapefile should be much quicker than executing SQL queries. On the other hand, if there's a huge amount of data that needs to be filtered, I would put my bet on a database and not shapefiles. Commented Jul 9, 2011 at 3:32
Which program will you use for rendering? This may influence the results. Anyway, having a shapefile with a spatial index (eg http://mapserver.org/utilities/shptree.html ) which is used is often the fastest technique. Apart from that: it depends on you application, but caching your rendered results is often much more useful for improving performance.
-
Thank you. I am using MapGuide at the moment (still kind of exploring our options), and I have been looking at Mapserver.– NateCommented Jul 8, 2011 at 20:19
Shapefile will be the quickest and probably your best bet. There is overhead for any SQL database, then there is managing the return of large result sets (conversion from database datatypes to native datatypes will also slow things).
Try using an open source package from maptools.org to do your reads. The ArcGIS tools, though purpose built, do have a bit of overhead to get started and are expensive.
Hope this helps
This question is probably older than the format. I'd add geofeather, which, base on my experience (not scientific study) is really fast.
It is supported natively on GeoPandas and QGIS (from certain version up). An unfortunate thing is that you can't edit them in QGIS.