Tell me more ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.
up vote 8 down vote favorite
6

I am wondering what storage method will result in the fastest reading of the map vectors for rendering. SHP? PostGres? SQLite? (They do not change often and I do not need spatial functions for these vectors).

share|improve this question
1  
How many features are you talking? – Roger D. Jul 8 '11 at 18:30
2  
Your question seems to imply that the bottleneck is the process of reading data. My experience is that actual load is in the rendering process itself, at least with MapServer. That is, I do not see that using shapefiles or an spatial database will make a great difference. – dariapra Jul 8 '11 at 18:41
@Roger - I'm not sure. Its a Map of the US using the highest resolution (states, counties, roads, rivers). I got the data from naturalearthdata.com – Nate Jul 8 '11 at 20:26
@Nate, that being the case, why note read in OSM, or Google for your map? Then overlay on top of them? To render the entire US live each-and-every time is a large process. Then you need to consider network performance, disk-io performance, CPU. There are many things that can and will impact performance. – D.E.Wright Jul 11 '11 at 3:32

4 Answers

up vote 12 down vote accepted

Speed Tests

There are some very speed tests of shapefiles versus database (PostGIS) for MapServer in this presentation (from 2007).

In summary:

  • For a dataset of 3 million features running requests for 30 features one after another PostGIS was faster than shapefile (although this may have since changed by a fix to reading the shapefile index)
  • For a dataset of 10,000 features shapefile was slightly faster.
  • For concurrent requests shapfile was faster

And the times in detail, which can also help to decide if the storage format is an important factor. 

                       PostGIS   Shapefile 
Start mapserv process  15ms      15ms
Load mapfile           3ms       3ms
Connect to DB          14ms      n/a
Query                  20ms      n/a
Fetch                  7ms       n/a
Draw                   11ms      28ms
Write Image            8ms       8ms
Network Delay          3ms       3ms

Always use FastCGI in MapServer if using a database, as the database connections can be reused, otherwise a new connection must be created on every request.

Implementations for Shapefile Readers

The speed of reading a shapefile (and data from a database) depends on the specific coding implementation.

The source code for MapServer opening a shapefile can be seen here. Following the comments you can see how important it is to have an index. Normally you can only read a file in one direction to get a record, but with an index you can read in two directions. 

345   /*    Read the .shx file to get the offsets to each record in             */
346   /*    the .shp file.

Another is example of opening a shapefile can be seen in the Python source for PyShp. Again you can see how an index is used to find specific shapes directly.

Other Factors to Condsider

The limitations of the DBF format (limits on field size, no null support, limits on text storage), should also be taken into consideration when deciding on whether or not to use a database.

A database also offers means of securing data, easier joining and creation of views, logging and many other features you won't get with a standalone file.

share|improve this answer
up vote 9 down vote

Contrary to what dariapra says, my experience in developing Maperitive tells me that the greatest bottleneck is in actual loading of the data before rendering. It all very much depends on how large the overall stored dataset is and how large is the dataset you are trying to render in one go. If you can load it all up into memory, then shapefiles are probably better than using a database engine.

share|improve this answer
4  
+1 for pointing out that the form of the bottleneck matters and providing information from actual experience. Good stuff. – whuber♦ Jul 8 '11 at 19:34
Where is the bottleneck seems an interesting and open question that depends on a number of variables, perhaps the kind of workload is one important. I remember that once I indexed some shapefiles with shapetree for getting a faster rendering of some vector layers with MapServer, and I did not get a significative performance gain. – dariapra Jul 8 '11 at 21:17
2  
@dariapra True, it very much depends on the use case. If he's able to load all of the data in one go, then the spatial index is not really necessary, but loading of data from a shapefile should be much quicker than executing SQL queries. On the other hand, if there's a huge amount of data that needs to be filtered, I would put my bet on a database and not shapefiles. – Igor Brejc Jul 9 '11 at 3:32
up vote 3 down vote

Which program will you use for rendering? This may influence the results. Anyway, having a shapefile with a spatial index (eg http://mapserver.org/utilities/shptree.html ) which is used is often the fastest technique. Apart from that: it depends on you application, but caching your rendered results is often much more useful for improving performance.

share|improve this answer
Thank you. I am using MapGuide at the moment (still kind of exploring our options), and I have been looking at Mapserver. – Nate Jul 8 '11 at 20:19
up vote 1 down vote

Shapefile will be the quickest and probably your best bet. There is overhead for any SQL database, then there is managing the return of large result sets (conversion from database datatypes to native datatypes will also slow things).

Try using an open source package from maptools.org to do your reads. The ArcGIS tools, though purpose built, do have a bit of overhead to get started and are expensive.

Hope this helps

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.